Reliable long-horizon prediction remains a central challenge for embodied world models. Although recent generative models can synthesize visually plausible action-conditioned futures, their latent dynamics are typically learned through unconstrained transition modules, which may gradually deviate from coherent physical evolution as predictions are recursively rolled forward. We investigate whether explicitly structuring latent state evolution can improve the stability and utility of such imagined trajectories.
To this end, we introduce HWM, a structured dynamics framework that represents backbone latents in a learned phase space consisting of generalized coordinates and momenta. Future states are propagated through dissipative Hamiltonian dynamics and mapped back to the original latent representation through a learnable inverse interface. The resulting formulation can be integrated into existing world-model architectures without redesigning their perception or decoding components, while multi-step transition supervision encourages the structured rollout to remain aligned with the latent trajectories induced by real observations.
Controlled simulation experiments show that HWM produces more accurate estimates of velocity, acceleration, and jerk and substantially reduces long-horizon energy drift relative to unstructured latent dynamics. We further evaluate the approach across three distinct embodied world-model settings, covering single-arm multi-view prediction with Ctrl-World, dual-arm single-view prediction with EVAC, and task-oriented single-arm prediction with Cosmos-Predict2.5-2B. On RoboCasa, incorporating HWM into Cosmos-Predict2.5-2B yields a 2.7% relative improvement in task success. On real-world AgiBot World trajectories, it improves nDTW by 9.7% with Ctrl-World and Logic by 8.4% with EVAC. Across these settings, the results indicate that introducing explicit dynamical structure into latent evolution can improve both long-horizon predictive stability and the downstream usefulness of embodied world-model rollouts.
A short walkthrough of Hamiltonian World Model: unstructured transitions can look plausible while the physics drifts, and phase-space dynamics keep imagined robot futures consistent.
Comparison between an ordinary generative world model and Hamiltonian World Model. Left: a generic transition can look plausible over a short horizon while speed change and energy transfer remain unconstrained, so long-horizon contact becomes physically implausible. Center: the same observation and action are evolved in phase space, and the predicted future stays physically consistent. Right: real-robot scores on EVAC, Ctrl-World, and RoboCasa, together with simulated position, velocity, acceleration, and jerk error.
Given an observation \(o_t\), the encoder \(E_\phi\) maps it to a latent \(z_t\). HWM lifts \(z_t\) into generalized coordinates and momenta \((q_t, p_t)\) and rolls them forward with a learned Hamiltonian \(H_\theta(q,p)=\tfrac{1}{2}p^\top M_\theta^{-1}(q)\,p+V_\theta(q)\) and a semi-implicit symplectic Euler step. Predicted phase states are mapped back to backbone latents and decoded to future observations. Training combines world-model reconstruction, phase-space reconstruction, and multi-step transition consistency.
| Method | Step | Pos. RMSE ↓ | Vel. RMSE ↓ | Acc. RMSE ↓ | Jerk RMSE ↓ |
|---|---|---|---|---|---|
| Vanilla | 30 | 6.217 | 3.455 | 10.966 | 14.135 |
| Ours | 30 | 6.212 | 3.372 | 9.235 | 11.401 |
| Vanilla | 50 | 5.590 | 2.545 | 8.332 | 12.329 |
| Ours | 50 | 5.524 | 2.525 | 7.483 | 10.750 |
Instantaneous physical-state prediction. All models are trained with a 10-step horizon and evaluated at 30 and 50 steps. Lower is better.
| Train | Test | MSE ↓ | MAE ↓ | ||
|---|---|---|---|---|---|
| Vanilla | Ours | Vanilla | Ours | ||
| 10 | 10 | 2.76 | 2.56 | 23.88 | 21.27 |
| 10 | 30 | 25.38 | 23.82 | 53.56 | 49.27 |
| 10 | 50 | 55.57 | 51.18 | 87.59 | 80.37 |
| 10 | 60 | 62.32 | 53.03 | 95.32 | 82.58 |
| 30 | 30 | 25.81 | 24.38 | 79.36 | 73.31 |
| 30 | 50 | 37.55 | 33.65 | 94.54 | 87.84 |
| 30 | 60 | 44.59 | 38.93 | 102.55 | 94.60 |
Video prediction under different training and evaluation horizons. Lower is better. Trained for 10 steps and rolled out to 60, HWM reduces MSE from 62.32 to 53.03 and MAE from 95.32 to 82.58.
| Method | nDTW | PSNR | BLEU | Scene | CLIP | Logic |
|---|---|---|---|---|---|---|
| Vanilla | 0.670 | 10.13 | 0.767 | 90.88 | 78.31 | 93.3 |
| Ours | 0.735 | 10.16 | 0.802 | 90.96 | 78.96 | 96.7 |
Single-arm multi-view manipulation on AgiBot World, evaluated with EWMBench. All metrics are higher-is-better. nDTW improves by 9.7% relative.
| Method | nDTW | PSNR | BLEU | Scene | CLIP | Logic |
|---|---|---|---|---|---|---|
| Vanilla | 0.282 | 16.61 | 0.104 | 91.3 | 77.5 | 80.0 |
| Ours | 0.283 | 16.64 | 0.111 | 92.2 | 78.4 | 86.7 |
Dual-arm single-view manipulation. Logic rises from 80.0 to 86.7, an 8.4% relative gain. Percent metrics are reported on the paper's 0–100 scale.
| Method | LIBERO (%) | RoboCasa (%) | ||||
|---|---|---|---|---|---|---|
| Spatial | Object | Goal | Long | Avg | Avg | |
| Vanilla | 97.6 | 99.8 | 96.2 | 97.6 | 97.8 | 66.7 |
| Ours | 98.6 | 99.8 | 98.0 | 98.2 | 98.7 | 68.5 |
The observation interface, tokenizer, and renderer stay fixed. Only the latent transition is replaced. RoboCasa average success moves from 66.7% to 68.5%, a 2.7% relative gain.
The paper plots compare an ordinary transition with HWM on velocity and energy. Below, Ctrl-World is on top and EVAC is underneath. Inside each clip the ordinary finetune is on the left and the Hamiltonian rollout is in the center.
Qualitative velocity curves from the paper. Dashed gray is ground truth, red is the transition without Hamiltonian structure, and teal is HWM. Without the Hamiltonian, speed spikes or drifts; HWM stays closer to the true curve on uniform, accelerated, and rebound motion.
System energy and relative Hamiltonian drift. Ordinary finetuning lets the energy wander away from the ground-truth level. HWM keeps a smaller drift, which is the same physical-consistency claim tested in the videos below.
Ctrl-World is on top and EVAC is below. Inside every clip, left is ordinary finetuning and center is the Hamiltonian transition. The right panel is only \(|\mathrm{Hami}-\mathrm{Vanilla}|\): bright pixels mark where the two disagree. That panel is not another result, so its upper and lower rows are not “better” or “worse.” On EVAC, the stick figure under the camera view is that method’s arm joints.
RoboCasa is on the left and LIBERO is on the right, in matching panels. Inside each panel the vanilla failure is on top and the Hamiltonian success is below. Clips marked Paper match the figure above. RoboCasa pairs share the same episode.
Paper figure. Top block: RoboCasa PnPCounterToStove. Bottom block: LIBERO, put both moka pots on the stove. Success rows are HWM; failure rows are the vanilla transition.
| Task | V195 | V196 | V197 | V avg | H195 | H196 | H197 | H avg | Δ |
|---|
Task-level success on Cosmos with RoboCasa, three seeds. V is vanilla and H is HWM. The reported average is the unweighted mean of 24 tasks; individual tasks can move in either direction.
HWM places physical structure in the latent transition itself. Phase variables evolve under controlled dissipative Hamiltonian dynamics, with a symplectic rollout and a residual path back to the backbone latent space. Across simulation, Ctrl-World, EVAC, and Cosmos, this changes predicted motion, energy drift, visual-temporal metrics, and downstream task success without replacing the renderer or the tokenizer.
The experiments do not involve human subjects or deployment on physical robots. A more physically consistent world model can make model-based evaluation more faithful, and it still needs the same care as any component used for autonomous control.