- The paper introduces a dynamics-enhanced latent dynamics model that uses compact discrete motion codes and a pretrained video diffusion model to separate action learning from visual appearance.
- VideoWorld 2 achieves up to 72.3% full paper-folding success and 85.8% block-task success on Video-CraftBench, substantially outperforming prior video-generation and latent-action baselines.
- The method transfers manipulation knowledge across domains, reaching an average CALVIN sequence length of 2.88 after OpenX pretraining and limited task-specific fine-tuning.
Motivation and problem setting
VideoWorld 2 (2602.10102) addresses a specific gap in video-based knowledge learning: whether models can acquire transferable knowledge for complex, long-horizon tasks directly from unlabeled real-world videos. Its predecessor, VideoWorld, demonstrated that autoregressive video generation can learn rules and planning from synthetic data (Go records, simulated robotics), but the authors show it fails on real-world footage. When applied to minute-long handicraft videos, VideoWorld produces distorted hand poses, incorrect object shapes, and inconsistent environments, while state-of-the-art video generation models (Wan2.2 14B, HunyuanVideo 13B, Cosmos) generate visually plausible frames but fail to reproduce task execution even with detailed per-step language annotations.
The paper's central diagnosis is entanglement of action dynamics and visual appearance. In standard VQ-VAE-based generative pipelines, latent codes must encode thousands of tokens of full visual information; in VideoWorld's Latent Dynamics Model (LDM), the reconstruction objective forces latent codes to absorb background motion, lighting changes, textures, and camera displacement. The authors conjecture this overfitting to appearance is what prevents generalization to novel environments—a claim supported by their UMAP analysis showing that without intervention, latent codes for identical actions cluster by environment rather than by action semantics.
Method: the dynamics-enhanced LDM
The core architectural contribution is the dynamics-enhanced Latent Dynamics Model (dLDM), which decouples appearance modeling from action learning by replacing the LDM's decoder with a pretrained Video Diffusion Model (VDM). A causal VQ-VAE encoder with N learnable query embeddings compresses future visual changes into discrete FSQ-quantized latent codes (N=4, vocabulary size 1000); these codes are injected into the pretrained Cosmos DiT 2B via causal cross-attention, so the diffusion model handles all high-fidelity appearance synthesis while the latent space is relieved of encoding fine-grained visual detail.
Two training details are important for stability. First, dLDM begins with a warm-up phase using the original VQ-VAE decoder under a pure reconstruction objective, allowing codes to rapidly learn motion compression. Second, after warm-up, gradients from the decoder are stopped, but its blurry, motion-rich reconstructions are retained as ControlNet-style conditioning for the VDM—providing coarse temporal cues (hand trajectories, object displacements) that prevent the VDM from having to infer motion from scratch. The ablations confirm both mechanisms matter: stopping decoder gradients alone yields roughly a 20% success-rate improvement, and adding the reconstructed-video conditioning contributes up to another ~20% plus a substantial LPIPS gain, with benefits concentrated in the longer paper-folding task.
The extracted latent codes are modeled autoregressively by a repurposed NVIDIA Cosmos AR 4B transformer, conditioned on an initial frame and task instruction. At inference, given a single frame from an unseen environment, the transformer predicts future latent dynamics and the dLDM decodes them into coherent long-horizon execution videos, chaining 93-frame segments beyond the VDM's native context.
Video-CraftBench
The paper introduces Video-CraftBench, ~7 hours (~9.5k clips) of first-person tutorial videos covering five long-horizon handcraft tasks: paper airplane folding, paper boat folding, and building towers/horses/people from blocks. Paper-folding sequences run 40–80 seconds across seven annotated key steps; block-building runs 20–30 seconds. The test set (~150 separately collected videos) uses novel backgrounds, paper textures, and block arrangements. Evaluation combines a DINOv2-based key-step classifier (96.1% test accuracy, trained on ~25k frames including manually verified generated outputs) reporting sequential success rates—where step k counts only if all predecessors succeeded—with SSIM/LPIPS for visual fidelity.
Results on Video-CraftBench
The headline results are stark. Fine-tuned frontier video generators collapse on long horizons: Wan2.2 14B reaches 81.2% at step 1 of paper folding but 0% from step 5 onward, despite step-by-step text supervision. Among latent-action baselines trained only on Video-CraftBench, none completes the full folding sequence; VideoWorld itself reaches 0% at step 7. VideoWorld 2 achieves 68.8% final-step paper-folding success and up to 81.5% on block stacking, with strong visual quality (SSIM 0.770, LPIPS 0.205)—the claimed "up to 70% improvement" in success rate refers to this comparison against prior methods.
| Method |
Step 4 |
Step 7 (full fold) |
Block (best) |
SSIM |
| Wan2.2 14B (text-conditioned) |
10.6 |
0.0 |
42.6 |
0.719 |
| VideoWorld |
21.3 |
0.0 |
33.9 |
0.680 |
| CoLA (OpenX+Craft) |
64.8 |
40.2 |
54.1 |
0.668 |
| VideoWorld 2 (Craft only) |
83.3 |
68.8 |
81.5 |
0.770 |
| VideoWorld 2 (OpenX+Craft) |
86.7 |
72.3 |
85.8 |
0.774 |
Scaling the dLDM with OpenX pre-training further raises the final-step rate to 72.3%, and qualitative analysis shows OpenX and Craft codes for similar motions align despite differing agents and settings. Notably, these gains are obtained without large-scale pre-training of the latent codes themselves—the Craft-only result already demonstrates that the dLDM's disentanglement, not data volume alone, drives transferability.
Results on CALVIN
On CALVIN, the paper follows LAPA's protocol: pre-train the AR transformer on latent codes, then fine-tune on limited ground-truth actions. With only 2k labeled trajectories (10% of ABCD→D), VideoWorld 2 pre-trained in-domain reaches an average sequence length of 1.87 versus 1.11 for the oracle baseline and 1.49 for LAPA—approaching the fully labeled oracle (2.36). Cross-domain pre-training on 1.3M OpenX trajectories then fine-tuned on 22k CALVIN trajectories yields an average length of 2.88, exceeding both the video next-token-prediction pre-training baseline (2.46) and LAPA (2.51). This supports the claim that latent-dynamics pre-training transfers manipulation knowledge more efficiently than direct video prediction pre-training.
Ablations
The ablation suite substantiates each design choice. Adding the VDM prior improves success rates by ~30% and tightens intra-action clustering of latent codes across CALVIN and Bridge environments. Query length N=4 balances information capacity against noise; N=8 slightly improves LPIPS but reduces success rate. Causal cross-attention between LDM and VDM outperforms non-causal variants (72.3 vs. 52.3 on paper folding), confirming temporal correctness matters. Context length T=2—a LAPA-like pairwise setting—performs poorly (19.1%), showing long-horizon temporal perception within the tokenizer is essential. Codebook size exhibits a clear optimum at 1000 for Craft tasks, with larger codebooks degrading performance through noise encoding. For the VDM, random initialization collapses entirely, freezing underperforms LoRA, and full fine-tuning performs best, indicating the VDM must adapt to fine-grained manipulation details rather than merely supply generic priors.
Limitations and open questions
The paper concedes several constraints. The VDM processes only 93 frames per segment, so full sequences are stitched autoregressively, and reconstruction noise accumulates—producing gradual lighting, texture, and color drift, though key steps remain accurate. The evaluation classifier, while achieving 96.1% accuracy, is itself trained partly on generated outputs from the evaluated models, a potential source of evaluator bias the paper does not analyze. Codebook sizing remains task-dependent (CALVIN prefers smaller codebooks than Craft), leaving open how to select capacity automatically for unseen domains. Finally, the authors explicitly defer continued scaling of the approach to future work; the question of whether dLDM-style disentanglement holds at internet-scale video corpora, or for tasks where appearance and dynamics are less separable, remains untested.
Conclusion
VideoWorld 2 provides evidence that decoupling action dynamics from visual appearance—by delegating appearance to a pretrained diffusion model—is the critical ingredient for learning transferable policies from raw real-world video. The combination of a compact discrete dynamics code space, coarse-decoder motion conditioning, and autoregressive policy modeling yields large gains on long-horizon handicraft tasks and effective cross-domain transfer from OpenX to CALVIN, establishing a concrete benchmark and methodological baseline for future work on video-derived world knowledge.
