VideoWorld: Unsupervised Video Models

• VideoWorld is a class of generative video world models that learns rules, planning, and dynamics from raw pixel sequences using autoregressive and diffusion-based architectures.
• It employs a two-stage system with VQ-VAE encoding and latent dynamics modules to jointly predict pixel-level details and high-level scene changes.
• Experimental results in domains like Go and robotic manipulation demonstrate robust rule internalization and forward planning, emphasizing the role of latent code expressivity and temporal abstraction.

VideoWorld denotes a class of generative video world models designed to acquire rules, reasoning, and planning abilities directly from raw video or pixel sequences, in the absence of explicit state, text, or reward supervision. Recent research frames VideoWorld as a paradigm that enables agents to learn structured, transferable knowledge—akin to policy, environment dynamics, or high-level world models—solely through video prediction tasks. Notably, the foundational "VideoWorld" models leverage autoregressive or diffusion-based architectures in conjunction with specialized latent dynamics modules to internalize complex game rules, visuomotor skills, and abstract planning templates from unlabeled video data, generalizing far beyond traditional video synthesis.

1. Architectural Foundations of VideoWorld Models

The canonical VideoWorld model (Ren et al., 16 Jan 2025) is structured as a two-stage generative system integrating:

• VQ-VAE Encoder and Transformer: Each video frame $x_t\in\mathbb{R}^{3\times H\times W}$ is discretized into a token sequence $v_t=[v_{t,1},...,v_{t,N}]$ via a VQ-VAE. A large, autoregressive Transformer, parameterized by $\theta$, is trained on a next-token objective:

$$L_{\mathrm{AR}} = -\sum_{t=1}^T \sum_{i} \mathrm{CE}\left( v_{t,i},\; p_\theta(v_{t,i} \mid v_{1:t-1}, z_{1:t-1}) \right)$$

where $z_{1:t-1}$ denotes auxiliary latent codes described below.

• Latent Dynamics Model (LDM): To extract temporally-compact, high-level causal factors, VideoWorld employs an LDM that encodes multi-step visual change (from $t$ to $t+H$) into $H$ quantized latent codes $z_t^h$ via cross-attention between learned queries $q^h$ and ViT-style encoded features $v_t=[v_{t,1},...,v_{t,N}]$0. The LDM is trained with an $v_t=[v_{t,1},...,v_{t,N}]$1 reconstruction loss:

$v_t=[v_{t,1},...,v_{t,N}]$2

These LDM codes are appended to the token vocabulary and jointly predicted in the autoregressive framework.

By appending LDM codes to visual tokens, the Transformer learns to predict both pixel-level appearance and compact representations of scene change, facilitating the emergence of latent planning, rule-following, and long-term reasoning.

2. Knowledge Acquisition and the Role of Latent Change Representation

VideoWorld departs from models that require explicit state signals or text prompts by demonstrating that pure next-frame or next-code prediction is sufficient for substantial knowledge acquisition in both rule-driven games and continuous control settings. Notable empirical findings (Ren et al., 16 Jan 2025):

• Rule Learning in Go: VideoWorld achieves $v_t=[v_{t,1},...,v_{t,N}]$3100% legal move rate on unseen boards, indicating successful internalization of complex, non-local constraints (e.g., Ko rules, superko) purely from video.
• Forward Planning in LDM Latents: UMAP projections of LDM codes reveal clustering associated with multi-step tactical motifs, and replacing specific codes at inference time shows causal control over predicted outcomes, validating their interpretation as internal "planning sketches."
• Compactness and Expressivity: Learning exclusively from pixels is slow to converge (action-value 47.5%), but adding LDM codes raises performance (73.9% for VideoWorld-50M). Using only state tokens is fastest (79.7%), but lacks richness. Codebook size and prediction horizon ablations confirm the necessity of a sufficiently expressive and temporally-spanning latent representation.

3. Experimental Evaluations in Complex Domains

VideoWorld is empirically validated on both symbolic and continuous domains, effecting strong world-model performance in the absence of explicit reward or state supervision:

• Go (Video-GoBench, 9×9 boards):
  • VideoWorld models scale from 50M to 300M parameters, reaching an Elo of 2,317 (5-dan professional), with 88.1% action-accuracy and 83.7% action-value, all without search or reinforcement learning algorithms.
  • The model's pure pixel+code prediction matches or exceeds a Transformer trained directly on state sequences.
• Robotic Manipulation (CALVIN, RLBench):
  • In multi-task robot benchmarks, VideoWorld achieves up to 75.4% success in "open/close" tasks (CALVIN), generalizes across domains in combination with as few as 10k labeled samples, and closely approaches oracle models that have privileged access to ground-truth action/state labels.
  • Ablations indicate that single-step LDM codes suffice for basic control, but longer horizons (e.g., $v_t=[v_{t,1},...,v_{t,N}]$4) further improve fine-grained manipulation.
• Cross-Domain Generalization: Performance drops sharply when trained only on unannotated human data, underscoring the advantage of mixed, high-quality task video sources.

4. Scaling, Ablations, and Generalization Properties

The scaling behavior of VideoWorld models is monotonic with respect to model size in terms of Elo, action-value, and action-accuracy, confirming suitability for large-scale unsupervised video knowledge extraction.

Key ablation findings (Ren et al., 16 Jan 2025):

Factor	Effect on Performance
LDM horizon $v_t=[v_{t,1},...,v_{t,N}]$5	Gains saturate at $v_t=[v_{t,1},...,v_{t,N}]$6 (Go), $v_t=[v_{t,1},...,v_{t,N}]$7 (CALVIN)
Codebook size	64,000 optimal in Go; too small/large impedes CALVIN convergence
LDM code intervention	Early codes exert maximal causal influence
Data mixing	Combining synthetic (KataGo) and human data outperforms either alone

The above results confirm that (1) the tightest bottleneck is expressivity of latent change codes, and (2) both temporal abstraction and codebook size must be matched to the domain's complexity.

5. Theoretical and Practical Implications for World Models

VideoWorld's empirical results substantiate that autoregressive video generation, when augmented with a compact multistep latent dynamics representation, constitutes an effective unsupervised world model. Formally, next-frame/code prediction can be interpreted as implicitly learning the world-model $v_t=[v_{t,1},...,v_{t,N}]$8 with a latent variable structure encoding both object dynamics and implicit policies.

Crucially, the LDM's latent codes correspond to semantically-meaningful, temporally-abstract representations that support generalization, planning, and transfer across domains, without explicit reward or symbolic supervision.

A plausible implication is that similar latent dynamics abstractions may be central to more general multimodal world models—potentially unifying vision, action, and language—if extended with multimodal conditioning.

6. Extensions, Limitations, and Future Directions

Several research directions are highlighted (Ren et al., 16 Jan 2025, Ren et al., 10 Feb 2026):

• Scaling to Real-World, Open-Domain Video: The original VideoWorld is applied to synthetic and constrained environments; extensions such as VideoWorld 2 (Ren et al., 10 Feb 2026) decouple appearance (modeled by pre-trained video diffusion) and dynamics (modeled by compact dLDM codes), demonstrating substantial improvements in generalization to real-world, long-horizon tasks (e.g., handicraft making, robotic manipulation) and robust transfer across domains.
• Integration with Multimodal and LLMs: Joint training on vision and language modalities could ground emerging visual rules in linguistic abstraction, potentially enabling text-to-policy or narrative-conditional world modeling.
• Actionable Latent Dynamics: Moving from discrete code prediction to continuous latent state transitions $v_t=[v_{t,1},...,v_{t,N}]$9 would tighten connections to control theory, reinforcement learning, and model-based planning.
• Hybrid Training Protocols: Combining autoregressive unsupervised learning with limited reward signals, search, or on-policy fine-tuning is expected to further improve the fidelity and deployability of learned world models.

The open-source release of code, data, and pretrained models is intended to accelerate research in video-only, scalable world model learning.

---

References:

• "VideoWorld: Exploring Knowledge Learning from Unlabeled Videos" (Ren et al., 16 Jan 2025)
• "VideoWorld 2: Learning Transferable Knowledge from Real-world Videos" (Ren et al., 10 Feb 2026)