Video Foundation Models

Updated 17 November 2025

Video Foundation Models are high-capacity neural architectures that integrate vision, language, and action modalities to build internal world models for planning and control.
They employ both explicit and implicit modeling approaches, using techniques like future-token alignment and residual prediction to capture state transitions efficiently.
These models deliver enhanced generalization, sample efficiency, and interpretability, proving essential in applications such as robotics, embodied AI, and autonomous driving.

Video Foundation Models are high-capacity neural architectures trained on multimodal video data to induce representations that capture both the dynamics and semantics of environments. These models typically integrate computer vision, language, and action modalities—often termed vision-language-action (VLA) models—and are designed to support task learning, world prediction, and control. A central theme is the emergence or explicit construction of an internal “world model,” which encodes environment state transitions and causal regularities, and is critical for effective planning, generalization, and sample efficiency in diverse tasks such as robotics, embodied AI simulation, spatial reasoning, and autonomous driving.

1. Principles of Implicit and Explicit World Modeling in Video Foundation Models

Video foundation models can be categorized by their treatment of world models—either explicit reconstruction of future states, or implicit modeling via latent embedding dynamics. Explicit approaches forecast future observations (often pixel-wise), incurring significant computational cost and redundancy. Implicit models sidestep full reconstruction, operating directly on abstract state or latent representations.

FLARE (Zheng et al., 21 May 2025) exemplifies implicit world modeling by introducing learnable future tokens into a diffusion transformer policy. At each time step $t$ , the model processes vision-language embeddings $\phi_t$ , the robot’s proprioceptive state $q_t$ , and action chunks $A_t$ . M future tokens $f_1,\ldots,f_M$ are added to the input sequence; after self-attention and cross-attention layers, future-token activations are projected and aligned, via a cosine similarity loss, with embeddings of actual future observations. The latent alignment objective augments the original flow-matching action loss, providing predictive capability inside the policy architecture without pixel decoding. The training objective combines action denoising and latent alignment,

$\mathcal{L}_{total} = \mathcal{L}_{fm} + \lambda\,\mathcal{L}_{align},$

with $\lambda = 0.2$ .

IR-WM (Mei et al., 19 Oct 2025) employs implicit residual modeling in a vision-centric autonomous driving context. Instead of reconstructing the entire future bird’s-eye-view (BEV), the model predicts only the residual change $\Delta S_t^{\mathrm{bev}}$ given previous BEV features and ego-vehicle trajectory, using an autoregressive transformer. A feature alignment module calibrates semantic and dynamic misalignments. Task decoders read out occupancy and planned trajectories from the aligned BEV state.

OpenVLA (Molinari et al., 29 Sep 2025) demonstrates that policy-based VLA transformers, trained without explicit dynamics models, can nevertheless harbor correct representations of state transitions in their middle layers, empirically revealed by linear probes on internal activations. The induced mapping $e_{t+K} = e_t + f(a_t)$ , where $e_t$ is the state embedding and $f(a_t)$ is a transition vector, is recoverable for multiple time horizons $K$ from the activation stream, suggesting emergent internal world models.

2. Architectures and Computational Workflows

Video foundation models uniformly begin with perception modules that encode raw video (and often instruction text) into dense latent representations. In FLARE:

Vision-language encoder (e.g., frozen SigLIP-2) produces cross-modal tokens, compressed via a Q-former to $M\times D$ embeddings.
Policy head is a DiT (Diffusion Transformer), with augmented input including state embedding, action chunk embedding, and future tokens.
Alignment between policy’s future-token activations and future observation embeddings is performed by a projection MLP and cosine similarity loss.
Training mixes robot data (both action and alignment losses) and human egocentric videos (alignment only), supporting co-training on unlabeled video.

IR-WM encodes past frames into BEV features via BEVFormer, predicts residual dynamics with an autoregressive transformer, aligns features using occupancy-conditioned layer normalization, and decodes for occupancy maps and planned trajectories.

AD3 (Wang et al., 15 Mar 2024) introduces the Implicit Action Generator (IAG) to infer “implicit” distractor actions, enabling disentanglement of agent-controlled and background dynamics. Two parallel Dreamer-style models, conditioned on real and inferred actions, are updated via a separated ELBO objective. IAG is trained with cycle-consistency and difference reconstruction losses, quantizing inferred distractor actions to one-hot codes.

3. Evaluation Protocols and Empirical Results

Concrete metrics for evaluating world modeling include:

Task success rates over repeated episodes
Mean L2 trajectory errors and collision rates (planning in driving)
Intersection-over-Union (IoU) for occupancy forecasting
Probing $R^2$ scores for state transition prediction over activation layers (OpenVLA)
Regression and classification metrics for spatial template prediction (implicit spatial models, (Collell et al., 2017))

Key results:

FLARE achieves 70.1% robotics task success on RoboCasa (+8.2% over prior world-modeling baselines), 95.1% success on real humanoid tasks, and strong zero-shot generalization to unseen objects.
IR-WM matches or exceeds preceding methods, with mIoU $_{0/f}$ = 20.3/19.9% (vs. 18.5/17.8%), average trajectory error 0.53 m (vs. 0.85 m), and 0.17% collision rate.
AD3 is robust to heterogeneous and homogeneous distractors, achieving near-oracle performance in DeepMind Control Suite tasks, verified by ablation on IAG and action factorization.
OpenVLA probes recover significant activation-based transition predictability ( $R^2 \approx 0.33$ for $K=10$ ), while embedding baselines are substantially weaker.

4. Interpretability, Probing, and Generalization

Interpretability pipelines, such as the Matryoshka Sparse Autoencoder (SAE) in OpenVLA, allow linear probe outputs (transition vectors) to be decomposed into sparse concept vectors, mapped to semantic patches or tokens, and inspected for anticipated environmental change. This supports human verification and safety-critical vetoing in deployment.

Implicit spatial world models (Collell et al., 2017) induce 2D spatial templates from object-relationship triplets $(S,R,O)$ , enabling prediction even for unseen objects by leveraging fixed pre-trained GloVe embeddings. This provides compositional generalization, interpretable network weights, and exposure of learned spatial priors.

Vafa et al. (Vafa et al., 6 Jun 2024) propose Myhill–Nerode-inspired diagnostics for the internal coherence of generative models' world representations, emphasizing the need for longer-horizon distinguishing suffixes, and revealing fragility in models passing standard next-token or linear-probe tests.

5. Theoretical Formalisms and Compositionality

World Automata (Capiluppi et al., 2013) extend hybrid I/O automata to encapsulate agent-environment systems with spatiotemporal world variables. These act as globally shared fields, supporting compositionality via parallel composition (additive field superposition) and hierarchical inplacement (nested world embedding). Explicit well-posedness conditions for implicit communication (sensing the sum of output perturbations) ensure trace substitutivity and closure under execution and composition, foundational for reliable agent coordination in complex environments.

Meta-reinforcement learning frameworks (Horibe et al., 19 Nov 2024) demonstrate spontaneous emergence of implicit world models and goal-directed exploration in recurrent agents trained for homeostasis across contextual variations. Internal hidden states become sufficient statistics for belief filtration, encoding environmental structure and supporting both adaptation and prediction. Empirical indicators (survival time, next-observation MSE, trajectory structure) substantiate the internalization of world dynamics, paralleling mechanisms in biological cognition.

6. Implications, Open Questions, and Future Directions

Empirical studies support that video foundation models—both explicitly and implicitly constructed—can recover rich world models, fostering generalization, sample efficiency, and robust control. OpenVLA and FLARE evidence that policy-based RL models can internally encode dynamic mappings conventionally attributed to model-based approaches, blurring long-standing distinctions and suggesting hybrid design opportunities.

However, next-token and shallow embed-probes are insufficient metrics; Myhill–Nerode–inspired boundaries reveal latent incoherence under distributional shift, adversarial input, and subtle task variation. This highlights the necessity of global structure-aware diagnostics and structurally regularized training.

Compositional formalisms (World Automata, implicit action-induced factorization) allow modular construction of agent-environment interactive systems, yet require careful semantic alignment and hierarchical encapsulation to avoid cyclic dependencies and ensure observable trace consistency.

A plausible implication is that future video foundation models must integrate world-model probing, modularity, and structural regularization as core training and validation mechanisms, rather than relying solely on next-step prediction or proxy metrics. This would yield models more suited to real-world deployment, zero-shot adaptation, and safety-critical interpretability.