Walk through Paintings: Egocentric World Models from Internet Priors

Abstract: What if a video generation model could not only imagine a plausible future, but the correct one, accurately reflecting how the world changes with each action? We address this question by presenting the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pretrained video diffusion model into an action-conditioned world model, enabling controllable future prediction. Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models and inject motor commands through lightweight conditioning layers. This allows the model to follow actions faithfully while preserving realism and strong generalization. Our approach scales naturally across embodiments and action spaces, ranging from 3-DoF mobile robots to 25-DoF humanoids, where predicting egocentric joint-angle-driven dynamics is substantially more challenging. The model produces coherent rollouts for both navigation and manipulation tasks, requiring only modest fine-tuning. To evaluate physical correctness independently of visual appearance, we introduce the Structural Consistency Score (SCS), which measures whether stable scene elements evolve consistently with the provided actions. EgoWM improves SCS by up to 80 percent over prior state-of-the-art navigation world models, while achieving up to six times lower inference latency and robust generalization to unseen environments, including navigation inside paintings.

• The paper introduces EgoWM, a framework that repurposes pre-trained video diffusion models to generate coherent, action-conditioned visual predictions.
• It integrates an architecture-agnostic action-conditioning pathway using MLP-based embeddings to inject motor commands without retraining the base model.
• It achieves up to 80% higher structural consistency scores and 6× lower inference latency, demonstrating robust generalization to diverse environments including painted scenes.

Egocentric Action-Conditioned World Modeling via Reusable Internet-Scale Video Priors

Introduction

This paper presents Egocentric World Model (EgoWM), a novel framework for transforming pre-trained video diffusion models into controllable world models capable of high-fidelity, action-conditioned future prediction. Departing from prior approaches that require extensive bespoke data collection and model designs for each embodiment and domain, EgoWM leverages Internet-scale passive video priors and injects motor commands through general-purpose conditioning layers. The result is a generalizable and scalable method enabling coherent visual rollouts for high-dimensional action spaces, including up to 25-DoF humanoid agents, while maintaining the flexibility to generalize to entirely novel and visually unrealistic scenarios.

Methodological Framework

EgoWM is instantiated by adding an action-conditioning pathway that is architecture-agnostic, exploiting the universal denoising timestep modulation features of diffusion models. Instead of retraining or modifying the base layers, actions are embedded via MLPs, temporally downsampled to match latent video frame rates, and injected additively into places where timestep embeddings are modulated during denoising. Thus, the framework supports both U-Net and DiT-based diffusion models without architectural changes, facilitating immediate adaptation to varying action dimensionalities and embodiments.

The conditioning module encodes action sequences in their natural parameterization (e.g., $(\Delta x, \Delta y, \Delta\phi)$ for 3-DoF navigation and joint angle vectors for humanoids) and allows the video diffusion model to produce future frames that align strictly with the control trajectory.

Figure 1: Action-embedding integration into pre-trained video diffusion architectures using the timestep-conditioning pathway enables passive video models to act as world models.

Structural Consistency Score (SCS)

To address the inadequacy of perceptual similarity metrics like LPIPS and FVD, which conflate appearance fidelity with physical correctness, the authors introduce the Structural Consistency Score (SCS). SCS quantitatively measures whether stable scene structures (e.g., walls, furniture, landmarks) evolve with the prescribed action sequence, independent of texture and appearance. Automatic segmentation and mask tracking, combined with trajectory overlap computation, yield robust, appearance-invariant comparisons to ground truth, enabling principled evaluation of action-conditioned prediction.

Figure 2: Perceptual metrics may favor visually sharp, physically implausible generations; SCS robustly identifies action-following structural consistency.

Figure 3: SCS computation illustrates robust mask tracking and objective evaluation of action-structure alignment, even with distorted predictions.

Empirical Evaluations

Experiments span classical 3-DoF navigation in real-world robot datasets, 25-DoF humanoid navigation, and high-dimensional egocentric manipulation. EgoWM is instantiated with SVD and Cosmos video diffusion backbones and compared to Navigation World Models (NWM), the leading open-source baseline.

Quantitative results—reported across prediction horizons, action spaces, and environments—show:

• EgoWM achieves up to 80% higher SCS scores than NWM on real-world navigation, with perceptual similarity also improving.
• Inference latency is up to 6$\times$ lower than NWM, with EgoWM producing chunked outputs versus frame-wise autoregressive baselines.
• For 25-DoF humanoid tasks, models maintain perceptual and action alignment margins over baselines, confirming the benefit of large-scale pretraining.

  Figure 4: EgoWM produces temporally and spatially coherent navigation rollouts, accurately following complex control trajectories; NWM exhibits drift.

  

  Figure 5: EgoWM models (SVD and Cosmos) significantly outperform autoregressive NWM baselines in inference latency as trajectory horizon increases.

  

  Figure 6: Action-conditioned prediction on humanoid tasks yields contact-rich manipulation and navigation sequences consistent with action commands.

Additionally, the paper demonstrates generalization to out-of-distribution domains, such as paintings, where the model still obeys action trajectories despite the lack of physical realism, and robust performance on unseen real-world environments.

Figure 7: EgoWM generalizes to visually unrealistic painted environments, preserving motion coherence aligned with action commands.

Figure 8: EgoWM robustly exploits 25-DoF navigation commands in novel real-world scenes, aligning agent and environment actions.

Robustness and Limitations

Failure analysis reveals key limitations:

• Manipulation: Small object permanence and physical shape consistency are challenging across occlusions and articulations.
• Generalization: Cosmos variant trained primarily on real data is more prone to domain collapse when deployed in painted scenes than SVD, attributable to data priors. SVD exhibits occasional structural inconsistency under aggressive action trajectories requiring large-scale scene imagination.

  Figure 9: Failure modes include manipulation shape inconsistencies, generalization collapse in Cosmos, and structural drift in SVD model generations.

Implications and Future Directions

The EgoWM framework demonstrates strong scalability and generalization by harnessing Internet-scale visual priors, enabling rapid adaptation to new tasks, embodiments, or domains without retraining core generative backbones. SCS establishes an objective metric rooted in physical scene evolution, critical for advancing world modeling beyond appearance-driven evaluation.

Practically, this approach accelerates the development of closed-loop agents capable of planning in realistic and synthetic domains, spanning robotics, embodied AI, and creative simulation. Theoretically, leveraging large-scale, passively learned representations to bootstrap controllers marks a paradigm shift away from narrow, data-intensive model specialization.

Potential future research avenues include enhancing small object control and shape consistency, extending SCS to dynamic object tracking, and exploring compositionality in multimodal control spaces.

Conclusion

EgoWM offers a pragmatic, general method for constructing action-conditioned world models by repurposing pre-trained video diffusion architectures with minimal additional data or computation. The framework is validated across challenging navigation, manipulation, and open-world generalization scenarios, providing significant performance, efficiency, and versatility improvements over prior work. The proposed SCS metric further advances evaluation methodology, supporting more rigorous assessment of world model fidelity. These contributions collectively advance the scalable design and deployment of general-purpose visual dynamics models.

(2601.15284)