Masked Visual Actions for Unified World Modeling

Abstract: Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation. We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video. Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome. Finetuned with only 15 hours of masked examples from real videos and simulation, a single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. In downstream manipulation settings, the model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing robot motion from desired object motion.

• The paper introduces Masked Visual Actions, a pixel-space control interface that unifies forward prediction and inverse action synthesis using a pretrained video model.
• It leverages dense visual masking with automatic segmentation and 3D mesh rendering to achieve robust cross-embodiment generalization and high-fidelity predictions.
• Empirical results show improvements in LPIPS, SSIM, and policy evaluation correlation, demonstrating effective planning and action extraction for robotic tasks.

Masked Visual Actions for Unified World Modeling

Motivation and Conceptual Foundation

Video models encode broad, structured priors over spatiotemporal dynamics, making them an appealing substrate for robotic world modeling. However, a persistent bottleneck for deploying such models in robotics is the design of an action interface that is both visually aligned with the pretrained model’s native data structure, yet agnostic to embodiment and action space. The paper introduces Masked Visual Actions (MVA) as a pixel-space control interface—action is expressed as partially revealed spatiotemporal regions corresponding to entities in the video input. By masking in visual space, the approach offers dense and direct conditioning, supporting both forward (action-conditioned prediction of world response) and inverse (finding actions to cause a target visual outcome) reasoning in a single model.

Figure 1: Masked visual actions allow conditioning a pretrained video model on masked trajectories, enabling both action-conditioned prediction and outcome-driven inverse synthesis using the same architecture.

Positioning Relative to Prior Work

Conventional video world models typically accept action channels as low-dimensional vectors (e.g., joint or end-effector positions) or sparse cues such as skeletons, which impedes cross-embodiment generalization and hinders leveraging the visual priors of the video model. MVA departs fundamentally by grounding actions in image space, making the representation fully visual, dense, and directly comparable across robots and object interactions.

Figure 2: Comparison between traditional action representations and MVA, with dense, visually aligned signals yielding better learnability and embodiment invariance.

Unlike approaches that treat forward and inverse models as separate modules (often requiring explicit architecture modifications or retraining), MVA reframes both as queries to the same conditional video model, based on which set of entities (robot or object) are revealed via the mask. The spatial masking operation thus acts as a unifying mechanism for diverse tasks.

Methodological Framework

The implementation leverages a state-of-the-art, large pretrained video generation model, Wan-Fun-Control 2.2 14B, fine-tuned with approximately 15 hours of real and simulated robot data. Two complementary mechanisms construct the conditioning masks: (1) automatic segmentation of robot arms using SegmentAnything, and (2) rendering 3D robot meshes aligned with ground-truth state for arbitrary trajectory visualization.

Figure 3: The dataset combines segmentation and mesh rendering, enabling the model to learn from diverse robot embodiments and support arbitrary masked conditioning at inference.

Only the LoRA layers are fine-tuned, keeping the model adaptation efficient and parameter-economical.

Core Functionalities and Applications

By controlling which entities’ motion is revealed in the input mask, the model realizes:

• Forward modeling: Given masked robot actions, the model predicts the visual consequences (object and scene evolution).
• Inverse modeling: Given masked object or desired outcome, it synthesizes plausible robot motion to achieve the effect.

This framing subsumes three key robotics utilities:

Figure 4: The architecture naturally supports policy evaluation, model-based planning (by simulating counterfactuals), and action extraction/inverse modeling from masked visual outcomes.

1. Policy Evaluation: Rollouts generated by the model statistically track actual execution outcomes, enabling proxy evaluation and simulation without further environment interaction.
2. Model-Based Planning: The model enables the ranking and selection of policy-sampled trajectories in a test-time planning loop, using video-LLMs (VLMs) as judges.
3. Action Extraction: Inverse-modeling pipelines extract robot actions capable of achieving target object motions specified via visual masks—with empirical evidence that zero-shot generalization is possible without explicit inverse finetuning.

Empirical Results: Quantitative and Qualitative Evaluation

Visual Fidelity and Action Conditioning Robustness

On DROID and BEHAVIOR datasets, Masked Visual Actions outperforms prior baselines (Ctrl-World, Wan-Move) by substantial margins—yielding LPIPS $0.0945$ and SSIM $0.887$ (DROID) and maintaining high fidelity even for unseen robot morphologies in BEHAVIOR. Crucially, MVA's conditioning generalizes gracefully to unseen embodiments, where standard state-based or skeleton-based representations collapse or hallucinate undesirable robot forms.

Figure 5: Qualitative comparison showing that only pixel-aligned masking enables accurate embodiment generalization and scene consistency.

Figure 6: Models trained on sparse signals fail on novel hardware, while MVA robustly generalizes, preserving interaction correctness and visual coherence.

Planning and Policy Evaluation

Rollout-based planning using MVA-imagined futures, filtered by VLM judges, convincingly improves task completion metrics with increasing candidate sample sizes, demonstrating the approach’s suitability for test-time optimization via counterfactual simulation.

Figure 7: Planning performance as a function of number of candidate rollouts, illustrating scaling benefits of model-based sample evaluation using MVA.

Further, policy evaluation using MVA-generated rollouts shows a nearly perfect correlation ($r=0.982$) with actual environment statistics, validating the proxy effectiveness for real and simulated settings. The model exhibits a slight positive bias towards success, attributed to overconfident simulation, a typical artifact of learned generative models.

Inverse Modeling and Action Extraction

MVA pipelines for extracting robot actions from desired outcome videos yield $90\%$ success on challenging RoboCasa tasks, surpassing imitation learning baselines (Diffusion Policy, ACT, SmolVLA) and achieving this in a zero-shot fashion without task-specific data.

Broader Implications and Future Directions

Practically, Masked Visual Actions provide a general and data-efficient interface for robotic world modeling, supporting planning, evaluation, and policy extraction without explicit architectural or data dependence on robot-specific action spaces. Theoretically, the work demonstrates that pixel-aligned action conditioning unlocks cross-domain generalization, enabling pretrained visual world models to function as flexible, multi-role reasoning engines for robotic interaction.

Limitations remain: The model encodes correlation rather than causality, so errors in physical reasoning (e.g., hallucinated outcomes inconsistent with contact or momentum) persist. The expressive power is bounded by the base generative model’s architecture and its prior training regime.

Conclusion

This work introduces a unified visual-masking interface for pretrained video models, enabling state-of-the-art visual fidelity, robust cross-embodiment generalization, and broad applicability in robotic world modeling tasks. Masked Visual Actions prove highly effective for policy evaluation, decision-time planning, and action inference, with strong experimental support for generalization and competitive quantitative results. The approach paves the way for more general, scalable, and modality-agnostic robot learning frameworks, though advances in causal modeling and model efficiency will be critical for the next generation of world models (2607.19343).