FlowWAM: Optical Flow as a Unified Action Representation for World Action Models

Abstract: World Action Models (WAMs) are able to leverage pretrained video generators for both world modeling and action prediction. However, directly leveraging such video generators for control raises a new challenge: how to represent actions in a suitable form that aligns with pretrained video generators while carrying enough motion cues for accurate control. Existing numerical actions fail to satisfy the former, and prior visual action representations overlook the temporal motion structure across frames. We address this issue with FlowWAM, a dual-stream diffusion framework that adopts optical flow as a unified, video-native action representation. Flow videos share the same format as RGB videos and encode rich per-pixel displacement. By jointly modeling them within a shared pretrained video generator, FlowWAM can naturally implement two modes of WAMs. In policy mode, FlowWAM generates flow for action prediction, while in world-model mode, it uses target flow sequences to guide future video generation. Moreover, since flow can be easily extracted from raw videos without action labels, FlowWAM can leverage large-scale action-unlabeled video datasets for pretraining. We empirically find that our flow-based action representation delivers gains across both modes. On RoboTwin manipulation, FlowWAM raises the success rate to 92.94% on the Clean setting and 92.14% on Random, outperforming both VLA and WAM baselines. On WorldArena world modeling, it achieves the best overall EWMScore (63.71) with an 18.4% relative improvement in trajectory accuracy. More results can be found on our project website: https://flow-wam.github.io .

• The paper presents optical flow as a unified action representation, enabling seamless video-to-video generation for both policy inference and world modeling.
• It employs a dual-stream diffusion framework with shared VAE and transformer blocks to achieve impressive success rates (92%+) on bimanual robotic tasks.
• Empirical and ablation studies demonstrate that the flow-based approach improves transferability, trajectory accuracy, and real-world robustness in robotic control.

FlowWAM: Optical Flow as a Unified Action Representation for World Action Models

Motivation and Conceptual Framework

FlowWAM proposes a paradigm shift in how World Action Models (WAMs) represent and leverage robot actions for visuomotor control and world modeling. Rather than relying on heterogeneous numerical or symbolic action tokens, or static image-space cues, FlowWAM encodes actions as dense optical flow videos matching the format of RGB frames. This representation bridges the modality gap between executable robot actions and visual motion priors captured by large-scale pretrained video generators, allowing for seamless action-unlabeled video pretraining, robust action inference, and accurate action-conditioned world modeling.

Figure 1: FlowWAM uses optical flow as a unified action representation, aligning robot actions with pixel-space video priors for pretraining, policy inference, and world modeling.

This approach addresses prominent WAM challenges: the lack of transferability across robotic embodiments, the absence of temporally dense motion cues in conventional visual action representations, and bottlenecks in decodability for accurate control. By making optical flow a first-class action representation, FlowWAM enables unified video-to-video generation tasks grounded in motion-centric priors.

Architecture and Dual-Stream Diffusion Design

FlowWAM employs a dual-stream diffusion framework built on a frozen pretrained video generator (Wan2.2-TI2V-5B), maintaining RGB and optical flow streams with shared VAE and transformer blocks, but lightweight stream-specific adapters and patch embeddings. Optical flow is HSV-encoded, allowing invertible transformation between displacement fields and RGB images, which ensures compatibility and reversibility for action decoding.

Policy inference and world modeling are realized as two modes within the same architecture. In policy mode, FlowWAM generates future RGB and flow latents, which are decoded by a transformer-based action expert into robot actions. For world modeling, FlowWAM takes desired flow trajectories as conditioning inputs and synthesizes corresponding RGB predictions, enabling trajectory-accurate action-conditioned planning.

Figure 2: FlowWAM encodes RGB and optical flow as two coupled video streams within a shared diffusion transformer, supporting both policy and world-model modes.

Training Protocol and Pretraining Strategy

A major advantage is FlowWAM's ability to leverage action-unlabeled large-scale egocentric video datasets (EgoDex) for motion-centric pretraining. Optical flow is readily extracted from human manipulation videos, enabling unsupervised acquisition of reusable motion priors. The model is trained with a flow-matching objective over both streams, augmented by motion-aware loss reweighting to counteract static-background dominance and stochastic latent conditioning to align distributions across training/inference.

Stage 1 pretrains the dual-stream video generator on unlabeled videos; Stage 2 attaches the action expert and performs end-to-end optimization on robot demonstrations for embodiment-specific action decoding.

Empirical Results and Quantitative Performance

RoboTwin Manipulation Policy

On RoboTwin 2.0 (50 bimanual tasks), FlowWAM outperforms VLA and state-of-the-art WAM baselines in both Clean and Random settings, achieving a success rate of 92.94% (Clean) and 92.14% (Random). Pretraining on action-unlabeled data amplifies robustness, especially under randomized scenarios, validating scalability and transferability of flow-based motion priors.

Figure 3: Real-world success rates across single-arm and dual-arm manipulation tasks.

Ablation studies reveal that replacing flow with numerical action vectors or raw $(u,v)$ fields degrades performance. The combination of action decoding from flow latents, motion-aware flow loss, and stochastic latent conditioning is decisive for robustness and decodability.

Figure 4: Action representation variants and decodability; design components critically impact policy success and flow prediction error correlates strongly with control reliability.

WorldArena Action-Conditioned World Modeling

On WorldArena, FlowWAM achieves the best overall EWMScore of 63.71 and an 18.4% relative improvement in trajectory accuracy over leading baselines. Dense flow conditioning significantly improves trajectory fidelity, depth accuracy, and semantic alignment, while maintaining high subject and background consistency.

The results demonstrate that flow-based input is superior to text-only, numerical-action, or mask/image-action conditioning for producing physically plausible, instruction-following rollouts.

Real-World Validation and Qualitative Analysis

FlowWAM generalizes to real-world single-arm (Franka) and dual-arm (ARX) robots, attaining 75.7% success across diverse tasks. Alignment between predicted flow plans and executed robot trajectories confirms the effectiveness of optical flow as a decodable, executable motion interface.

Figure 5: Real-world execution examples demonstrate the alignment between predicted optical-flow plans and physical robot manipulation.

Qualitative visualization further exhibits the spatially grounded nature of FlowWAM's predictions, robust transfer across scene randomization, and high fidelity in action-conditioned rollouts.

Figure 6: The action-conditioned world model generates future frames aligned with requested manipulation dynamics, preserving object layout and scene integrity.

Implications and Future Prospects

FlowWAM's dense, video-native action representation advances unified WAM frameworks, opening avenues for internet-scale action-free pretraining, enhanced transfer across robot embodiments, and reliable policy inference under scene diversity and long-temporal horizons. The methodology bridges manipulation-centric motion encoding, decodability, and pretraining, moving toward multi-embodiment generalist policies grounded in motion priors.

Theoretically, treating optical flow as a primary modality aligns action-generation with pixel-level dynamics and leverages advances in diffusion modeling and large-scale video representation. Practically, FlowWAM can be adapted for hierarchical planning, cross-domain transfer, and continual learning scenarios in complex, multi-agent embodied AI systems.

Conclusion

FlowWAM introduces optical flow as a unified, video-native action representation for world action models, enabling dense, decodable control signals and scalable pretraining on action-unlabeled data. Empirical results establish FlowWAM as highly effective for both manipulation policies and world modeling, with demonstrated transfer and robustness across simulation and real-world platforms. The dense flow representation closes the gap between low-level control and pretrained video priors, promising practical advances in scalable, transferable robotic learning and world modeling frameworks (2607.13017).