World-Action Models (WAMs): Overview and Applications

Updated 14 October 2026

World-Action Models (WAMs) are predictive-action models that integrate forecasts of future world states with executable robot actions, enabling robots to generate, select, score, verify, or jointly train actions.
WAMs can be categorized into cascaded and joint models, with the cascaded versions explicitly separating world prediction from action generation and joint models modelling both within the same system.
WAMs are trained using action imitation along with predictive objectives, and their inference can range from open-loop to single-step closed-loop control, with various predictive substrates and modalities, including visual, tactile, depth, semantic, and auditory inputs.

World-Action Models (WAMs) are embodied predictive-action models that couple forecasts of future world states with executable robot actions. Their defining object is a joint or functionally coupled distribution over future states and action chunks, such as

$p_\Theta\!\left(s_{t+1:t+H},a_{t:t+H-1}\mid o_{\leq t},a_{<t},l\right),$

where observations may include RGB, depth, proprioception, tactile or force signals, states may be pixels, video latents, geometric representations, semantic features, physical variables or other predictive substrates, and $l$ denotes a task instruction or goal. A model qualifies as a WAM when the predicted future remains in the action pathway: it helps produce, select, score, verify or jointly train the action. This distinguishes WAMs from reactive Vision-Language-Action (VLA) policies, broad world models used only as simulators, and video-generation systems without executable action grounding (Shen et al., 18 Jun 2026).

1. Definition, scope and conceptual foundations

A conventional VLA commonly represents a direct policy,

$p(a\mid o,l),$

or predicts an action chunk from current observations and an instruction. Such models benefit from vision-language pretraining and can recognize objects, interpret language and associate instructions with familiar manipulation primitives. Their basic objective, however, does not require explicit modeling of how the environment changes after intervention. Consequently, semantic generalization does not necessarily imply generalization to unfamiliar physical motions, contact configurations or dynamics (Ye et al., 17 Feb 2026).

A conventional world model instead predicts environment evolution under an action, for example,

$p(o'\mid o,a),$

or in latent form,

$p(z_{t+1}\mid z_t,a_t).$

It may support model-based reinforcement learning (MBRL), model-predictive control (MPC), reward estimation, policy evaluation or imagined rollouts, but it need not generate actions itself. A WAM incorporates predictive world modeling into action generation or action evaluation. The distinction is functional rather than architectural: WAMs may be autoregressive transformers, diffusion or flow-matching models, latent dynamics systems, video-foundation-model adaptations, or modular combinations of world predictors and inverse-dynamics policies (Lu et al., 13 Sep 2026).

The defining contract can be expressed in several equivalent forms. A jointly trained WAM models

$p_\theta(o',a\mid o,l),$

whereas a cascaded system may factorize the relationship as

$p(o',a\mid o,l)=p(o'\mid o,l)\,p(a\mid o,o',l).$

An action-first system may instead use

$p(o',a\mid o,l)=p(a\mid o,l)\,p(o'\mid o,a,l).$

Thus, a future may be generated before action selection, predicted as the consequence of a proposed action, or jointly generated with the action. The essential requirement is that the future remains action-facing (Wang et al., 12 May 2026).

The term “World Action Model” is broader than “Video Action Model.” Video is one possible world-modeling modality, but future WAMs may predict tactile observations, forces, depth, point clouds, optical flow, 3-D geometry, semantic masks, audio latents, proprioceptive states or learned physical representations. AquaWAM, for example, predicts compact underwater physical states rather than future images, including velocity, angular motion, depth, arm configuration, object pose and disturbances such as drift (Zhu et al., 27 Sep 2026).

2. Taxonomy of architectures and predictive substrates

Two complementary taxonomies organize WAMs. The first classifies what the model generates; the second analyzes predictive substrate, action coupling, backbone and deployment regime.

Cascaded WAMs

Cascaded WAMs explicitly separate world prediction from action generation:

$(o,l)\longrightarrow \text{future state or visual plan} \longrightarrow \text{action decoder} \longrightarrow a.$

The world model may generate RGB video, RGB-D sequences, optical flow, pose trajectories, point maps, semantic masks, affordance maps or latent futures. An inverse-dynamics model, trajectory optimizer, policy or geometric retargeting module then converts the predicted future into executable actions.

This design gives the world model a specialized predictive role and the action module a specialized inverse-dynamics or robot-kinematics role. Its principal disadvantages are error propagation, representation mismatch, latency and the possibility that a future is visually plausible but unreachable. Representative methods include UniPi, VLP, RoboEnvision, ThisThat, TesserAct, MVISTA-4D, Vidar, Gen2Act, Veo-Act, VAG, $\pi_{0.7}$ , NovaFlow, Dream2Flow and TraceGen (Wang et al., 12 May 2026).

Cascaded systems may perform explicit planning in pixel or structured visual space, implicit planning in latent space, or geometric action extraction. Optical-flow systems convert predicted displacement into trajectories; pose-based systems track objects, tools or hands and may use inverse kinematics; latent systems decode actions from VAE representations, intermediate denoising states, semantic masks or predictive features.

Joint WAMs

Joint WAMs model future states and actions within one predictive system:

$l$ 0

Autoregressive systems serialize visual tokens, latent states and action tokens. Diffusion and flow-based systems denoise future world and action variables, either in a unified stream or through coupled streams. Multi-stream systems may connect video and action experts through cross-attention, hidden-state transfer or shared representations.

Unified streams offer parameter sharing and close synchronization between world prediction and action generation. Multi-stream architectures provide modality specialization, independent noise schedules and selective inference. Both face objective interference: visual losses may reward detailed appearance while action losses prefer a compact control-relevant representation (Shen et al., 18 Jun 2026).

DreamZero is a joint WAM based on a 14-billion-parameter autoregressive diffusion transformer initialized from Wan2.1-I2V-14B-480P. It jointly predicts video and action chunks and replaces predicted visual context with actual observations after execution, preventing indefinite accumulation of autoregressive visual errors. The system achieves approximately 7-Hz closed-loop control using two GB200 GPUs (Ye et al., 17 Feb 2026).

Auxiliary-prediction and representation-centered WAMs

Some WAMs use future prediction during training but remove explicit future generation at inference. Their objective has the form

$l$ 1

The predictive branch shapes the policy representation, while deployment uses only the action pathway. Fast-WAM and GeoSem-WAM exemplify this interpretation. GeoSem-WAM predicts future RGB, geometry and semantic representations during training, but does not generate future video or consume depth or semantic masks at test time. Its reported results are 98.55% average success on LIBERO, 92.52% on RoboTwin 2.0 and 95.4% average success in its real Franka evaluation (Ma et al., 2 Jun 2026).

LeWAM extends the representation-centered approach using frozen I-JEPA embeddings rather than VAE or video-diffusion latents. It predicts future embeddings in JEPA space and generates actions from the same predictor, with future queries discarded at inference. With 0.4 billion trainable parameters, LeWAM reports 92.28% average success on RoboTwin 2.0 (Fang et al., 23 Sep 2026).

Modality-structured WAMs

ModAR autoregressively denoises multiple future modalities before actions. Its default order is

$l$ 2

Point tracks provide motion and correspondence, DINO features provide semantic and scene structure, depth provides geometry, and RGB provides appearance. At $l$ 3 demonstrations, ModAR achieves 75% success, compared with 72% for Flex- $l$ 4, while using approximately 20 times fewer training FLOPs and no video pretraining. Its reported real-world average success on three bimanual tasks is 83.3% (Hung et al., 15 Sep 2026).

SA-WAM jointly predicts actions, future RGB and future depth latents in a shared diffusion sequence. It encodes metric depth through camera-specific clipping and log-scale normalization into the bounded domain expected by a frozen RGB VAE. SA-WAM reports 76.6% on RoboCasa, 86.6% on LIBERO-Plus, and 90.0% clean and 77.5% randomized success on its real UR5 evaluation (Gonzalez et al., 2 Sep 2026).

3. Training objectives, representations and data

WAM training combines action imitation with predictive objectives. Common terms include action likelihood or flow matching, future-state prediction, inverse dynamics, forward dynamics, representation alignment, value prediction and auxiliary perception losses.

DreamZero uses a teacher-forced, chunk-wise flow-matching objective. Video and normalized action chunks are independently corrupted, and the transformer predicts their joint velocity. Previous chunks are supplied as clean context. DreamZero-Flash decouples video and action noise schedules, keeping video much noisier while denoising actions normally. On table bussing, its one-step configuration reaches 74% task progress at 150 ms, compared with 52% for one-step standard DreamZero and 83% for four-step standard DreamZero (Ye et al., 17 Feb 2026).

A different WAM formulation augments DreamerV2 with an inverse-dynamics head. From consecutive encoder embeddings,

$l$ 5

the model predicts the action causing the transition. Its objective combines KL regularization, image reconstruction and $l$ 6 action prediction. On eight CALVIN tasks, the detailed behavioral-cloning table reports an improvement from 45.8% to 61.7%, while the abstract reports 59.4% to 71.2%; these figures are internally inconsistent. The reported PPO averages are 92.8% for WAM and 79.8% for the baseline, with two tasks reaching 100% (Han et al., 30 Mar 2026).

Structured predictive supervision is increasingly used to make latent states spatially and semantically useful. GeoSem-WAM adds DPT-style geometry and semantic branches to future RGB prediction. Magic-W0 separates a Structured World Transition into Current 3D Geometry, action-induced 3D Motion and Future Semantics, with repeated bidirectional interaction between world streams and continuous action hypotheses (Chen et al., 30 Sep 2026).

The data ecosystem combines several sources:

Robot teleoperation provides synchronized observations, actions, proprioception and embodiment-specific control, but is expensive and relatively limited.
Portable and wearable human demonstrations provide diverse object interactions and hand motion, but introduce morphology, camera and force-correspondence gaps.
Simulation provides exact state, pose, segmentation, collision and sometimes tactile information at scale, but introduces the sim-to-real gap.
Internet-scale egocentric and instructional video provides broad motion, object-interaction and language–video priors but generally lacks executable robot actions.
Actionless video can supervise future representations, visual dynamics, tracks, semantics and geometry without requiring robot action labels.

DreamZero reports cross-embodiment gains from video-only demonstrations: human or YAM robot videos collected in approximately 10–20 minutes produce more than 42% relative improvement on unseen-task performance. ModAR likewise improves from 70.0% with robot demonstrations alone to 81.1% with 200 in-domain human videos and 83.3% with an additional 1,000 out-of-domain EgoDex videos (Ye et al., 17 Feb 2026, Hung et al., 15 Sep 2026).

4. Inference, planning horizons and control regimes

WAM deployment ranges from open-loop execution to single-step closed-loop control. Open-loop systems generate a trajectory once and execute it, but cannot correct deviations. Chunked closed-loop systems predict an action chunk, execute part of it, reobserve and replan. Single-step systems maximize responsiveness but require sufficiently fast inference.

A typical receding-horizon loop is

$l$ 7

The distinction between prediction horizon and commitment horizon is central. GameWAM predicts $l$ 8 native keyboard–mouse actions and executes $l$ 9, preserving look-ahead while receiving feedback twice as frequently as full-plan execution. It reports 50.7% Mini and 46.6% All success on its Minecraft benchmark, with 138, 155 and 203 average native actions for embodied, GUI and combat categories respectively (Guo et al., 25 Aug 2026).

DualWAM separates global planning and local refinement along a shared denoising trajectory. System 2 denoises a broad 32-frame global-wrist-action trajectory at high noise; System 1 extracts an 8-frame wrist-action window and refines it at low noise using fresh wrist observations. The systems run asynchronously. DualWAM reports 54% success on Franka and 45% on Galbot, with a 63-ms critical-path latency compared with 1043 ms for Cosmos3-Nano-Policy, a reported 16.6-times speedup (Zheng et al., 21 Sep 2026).

ProWAM occupies an intermediate point on the imagination spectrum. Rather than generating dense video or only a final frame, it predicts an ordered sequence of sparse visual sub-goals indexed by relative progress. A single video-backbone pass caches sub-goal features, after which lightweight action denoising produces actions. The reported results include 85.8% on LIBERO-Plus, 75.7% on randomized RoboTwin, 48.1% on RoboCasa365 and 70.0% in zero-shot real-world experiments (Zhang et al., 1 Oct 2026).

AquaWAM demonstrates that model-predictive WAMs need not use visual futures. It predicts 19- or 35-dimensional underwater physical states, evaluates 128 candidate action sequences over a two-second horizon and executes the first 0.5 seconds. The model represents thruster dead band, inertial glide, buoyancy, drag, ambient currents and actuator variation. It achieves 72.6% success across 20 USIM tasks and makes complete action decisions 2.7 times faster than U0 on an NVIDIA Jetson AGX Orin (Zhu et al., 27 Sep 2026).

5. Evaluation, empirical findings and design trade-offs

WAM evaluation spans predictive fidelity, physical plausibility, action plausibility, task success, robustness, latency and memory. Pixel metrics such as PSNR, SSIM, LPIPS and FVD evaluate visual similarity or temporal statistics, but do not establish whether a predicted future is executable. Physical benchmarks examine inertia, collision order, material behavior, object persistence and trajectory plausibility. Closed-loop task success remains the strongest direct measure of action utility.

The empirical literature does not identify a universally optimal causal structure. A controlled study compares Unconditional, Video-to-Action, Action-to-Video, Bidirectional, Joint and Causally Interleaved structures. On LIBERO-Plus, allowing generated futures to influence actions improves matched models by 16.14 and 14.93 percentage points, while the same route-enabled models can be worse on in-distribution RoboCasa-GR1 by 2.33 and 4.00 points. Direct interventions show that temporal reversal affects actions and success much more than future-content corruption, suggesting that temporal organization can matter more than exact future content (Tang et al., 21 Sep 2026).

The same study finds that inter-frame visual representations are stronger on familiar in-distribution data, whereas framewise representations are more robust under viewpoint, sensor-noise and other distribution shifts. On DROID, Causally Interleaved obtains lower MSE and L1 and higher [email protected] and [email protected] than Unconditional. DINOv3 outperforms DA3 in the reported latent comparison, and BC plus video generation outperforms BC-only (Tang et al., 21 Sep 2026).

Several recurring trade-offs define WAM design:

Design axis	Main advantage	Main limitation
Rendered future versus latent future	Interpretability and visual richness	Compute, memory and visual drift
Joint generation versus cascade	Tight world–action coupling	Objective interference and optimization complexity
Autoregressive versus diffusion/flow	Causal histories versus multimodal action chunks	Serial latency versus iterative denoising cost
Long horizon versus short horizon	Global coherence versus responsiveness	Stale actions versus expensive replanning
RGB versus structured state	Appearance and semantic breadth versus direct control relevance	RGB can model irrelevant detail; structured states require supervision
Inter-frame versus framewise features	Strong temporal priors versus adaptability under shift	Fixed temporal relations can become brittle
Explicit imagination versus auxiliary prediction	Inspectable future plans versus efficient deployment	Test-time imagination is expensive; auxiliary prediction may be ignored

Reported results across several systems support the value of structured predictive information, but comparisons often differ in model size, pretraining, action representation, data mixture and evaluation protocol. For example, ModAR’s comparison with Flex- $p(a\mid o,l),$ 0 is not fully controlled, and GameWAM’s baseline comparisons differ in training scale and action abstractions. Such results support design hypotheses rather than universal rankings.

6. Limitations, controversies and future directions

A central controversy concerns whether WAMs require explicit future imagination during inference. DreamZero uses future video directly in a closed loop, whereas GeoSem-WAM, Fast-WAM and LeWAM argue that predictive training can improve representations without test-time future-video generation. ProWAM retains sparse visual sub-goals as an inference-time planning interface. The evidence therefore supports multiple regimes rather than a single answer: future prediction may function as explicit planning, latent action guidance or auxiliary representation learning.

Visual realism is not equivalent to action usefulness. A video may be perceptually convincing while violating contact geometry, reachability, friction, force or kinematic constraints. Conversely, a compact physical or latent state may lack visual realism while supporting successful control. This motivates evaluation that jointly reports task success, physical consistency, action accuracy, latency, memory, horizon and failure modes.

Inverse dynamics is inherently ambiguous. Multiple actions may produce similar visual transitions, and small visual errors may imply substantially different motor commands. Hidden state, tactile information, forces, compliance and contact dynamics can be visually underdetermined. AquaWAM addresses this by predicting compact physical variables; Magic-W0 introduces 3-D geometry and motion; SA-WAM adds depth; GeoSem-WAM adds geometry and semantics; OmniVTA and related systems add tactile or visual-tactile futures.

Long-horizon memory remains unresolved. Current systems use observation replacement, bounded history, persistent latent states, sparse temporal memory, hierarchical history or external memory. GameWAM retains recent observations densely and summarizes older cycles; DualWAM uses temporal realignment to prevent stale global plans from controlling expired windows. Longer context alone does not guarantee coherent task-state tracking.

Computation is a persistent constraint. DreamZero requires two GB200 GPUs for approximately 7-Hz control; SA-WAM uses a 2-billion-parameter DiT; Magic-W0 recomputes interacting streams at every flow step; ProWAM reduces video-backbone evaluations through caching; DualWAM separates high-noise planning from low-noise refinement; LeWAM avoids a video-diffusion backbone and reports 31.90-ms graph-enabled latency with 1.988 GiB peak memory in its evaluation.

Data and embodiment dependence also constrain generalization. Human and internet videos provide broad physical priors but lack robot actions, forces and embodiment-specific kinematics. Few-shot transfer is easiest when source and target embodiments are visually and mechanically related. Physical-state WAMs can be highly efficient but are more task- and embodiment-specific than visual WAMs.

Future research directions include multimodal physical representations combining vision, depth, tactile signals, forces, audio, proprioception and 3-D geometry; action-conditioned and uncertainty-aware prediction; adaptive horizons and denoising budgets; structured long-term memory; causal and counterfactual evaluation; safer imagined-rollout optimization; embodiment-aware transfer; interactive learning from real-world feedback; and methods that preserve action-relevant predictive information while generating less of the future.

The emerging design principle is therefore not to maximize visual generation. It is to predict the coarsest, most invariant and most physically grounded future representation that still supports reliable action selection. In this sense, WAM research is moving from “observe and react” toward “predict, evaluate and act,” while increasingly compressing prediction into the geometry, motion, semantics, physical state and temporal structure that control actually requires.