World Action Models & Predictive Control

• World Action Models are predictive-action systems that integrate future-state modeling with action generation to enable robust, control-relevant representations.
• They leverage diverse substrates such as RGB, latent spaces, and geometric features to balance visual fidelity and computational efficiency.
• WAMs are pivotal in robotics and autonomous driving, addressing challenges in coupling prediction with action and managing latency and resource trade-offs.

Searching arXiv for papers on World Action Models to ground the article with current references. World Action Models (WAMs) are embodied predictive-action models that couple future-state modeling with action generation. In the broadest formulation, they extend action-conditioned world models—predictors of future observations or states under intervention—by ensuring that the predicted future is usable for control rather than forecasting alone. A one-step world model can be written as $y_{t+1} \sim p_\theta(\cdot \mid o_{0:t}, a_t)$, whereas a WAM explicitly couples future prediction and action production, for example through $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$ or, more generally, $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$ (Zhang et al., 1 Jul 2026, Shen et al., 18 Jun 2026). Across robotics and autonomous driving, WAMs have emerged as a unifying paradigm for bringing predictive physical priors into control, while simultaneously exposing a central design tension: the richer the future representation, the greater the burden in compute, memory, latency, and action-label cost (Wang et al., 12 May 2026).

1. Conceptual scope and relation to adjacent model families

A central contribution of recent tutorials and surveys is terminological disambiguation. Standard Vision-Language-Action (VLA) policies optimize direct action prediction, typically $p_\theta(a\mid o,l)$, without requiring an explicit predictive commitment about how the world evolves. World models optimize future prediction conditioned on action, such as $p_\theta(o'\mid o,a,l)$, but need not produce actions. WAMs occupy the intermediate position: they are predictive-action systems in which forecasted futures are retained in the action loop, whether to produce, decode, score, or train the action (Shen et al., 18 Jun 2026, Wang et al., 12 May 2026).

This boundary is not entirely uniform across the literature. One survey adopts a strict criterion: a model is a WAM only when the predicted future stays in the action path; a world model used only for simulation or as an auxiliary loss is not counted as a WAM (Shen et al., 18 Jun 2026). By contrast, the robotics tutorial on world action models includes “auxiliary video prediction for policy learning” among the four representative paradigms of WAMs, reflecting a broader, policy-oriented usage in which predictive supervision may be sufficient if it shapes control-relevant representations (Zhang et al., 1 Jul 2026). This definitional divergence is itself part of the field’s maturation.

The distinction from video generation models is equally important. A video generator learns to render plausible futures from prompts or conditioning signals, but plausibility alone does not guarantee control relevance. Recent work explicitly warns that “a visually plausible predicted future is not the same thing as an action-readable future,” identifying a representational mismatch between reconstruction-optimized video features and low-level action decoding (Qiu et al., 10 Jun 2026). This has made the world-action interface—not merely the generator backbone—a first-class research object.

2. Predictive substrates and representational design space

The current design space can be organized first by the substrate in which prediction occurs. The robotics tutorial distinguishes observation-space world models, which predict future observations directly, from state-space world models, which predict structured task states (Zhang et al., 1 Jul 2026). Observation-space models include RGB, multi-view RGB, RGB-D, and point-cloud futures. Their main advantage is visual fidelity and scalability, particularly for ordinary RGB/video, but they can be costly to generate and less physically interpretable when they remain in pixel space. State-space models instead target latent states, point tracks, neural-symbolic predicates, or physical states such as object poses, velocities, contacts, and friction; these representations trade photorealism for compactness, interpretability, and direct control utility (Zhang et al., 1 Jul 2026).

A complementary survey taxonomy asks what the method is required to generate: rendered futures, latent futures, or no video generation at all (Shen et al., 18 Jun 2026). A second survey frames the same territory as Cascaded WAMs and Joint WAMs, with further subdivisions by generation modality, conditioning mechanism, and action decoding strategy (Wang et al., 12 May 2026). Taken together, these taxonomies make clear that “world” in WAM need not mean raw RGB video; it can denote flow, point tracks, depth, geometric state, affordance maps, teacher embeddings, latent action codes, or other predictive substrates.

Representation split	Typical target	Main trade-off
Observation-space	RGB, multi-view RGB, RGB-D, point clouds	Visual realism and scalability vs. cost and weaker physical interpretability
State-space	Latents, point tracks, predicates, physical states	Compactness and control utility vs. stronger modeling assumptions
Video-generation-free	JEPA features, teacher embeddings, geometry, latent actions	Efficiency and abstraction vs. reduced direct visual grounding

Several recent systems make these trade-offs concrete. JOPAT augments latent visual prediction with 2D point tracks and visibility, arguing that tracks provide explicit motion correspondences, robustness to occlusion, and object permanence over time; on LIBERO it reports $97.8\%$ average success rate, with the largest gains on LIBERO-Long (Guan et al., 22 May 2026). OA-WAM replaces holistic futures with object-addressable slot states, each decomposed into a persistent address vector and a time-varying content vector, and reports $97.8\%$ on LIBERO, $79.3\%$ on SimplerEnv, and a swap-binding cosine of $0.87$ versus at most $0.09$ for holistic baselines (Liu et al., 7 May 2026). These results support a broader trend already emphasized by surveys: the field is moving toward generating less of the future while preserving exactly what control requires (Shen et al., 18 Jun 2026).

3. Coupling future prediction to action

The canonical WAM question is not only what future to represent, but how that future enters action generation. A concise factorization is the predict-then-act cascade,

$$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$0

which decouples forecasting from control. The stricter survey taxonomy also highlights action-conditioned rollout,

$$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$1

and joint future-action prediction under a single model (Shen et al., 18 Jun 2026).

The robotics tutorial gives four representative paradigms of WAMs in robotics (Zhang et al., 1 Jul 2026).

Paradigm	Core mechanism	Principal trade-off
Imagine-then-execute	Generate a future subgoal, then decode actions via inverse dynamics or goal-conditioned policy	Interpretable and modular, but heavily dependent on imagined-future quality
Video-feature-conditioned action prediction	Use internal spatiotemporal video features as policy conditioning	Lower inference cost, but less transparent future-action coupling
Joint video-action modeling	Learn a unified distribution over future observations and action sequences	High consistency, but expensive and action-label dependent
Auxiliary video prediction for policy learning	Use future prediction as an auxiliary objective during policy training	Efficient at test time, but the future is not an explicit execution-time plan

Much of the architectural literature can be understood as different instantiations of these patterns. WorldVLA places action tokens and image tokens into a single autoregressive model and trains a weighted joint objective $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$2 with $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$3; it reports that world-model pretraining improves average success from $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$4 to $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$5, while a specialized action attention mask improves grasping success by approximately $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$6 to $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$7 relative to naïve autoregressive chunking (Cen et al., 26 Jun 2025). DreamZero, by contrast, treats WAMs as joint video-action predictors built on a pretrained video diffusion backbone and argues that better video prediction leads directly to better robot behavior; it reports real-time closed-loop control at $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$8 Hz and more than $$(o_{t+1:t+H}, a_{t:t+H-1}) \sim p_\psi(\cdot \mid o_t, l)$$9 improvement in generalization to new tasks and environments relative to state-of-the-art VLA baselines in its real-robot experiments (Ye et al., 17 Feb 2026).

A recurrent misconception is that the strongest WAM must always decode full future video at inference. Several recent systems explicitly reject that premise. GeoSem-WAM preserves the WAM training signal but “does not do explicit future rollout at test time,” instead using geometry and semantic prediction branches only during training to improve the shared latent space (Ma et al., 2 Jun 2026). GigaWorld-Policy likewise makes future-video generation optional at inference by preventing future-video tokens from influencing action tokens through a blockwise causal self-attention mask (Ye et al., 18 Mar 2026).

4. Representation learning, action-grounding, and memory

As WAMs have matured, the dominant research problem has shifted from backbone choice to representation shaping. AGRA identifies an “action-grounding gap”: video diffusion features optimized for reconstruction may attend to background clutter or irrelevant hands rather than to the actual hand-object interaction site. It introduces Action-Grounded Representation Alignment, aligning selected intermediate video-model features with spatially coherent DINOv2 features, and reports an increase from $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$0 to $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$1 success on real-world manipulation, together with $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$2, $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$3, and $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$4 gains under semantic, instance-level, and attribute generalization settings, respectively (Qiu et al., 10 Jun 2026).

GeoSem-WAM reaches a similar conclusion from a different angle. It argues that existing WAMs work largely because predictive supervision improves latent representations during training rather than because explicit imagination is required at test time. It adds future geometry and future semantic supervision alongside RGB prediction, keeping inference efficient by discarding those dense prediction heads at deployment. On LIBERO it reports an average success rate of $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$5 versus $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$6 for Fast-WAM; on RoboTwin 2.0, $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$7 versus $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$8; and on real Franka tasks, $$p_\Theta(s_{t+1:t+H}, a_{t:t+H-1}\mid o_{\le t}, a_{<t}, l)$$9 versus $p_\theta(a\mid o,l)$0 (Ma et al., 2 Jun 2026).

Other work makes the predictive substrate itself more control legible. JOPAT adds point tracks and visibility to expose motion correspondences explicitly, while OA-WAM enforces identity-preserving object addressability through address-only attention keys and per-layer address resetting (Guan et al., 22 May 2026, Liu et al., 7 May 2026). DiLA attacks the abstraction–fidelity trade-off in latent action models by splitting structure from content and learning a continuous latent action space; on VP$p_\theta(a\mid o,l)$1, it reports aggregate success $p_\theta(a\mid o,l)$2 versus $p_\theta(a\mid o,l)$3 for AdaWorld, while also improving video quality on SSv2 and RT-1 (Zhang et al., 15 May 2026). LAWM extends latent-action world modeling to mixtures of action-conditioned and action-free trajectories, aligning observed controls with inferred latent actions and reporting an average normalized return of $p_\theta(a\mid o,l)$4 with only $p_\theta(a\mid o,l)$5 action-labeled data, close to the $p_\theta(a\mid o,l)$6 oracle C-LAP result (Alles et al., 10 Dec 2025).

Long-horizon control has made memory another core axis. DiM-WAM augments a base WAM with multi-bank historical event memory and a progress-supervision objective so that the model conditions video and action denoising on recent local context, cross-stage historical events, and global task progress. On RMBench it raises average success from $p_\theta(a\mid o,l)$7 with LingBot-VA to $p_\theta(a\mid o,l)$8, exceeding Mem-0 at $p_\theta(a\mid o,l)$9; on four real-world Franka tasks it improves average stage success from $p_\theta(o'\mid o,a,l)$0 to $p_\theta(o'\mid o,a,l)$1 and full-task success from $p_\theta(o'\mid o,a,l)$2 to $p_\theta(o'\mid o,a,l)$3 (Wang et al., 26 Jun 2026).

A parallel development in autonomous driving reaches similar conclusions about intermediate features. ReWorld treats internal representations of both Video DiT and Action DiT as direct optimization targets rather than byproducts, adding future-predictive supervision, cross-modal alignment, and hard-negative safety shaping. On nuScenes it reduces FVD from $p_\theta(o'\mid o,a,l)$4 to $p_\theta(o'\mid o,a,l)$5; on NAVSIM it raises PDMS from $p_\theta(o'\mid o,a,l)$6 to $p_\theta(o'\mid o,a,l)$7 without RL or post-processing and accelerates from-scratch convergence by approximately $p_\theta(o'\mid o,a,l)$8 (Xia et al., 25 Jun 2026).

5. Temporal organization, efficiency, and deployment regimes

A consistent practical objection to WAMs is inference cost. The robustness study comparing WAMs and VLAs reports that, on the same device, $p_\theta(o'\mid o,a,l)$9 takes $97.8\%$0 ms, whereas GE-Act takes $97.8\%$1 ms, Cosmos-Policy $97.8\%$2 ms, LingBot-VA $97.8\%$3 ms in a real-world setting and $97.8\%$4 ms in the RoboTwin setting, and MOTUS $97.8\%$5 ms; the surveyed WAMs are at least $97.8\%$6 slower per inference than $97.8\%$7 (Zhang et al., 23 Mar 2026). This has driven a wave of action-centered and asynchronous designs.

GigaWorld-Policy responds by making future video a training-time supervisory signal rather than a mandatory runtime path. Initialized from Wan 2.2 5B and pretrained on an embodied corpus totaling about $97.8\%$8 hours, it reports $97.8\%$9 ms inference latency versus $97.8\%$0 ms for Motus, approximately a $97.8\%$1 speedup, with real-world average success rate $97.8\%$2 versus $97.8\%$3 for Motus and $97.8\%$4 for $97.8\%$5 (Ye et al., 18 Mar 2026). AHA-WAM pushes the same direction further by decoupling a low-frequency Video DiT “world planner” from a high-frequency Action DiT executor, adding horizon-adaptive offset training and Observation-Guided Video-Context Routing. It reports $97.8\%$6 average success on RoboTwin, $97.8\%$7 across four real-world tasks, $97.8\%$8 Hz closed-loop control, and a $97.8\%$9 speedup over Fast-WAM; its distilled variant reaches $79.3\%$0 Hz (Cai et al., 8 Jun 2026).

MotionWAM generalizes the efficiency problem from tabletop manipulation to humanoid loco-manipulation. It conditions a Motion DiT on intermediate denoising features of a Video DiT rather than on fully denoised futures, uses a unified motion latent instead of an upper–lower hierarchical split, and reports $79.3\%$1 overall success versus $79.3\%$2 for the strongest VLA baseline, GR00T-N1.7, across nine real-world Unitree G1 tasks (Zheng et al., 8 Jun 2026). HarmoWAM offers another deployment compromise: it combines a predictive expert for precise interactions with a reactive expert for generalized transit, then switches between them with a Process-Adaptive Gating Mechanism. On six real-world Franka tasks and three training-unseen OOD settings, it reports $79.3\%$3 in-domain average success and an overall OOD average of $79.3\%$4, significantly outperforming prior VLA models and WAMs by margins of $79.3\%$5 and $79.3\%$6, respectively (Feng et al., 11 May 2026).

These systems illustrate an increasingly explicit deployment principle already visible in survey accounts: the most useful WAM is often not the model that dreams the most, but the one that preserves only the predictive substrate that control still needs at execution time (Shen et al., 18 Jun 2026, Wang et al., 12 May 2026).

6. Empirical landscape, misconceptions, and open problems

Empirically, WAMs are strongest where robustness to perturbation, physical change, or weak supervision matters. On RoboTwin 2.0-Plus, the comparative robustness study reports LingBot-VA at $79.3\%$7 total success, with $79.3\%$8 under lighting perturbations, $79.3\%$9 under noise, and $0.87$0 under layout; on LIBERO-Plus, Cosmos-Policy achieves $0.87$1 and GE-Act $0.87$2 (Zhang et al., 23 Mar 2026). Yet the same study shows that WAMs are not universally superior: $0.87$3 reaches $0.87$4 on LIBERO-Plus and can match or exceed WAMs in some settings when trained with very diverse robotic datasets, web-scale data, multiple learning objectives, and strong geometric grounding (Zhang et al., 23 Mar 2026). The appropriate conclusion is therefore conditional rather than absolute: WAMs often generalize better to visual perturbations because they exploit spatiotemporal dynamics priors, but strong VLAs can be competitive given sufficient data and training sophistication.

Another widespread misconception is that predictive quality and control quality are interchangeable. AGRA shows that action decoders can remain sensitive to task-irrelevant regions even when futures look plausible (Qiu et al., 10 Jun 2026). ReWorld shows that generation improvements do not automatically translate into planning improvements unless action representations are world-grounded and safety-discriminative (Xia et al., 25 Jun 2026). The surveys accordingly distinguish visual fidelity metrics such as FVD, FID, LPIPS, PSNR, SSIM, and DreamSim from closed-loop policy benchmarks and physical plausibility criteria, and argue that evaluation must jointly report success, latency, memory, horizon length, failure tags, and compute budget (Shen et al., 18 Jun 2026, Wang et al., 12 May 2026).

Open problems remain consistent across the literature. Architectural coupling is unresolved: cascaded and joint designs both persist, and the optimal amount of explicit future generation is still task dependent (Wang et al., 12 May 2026). Long-horizon persistence remains difficult because repeated action–observation–replan cycles induce drift, forgetting, and memory blow-up (Shen et al., 18 Jun 2026). Multimodal physical grounding is incomplete, since many models remain RGB-centric despite the control relevance of proprioception, depth, tactile signals, contact, and force (Wang et al., 12 May 2026). Data-mixture design is poorly understood, especially how to assign internet video, portable human demonstrations, simulation, teleoperation, and synthetic trajectories to pretraining, alignment, and action learning stages (Shen et al., 18 Jun 2026). Finally, latency remains a defining systems bottleneck: the field’s broader trajectory is toward latent, geometric, sparse, addressable, or asynchronous futures that preserve action-relevant sufficiency while reducing the runtime burden of full video rollout (Zhang et al., 23 Mar 2026, Shen et al., 18 Jun 2026).

In this sense, World Action Models are best understood not as a single architecture but as a research program. Their unifying commitment is that embodied control should be conditioned by a predictive model of intervention-induced world evolution. Their central research problem is determining how much of that future must be represented, where it should enter the action pathway, and how to make it simultaneously causal, persistent, physically grounded, computationally tractable, and executable.