WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory

Abstract: World Action Models (WAMs) offer a promising paradigm for robotic manipulation by jointly modeling visual state transitions and robot actions. However, existing WAMs are constrained by limited temporal context, coarse episode-level language supervision, and predominantly text-only conditioning, which hinder task-progress tracking and fine-grained language-video-action grounding while limiting visual-context reasoning and cross-embodiment transfer. In this paper, we introduce WorldScape Policy 2.0, a controllable WAM with reasoning-augmented long short-term memory. Its causal short-term visual memory supplies recent observations as DiT prefill to preserve local interaction dynamics, while its long short-term event memory organizes historical VLM outputs into global-history, local-active, and event-boundary representations for progress-aware retrieval. The retrieved history augments perception and autoregressively generated planning tokens, yielding an implicit subgoal condition for autonomous planning; semantic forcing further transfers event-level instruction semantics into this latent planning pathway. To establish fine-grained multimodal controllability, we construct ManipEvent-5M, an event-grounded embodied pretraining dataset containing nearly 5 million event segments with aligned action trajectories, episode-level task instructions, segment-level subtask captions, goal images, and video demonstrations. These designs provide a unified interface for autonomous planning from high-level instructions and controllable execution from fine-grained text, goal-image, or video-context prompts. Experiments in both simulation and real-world platforms demonstrate superior capabilities in long-horizon autonomous planning, fine-grained instruction following and in-context adaptation.

• The paper introduces a controllable World Action Model that combines progress-aware event memory, causal visual memory, and latent subgoal reasoning, achieving 94.3% average success on RoboTwin 2.0 and outperforming leading baselines.
• The paper presents ManipEvent-5M, a 4.89-million-segment dataset, and a three-stage training curriculum that transfers fine-grained event supervision into autonomous planning without requiring detailed instructions at inference time.
• The paper shows that memory improves robustness under distribution shift and supports real-world folding, shell-game, stacking, cleaning, and instruction-following tasks, while leaving open questions about event-boundary reliability and longer-horizon scaling.

WorldScape Policy 2.0 is a controllable World Action Model (WAM) for long-horizon robotic manipulation that couples progress-aware event memory, causal short-term visual memory, and latent subgoal reasoning within a unified video-action backbone (2607.18840). The paper identifies three limitations of existing WAMs: restricted temporal context that obscures task progress, coarse episode-level language supervision that weakens fine-grained grounding, and text-only conditioning that precludes visual prompting. The proposed system addresses all three through an architectural design, a new dataset (ManipEvent-5M), and a three-stage training curriculum.

Motivation and design insight

The central observation is that long-horizon controllability requires two complementary forms of temporal context: semantic event memory for reasoning about task progress, and frame-level visual memory for preserving local interaction dynamics. A tabletop scene may look nearly identical before and after an intermediate subgoal, so the correct next action depends on what has already been completed — information that neither a static observation nor a short history window provides. Prior work such as MemoryWAM expands access to full visual history but does not organize it around task progress or convert it into a latent subgoal condition for autonomous planning.

Architecture

The model predicts an $H$-step action chunk and $H$ future visual latents conditioned on the current multi-view observation, past actions, mode-dependent prompts, and memory state $\mathcal{M}_t$. Actions are represented as chunk-relative delta positions, 6D continuous relative rotations, and absolute gripper commands (10D per arm, 20D dual-arm). Embodiment-specific action encoders and decoders interface raw action chunks to a shared causal video-action DiT initialized from Wan2.2-5B; flow matching is performed directly in raw action space, with adapters serving only as input/output interfaces.

Reasoning-augmented long short-term memory operates at two levels. The long-term event-memory branch maintains a queue of historical chunk-level VLM outputs from a lightweight Qwen3-VL-4B backbone. Each entry combines perception tokens from a single VLM prefill and $K=4$ autoregressively generated planning tokens whose final-layer hidden states serve as latent planning features. Three complementary memory views are constructed: global-history latents (task-fused pooled history), local-active latents (the latest $S_e$ chunks), and event-boundary latents selected by cosine-distance changes between consecutive chunk representations via TopK selection with minimum temporal separation — requiring no explicit boundary annotations. Retrieved memory is fused into current reasoning tokens through cross-attention retrieval followed by a learned token-wise sigmoid gate. In parallel, short-term visual memory supplies up to 4 recent VAE-latent chunks as causal DiT prefill, alongside optional persistent goal-image or video-demonstration prompts retained across rollout steps under a causal attention mask.

Implicit subgoal planning via semantic forcing

In fine-grained instruction following, T5 embeddings of segment-level captions directly condition the DiT through cross-attention. In autonomous planning, the model receives only an episode-level instruction; the VLM branch produces memory-enhanced reasoning latents $\hat{q}_t$ that condition the DiT instead. During training, the fine-grained caption embedding acts as a fixed, stop-gradient semantic target, and a trainable projector aligns the reasoning summary to it via a cosine loss ($\lambda_s = 0.001$). This "semantic forcing" transfers event-level supervision into the latent planning pathway without exposing fine-grained instructions at inference time. Fixing the target encoder prevents collapse toward a degenerate joint solution.

ManipEvent-5M

The dataset aggregates human-arm egocentric videos, robot-free UMI data, simulated trajectories, and real-robot demonstrations into roughly 4.89 million event segments spanning 512M frames and about 4,983 hours across 744K episodes. Annotation uses Qwen3-VL-32B in a four-stage pipeline: two-stage temporal-semantic labeling (motion/gripper-signal boundaries followed by open-vocabulary relabeling), hierarchical episode/event frame sampling with hindsight task descriptions, embodied prompt conditioning specifying acting end effector, object states, contact mode, and failure recovery, and multi-view conflict resolution prioritizing gripper-view over head-view evidence. Goal-image prompts include first-view terminal observations and third-view images sampled from matched human demonstrations; video prompts pair human UMI demonstrations with robot trajectories matched by task, event sequence, and outcome for cross-embodiment transfer.

Training follows three stages: (1) event-grounded multimodal WAM pretraining establishing controllability and causal visual memory; (2) memory-aware mid-training introducing the VLM branch, event memory, and semantic forcing; (3) downstream post-training per interaction mode.

Results

On RoboTwin 2.0 (50 tasks, 100 trials each, clean-plus-randomized training), WorldScape Policy 2.0 achieves 94.3% average success (94.3% clean / 94.2% randomized), exceeding $\pi_{0.5}$ by 14.5 points on average and surpassing recent natively pretrained WAMs including LingBot-VA 2.0 (93.6%) and Abot-M0.5 (94.1%). Under the stricter Clean-to-Randomized setting (clean-only training), it reaches 47.9% average success, outperforming $\pi_0$, $\pi_{0.5}$, and Fast-WAM by 16.5, 10.4, and 8.8 points respectively — indicating that much of its robustness derives from the training curriculum rather than exposure to randomized data.

Real-world evaluation on a dual-arm PiPER platform covers five tasks with 20 trials each. The model achieves 75% on both long-horizon folding tasks, 75% on the memory-dependent shell game, 60–70% on cross-embodiment block stacking with goal-image or video prompts, and 80% on sequential table cleaning, consistently above $H$0 and DreamZero. On atomic instruction following, it attains 86.7% in-domain and 60.0% out-of-domain success versus 43.3% ($H$1) and 30.0% (DreamZero) OOD, supporting the claim that event-level pretraining improves compositional generalization to held-out objects.

Ablations isolate component contributions under clean-only training. Adding short-term visual memory improves average success from 40.91% to 44.67%; long-term event memory raises it to 46.25%; latent subgoal reasoning reaches 47.89%. Separately, Stage-1 pretraining contributes the largest gain (43.19% → 46.63%), with Stage-2 mid-training and semantic forcing adding further consistent improvements. Notably, the largest single effect is the randomized-setting gap between no-memory and full-memory variants (17.22% → 26.03%), suggesting memory is most valuable under distribution shift.

Limitations and open questions

Several caveats bear on the reported results. The ablation protocol differs from the headline comparison (clean-only versus clean-plus-randomized training), so absolute ablation scores are not comparable to the main benchmark numbers, and the paper explicitly warns against direct comparison. Several public datasets (RoboMIND, RoboCOIN, DROID) lack event decomposition and enter pretraining as single-segment episodes, meaning a substantial fraction of the corpus does not carry the fine-grained supervision the method is built around. Event-boundary detection relies on cosine distance over VLM representations with hand-tuned temporal separation, and the paper does not evaluate sensitivity to these hyperparameters or validate boundary quality against ground truth. Semantic forcing uses a single fixed planning-prompt template, leaving open whether the latent subgoal pathway generalizes across prompt phrasings. Finally, real-world evaluation is limited to five tasks on one platform with 20 trials per task, so statistical confidence on the visual-reasoning and cross-embodiment results is modest; whether the memory mechanism scales to substantially longer horizons than those tested remains unexamined.

Conclusion

WorldScape Policy 2.0 demonstrates that combining progress-aware event memory, causal visual memory, and semantically forced latent subgoal reasoning within a jointly pretrained video-action model yields measurable gains in long-horizon planning, fine-grained instruction following, and visual-prompted adaptation, supported by ManipEvent-5M's event-grounded supervision. The ablations confirm that each memory component contributes independently, with the strongest effects under distribution shift. The main open questions concern the robustness of unsupervised event-boundary discovery, generalization of the fixed planning template, and validation at longer horizons and across more embodiments.