MemoryWAM: Efficient World Action Modeling with Persistent Memory

Abstract: Robust robotic manipulation in the real world requires not only an understanding of the current observation, but also memory and dynamics modeling. World action models (WAMs) possess these capabilities by jointly modeling visual foresight and actions conditioned on both current and historical observations, making them a promising paradigm for robotic manipulation. However, existing WAMs face a fundamental trade-off: methods with efficient inference typically condition only on a bounded window of recent observations and therefore struggle in non-Markovian environments, whereas methods that preserve long histories incur time and space costs that grow substantially with sequence length. To address this challenge, we introduce MemoryWAM, a world action model with efficient persistent memory. MemoryWAM uses a hybrid memory design that combines recent frames, event-boundary anchor frames, and compact gist tokens that summarize long-range history. A tailored attention mechanism enables retrieval of both detailed short-term context and compressed long-term context, supporting memory-dependent decision-making with reduced inference latency and GPU memory usage. Across long-horizon, memory-dependent manipulation tasks in both simulation and the real world, MemoryWAM outperforms strong vision-language-action (VLA) and WAM baselines while maintaining favorable computational efficiency.

• The paper introduces MemoryWAM, which combines short-term sliding windows, anchor frames, and gist tokens to efficiently retain long-range context.
• The methodology significantly reduces GPU memory and inference latency while maintaining high success rates in both simulated and real-world manipulation tasks.
• Empirical results demonstrate that MemoryWAM outperforms traditional RNN/TTT methods with success rates up to 100% in certain tasks and effective scaling for long-horizon challenges.

MemoryWAM: Efficient World Action Modeling with Persistent Memory

Introduction and Core Problem

World Action Models (WAMs) have become central in enabling long-horizon, memory-intensive robotic manipulation tasks by integrating dynamics-aware visual foresight with action prediction. However, a critical bottleneck in WAM deployment arises from the antagonism between efficient memory usage and persistent access to long-range temporal context. Sliding-window strategies provide low inference latency and bounded compute but forget temporally distant cues, resulting in catastrophic performance degradation on non-Markovian tasks. In contrast, approaches that cache the entire observation/action history offer full context retention but induce inference latency and GPU memory demands that scale linearly with trajectory length, severely limiting real-world operability.

MemoryWAM addresses this with a hybrid memory design, combining short-term fidelity, targeted long-term abstraction, and prioritized event-boundary retention. This approach draws formal inspiration from human cognitive architectures, which allocate distinct resources for recency, long-term abstraction, and event-state anchoring. The method fundamentally restructures the trade-off between contextual completeness and computation, enabling robust, memory-dependent control in both simulation and real-world settings.

Figure 1: MemoryWAM employs hybrid memory—short-term sliding window, anchor frames at event boundaries, and compressed gist tokens—for efficient scaling ($O(N/d)$) and persistent context, outperforming both bounded and full-history WAMs.

Methodology and System Design

Hybrid Memory Mechanism

MemoryWAM maintains a compact yet persistent cache by architecting its memory into three modules:

• Short-Term Memory: A sliding window cache of the most recent $N_{recent}$ visual frames, essential for capturing fine, rapidly-evolving interaction details.
• Anchor Frames: Select event-boundary frames (e.g., task onset frames) are retained in full fidelity, motivated by empirical findings that initial scene context often governs latent task constraints and may become occluded.
• Gist Tokens: For frames neither recent nor anchor, high-dimensional token sets are compressed aggressively into a small set of learnable "gist" vectors. At inference, only the gist tokens persist, while the full frame tokens are evicted. This achieves a compression rate $d = L / M$ (where $L$ is the number of tokens per frame; $M$ is the number of gist tokens, typically $M \ll L$), yielding sublinear temporal cache growth.

Given a video transformer backbone, the action generation transformer attends over this hybrid memory. The attention mask is block-sparse, ensuring that action tokens query both the recent full-resolution context, anchor events, and all long-term gists in a single operation. This sharply reduces the memory and computation required to deploy WAMs in long-horizon tasks.

Figure 2: The hybrid memory architecture leverages a Video DiT and Action DiT within a Mixture-of-Transformers (MoT) framework; persistent context is preserved via sliding windows, anchor frames, and gist tokens.

Empirical Evaluation

Memory Mechanism Comparisons

The paper benchmarks the hybrid memory against full attention, test-time training (TTT), and recurrent neural network (RNN) alternatives, analyzing both computational cost and policy optimality.

• Latency and GPU Memory: Full attention grows linearly in both latency and VRAM with longer trajectories. TTT and RNNs keep cost flat via aggressive state compression, but incur high baseline cost due to additional update steps and network complexity. The hybrid mechanism achieves efficiency competitive with and often superior to RNN/TTT, while avoiding linear scaling.
• Policy Success: On memory-critical tasks such as RMBench’s Press Button, only hybrid memory and full attention achieve high success rates ($87\%$), while RNN/TTT variants suffer due to insufficient detail retention.

  Figure 3: Latency, GPU usage, and task success comparison across memory mechanisms, evidencing that hybrid memory matches full context performance at substantially lower computational cost.

Simulation and Real-World Manipulation

In RMBench, which probes a diverse set of dual-arm, long-horizon tasks, MemoryWAM surpasses sliding-window and full-history strategies:

• RMBench Average Success Rate: MemoryWAM attains $83.0\%$, eclipsing prior full-history SOTA (LingBot-VA: $78.2\%$) and dramatically outperforming observation-window models (below $11\%$).
• Task-Wise Comparison: On tasks such as Rearrange Blocks and Swap Blocks, MemoryWAM achieves $N_{recent}$0 success, while window-limited baselines collapse.

In physical robot deployments (dual-arm ARX with RealSense vision), two memory-bound tasks are evaluated:

• Shell Game: MemoryWAM succeeds in $N_{recent}$1 cases versus $N_{recent}$2 for LingBot-VA and $N_{recent}$3 for window-only baselines. The difference is directly attributable to inference speed—MemoryWAM’s lower latency enables timely observation of cup swaps, preventing context loss.
• Look and Press: MemoryWAM succeeds in $N_{recent}$4, modestly outperforming LingBot-VA.

Ablation Studies

Three ablation variants (removing gist tokens, anchor frames, or sliding window) confirm:

• Gist tokens are critical: Their removal causes the largest average drop ($N_{recent}$5 vs $N_{recent}$6 hybrid).
• Anchor frames and recency are non-redundant: Both significantly degrade performance when omitted.
• Full attention is slightly suboptimal: Retaining all history can introduce distraction or redundancy, supporting the hypothesis that event-boundary-based memory abstraction is beneficial.

Theoretical and Practical Implications

MemoryWAM demonstrates that a hybrid, event-aware memory structure achieves near-optimal long-range dependency modeling while incurring sublinear computational costs. The design is robust against the shortcomings of sliding windows (context amnesia) and outperforms full-history caching by mitigating redundancy and focus drift. Practically, this enables tractable deployment of world action models in real-time and extended manipulation tasks, including settings where instantaneous inference is critical.

Theoretically, the hybrid scheme points towards a new family of WAM architectures that integrate schedule-based frame retention, context-based abstraction, and prioritized attention—all harmonized with classic results in human memory research. This structure is readily extensible to foundation models requiring context-dependent reasoning over long time horizons, such as continuous learning in embodied agents.

Future Directions

Outstanding questions include:

• Semantic Abstraction: Current gist tokenization may be further improved by fusing explicit symbolic knowledge or leveraging dual-system architectures for semantic reasoning.
• Learning Anchor Policies: Automating the selection of anchor frames/events using learned event segmentation algorithms or instruction-conditioned policies.
• Unified Multi-Task/Multimodal Models: Integrating this hybrid memory into multi-modal or generalist world models, further reducing the gap between world modeling and language/action interfaces.

Conclusion

MemoryWAM establishes that hybrid, cognitive-inspired memory designs enable high-fidelity, persistent context in world action models, facilitating efficient, robust long-horizon manipulation. By strategically abstracting, prioritizing, and compressing experience, MemoryWAM attains SOTA policy performance with practical deployment costs in both simulated and physical environments.

Reference: "MemoryWAM: Efficient World Action Modeling with Persistent Memory" (2606.20562)