Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Abstract: Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

• The paper introduces a dual latent memory framework that integrates both short-term and long-term memory tokens into vision-language-action models for robotic manipulation.
• It demonstrates improved success rates on benchmarks, outperforming state-of-the-art methods with gains up to 16.6 percentage points in temporal tasks.
• The approach enables temporally coherent action generation by embedding memory tokens directly into the model’s reasoning path.

Dual Latent Memory Integration for Temporally-Aware Vision-Language-Action Models

Introduction

The paper "Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation" (2607.07608) addresses a core limitation in contemporary Vision-Language-Action (VLA) models used for robotic manipulation: the temporal short-horizon bias induced by Markovian action policies. Most existing VLA models, including those augmented with episodic memory or external retrieval mechanisms, process historical context outside the model’s native latent reasoning space. This extra-model treatment impedes the effective integration of temporally relevant evidence into multimodal action formation, particularly in long-horizon or multi-step scenarios. The proposed LaMem-VLA framework introduces a paradigm shift by encoding historical robotic experience as context-native latent memory tokens, enabling direct and flexible memory integration within the VLA’s continuous reasoning sequence.

Architectural Contributions

LaMem-VLA operationalizes latent memory through four coordinated components:

1. Memory Curator: Encapsulates historical experience into two distinct but complementary vaults—short-term (visually dominant) and long-term (semantically dominant)—at each timestep. The short-term vault encodes compact representations of recent perceptual context, while the long-term vault archives action tokens summarizing semantic task progress.
2. Memory Seeker: Utilizes multimodal (visual and linguistic) cognitive state to formulate context-aware queries that retrieve the top-K relevant evidence from both memory vaults, ensuring retrieval is guided by task intent and situational awareness rather than raw observation.
3. Memory Condenser: Transforms potentially redundant retrieved content into fixed-length, context-sensitive latent memory tokens optimized for seamless integration into the VLA model’s embedding space. Compression prevents context bloat and preserves relevance.
4. Memory Weaver: Injects the dual-scale latent memory tokens into the joint embedding sequence, preceding the action query tokens. This method enables the memory, alongside the current observation and instruction, to participate in mutual self-attention and joint reasoning throughout the VLA backbone, rather than being appended as external conditioning.

The unified latent space for observation, instruction, action, and memory tokens enables temporally consistent reasoning and grounds action generation in context-rich temporal evidence.

Empirical Validation

SimplerEnv-Bridge Suite

LaMem-VLA was evaluated on the SimplerEnv-Bridge benchmark, geared towards testing real-to-sim transfer and temporal awareness in manipulation. The model achieved an average task success rate of 73.9%, outperforming strong baselines such as CogACT (+16.6 points), To (+4.7 points), and Memory VLA (+2.0 points). On individual tasks, the method consistently reached or surpassed 75% success in complex temporally-dependent settings, demonstrating the efficacy of latent-native dual memory integration.

LIBERO Benchmark

On the LIBERO suite, LaMem-VLA achieved 97.6% average success across five evaluation suites, registering improvements over the dominant policy-side memory method Memory VLA (+1.1 points) and CogACT (+4.4 points). Importantly, in long-horizon suites (Long-10 and Long-90), the framework attained 95.8% and 97.0% success respectively, confirming the system's ability to utilize long-range task progress and episode milestones in generating temporally coherent actions.

Ablation and Component Analysis

Selective ablation underscored the complementary necessity of short-term and long-term latent memory: removing either stream notably reduced performance (to 64.6–65.6% on SimplerEnv and 94.8–95.4% on LIBERO-90), and removing both reduced the model to near-baseline. Furthermore, policy-side memory conditioning yielded clear improvements over memory-free baselines but consistently underperformed compared to the full latent-native integration. Increasing the retrieval budget and latent token capacity improved performance up to a saturation threshold, beyond which benefits plateaued or declined because of redundancy and compression bottlenecks. These results validated the architectural design choices of dual-scale memory and bounded, compressed latent tokenization.

Theoretical and Practical Implications

LaMem-VLA fundamentally demonstrates that context-native latent memory provides a superior mechanism for integrating temporally structured evidence in VLA models. By directly involving memory tokens in the self-attention and reasoning pathway, the model captures both fine-grained perceptual changes and coarse-grained task progression without the context-length explosion typically associated with history expansion. Practically, this approach enhances robustness, progress awareness, and action reliability for temporally extended and multi-phase robotic manipulation.

On the theoretical front, LaMem-VLA's approach aligns with emerging trends in latent reasoning (Bai et al., 1 Feb 2026, Yu et al., 2 Apr 2026), supporting the hypothesis that memory should be architected as a generative, context-native faculty. It bridges the gap between memory-augmented reasoning and action synthesis, offering a foundation for investigating generative memory dynamics, memory consolidation strategies, and hierarchical task decomposition.

Future Directions

While the current instantiation of LaMem-VLA is limited to simulated evaluation, the approach is architecture-agnostic and scalable to real-world robot deployments once the corresponding integration with multimodal, continuous sensory streams and real-time constraints is achieved. Possible avenues of extension include:

• Incorporating cross-modal sensory memory (e.g., proprioceptive, haptic data) within the latent memory stream.
• Autonomously modulating the retrieval and condensation strategy via adaptive attention or reinforcement learning.
• Extending the dual-memory framework to multi-agent interactive scenarios and more complex embodied reasoning domains.

Conclusion

LaMem-VLA establishes a new paradigm for context-native memory in VLA models by architecting historical experience as dual-scale, latent memory tokens directly embedded into the reasoning path. Empirical results across demanding manipulation benchmarks corroborate significant gains in long-horizon action synthesis and robustness. The framework provides a fertile testbed for further developments in temporally-aware, memory-augmented embodied intelligence and opens the field to richer forms of generative memory and lifelong learning in robotics.