MemoryVLA++: Memory and Imagination in Robotics

• MemoryVLA++ is a vision-language-action framework that integrates explicit temporal modeling with perceptual, cognitive, and imaginative components.
• It leverages a Perceptual-Cognitive Memory Bank and a latent-space video diffusion world model to retrieve past context and predict future states.
• The architecture delivers significant performance gains on both simulation and real-robot tasks, underscoring its robust temporal reasoning in long-horizon manipulation.

MemoryVLA++ is a vision-language-action (VLA) framework for robotic manipulation that introduces explicit temporal modeling through memory and imagination. It is designed for settings in which control depends not only on the current RGB observation and language instruction, but also on past interactions and possible future state evolution. In the formulation reported by the authors, robotic manipulation is modeled as $\mathcal{A} = (a_1,\dots,a_T) = \pi(I,L)$, where $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$ denotes RGB views, $L$ is a language instruction, and the policy outputs a sequence of 7-DoF end-effector actions. The system combines a frozen pretrained VLM encoder, a Perceptual-Cognitive Memory Bank (PCMB), a latent-space video diffusion world model, and a diffusion action expert, with the stated aim of unifying present perception, past memory, and future imagination in a single end-to-end VLA framework (Shi et al., 8 Jun 2026).

1. Conceptual position within VLA research

MemoryVLA++ is motivated by the observation that temporal modeling is essential for robotic manipulation and that many VLA policies rely primarily on the current observation, which limits performance on long-horizon and temporally dependent tasks. The reported framing is explicitly cognitive: working memory buffers short-lived context, a hippocampal-like memory system preserves episodic information from past experience, and an internal model supports imagination of future state evolution (Shi et al., 8 Jun 2026).

Within the lineage represented in the provided sources, MemoryVLA++ extends an earlier line of work on memory-augmented VLAs that used a language scratchpad as an external textual buffer. That earlier formulation maintained a scratchpad state $S_t = [d_1,d_2,\dots,d_n]$ of short natural-language descriptions and modeled $p(a_t,d_t \mid o_t,S_t,\ell)$, where the scratchpad stored past sub-tasks, object positions, or other task-relevant “thoughts” (Haresh et al., 24 Feb 2026). In contrast, MemoryVLA++ replaces a purely text-based external memory with a structured temporal stack composed of perceptual tokens, cognitive tokens, explicit memory retrieval, redundancy-aware consolidation, and latent future imagination. This suggests a shift from human-readable symbolic memory to a hybrid token-based temporal representation designed to capture both low-level and high-level temporal dependencies.

A common simplification is to treat temporal competence in VLAs as equivalent to recurrence. The reported results do not support that simplification. The scratchpad study states that even recurrent VLAs benefit from explicit external memory once the horizon exceeds roughly 20 steps, while MemoryVLA++ adopts not only memory retrieval but also a world model for future-state imagination, indicating that recurrence alone is not treated as a sufficient temporal mechanism in this line of work (Haresh et al., 24 Feb 2026).

2. Core architecture and data flow

The architecture comprises four high-level modules. First, a frozen pretrained VLM encoder produces perceptual tokens $p \in \mathbb{R}^{N_p \times d_p}$ and a cognitive token $c \in \mathbb{R}^{1 \times d_c}$; together these form the working memory. Second, a Perceptual-Cognitive Memory Bank stores up to $L$ past entries of both streams, denoted

$$M_{\mathrm{pcmb}} = \bigl\{\,m^p_i \in \mathbb{R}^{N_p \times d_p},\; m^c_i \in \mathbb{R}^{1 \times d_c}\bigr\}_{i=1}^L.$$

Third, a world model based on latent-space video diffusion performs partial denoising conditioned on the current observation and instruction to produce imagined future latent tokens $z_{\mathrm{img}}$. Fourth, a diffusion action expert, implemented as a Transformer plus DDIM, denoises noisy action trajectories into the output action sequence $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$0 (Shi et al., 8 Jun 2026).

The working-memory encoder itself is bifurcated. For perceptual tokens, each view is passed through two parallel visual encoders, DINOv2 and SigLIP, whose outputs are concatenated and reduced by an SE bottleneck:

$$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$1

For the cognitive token, the same raw visual tokens are projected into the language embedding space, concatenated with the tokenized instruction, fed into LLaMA-7B, and the final EOS embedding is taken as

$$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$2

No additional loss is applied to $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$3 during policy training, and the VLM remains frozen (Shi et al., 8 Jun 2026).

This organization distinguishes MemoryVLA++ from memory mechanisms that are appended externally to the policy input. In the scratchpad-augmented setting, the model reads the entire text context by re-feeding the concatenated instruction, scratchpad, and current image tokens into a transformer at each step (Haresh et al., 24 Feb 2026). MemoryVLA++ instead embeds memory, retrieval, and future prediction directly in the token pipeline before action generation.

3. Perceptual-Cognitive Memory Bank

The PCMB is the system’s explicit storage mechanism for historical context. At each time step $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$4, after memory fusion, the augmented tokens $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$5 and $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$6 are appended to the bank. Retrieval is performed separately for the perceptual and cognitive streams. With current queries $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$7 and $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$8, keys and values are formed by combining stored entries with sinusoidal timestep embeddings $$I=\{I^v\}_{v=1}^V \subset \mathbb{R}^{H\times W\times 3}$$9:

$L$0

Cross-attention then produces retrieved context

$L$1

followed by two Transformer layers to produce $L$2 (Shi et al., 8 Jun 2026).

Fusion of current and retrieved tokens is adaptive rather than additive. The gating rule is

$L$3

This means the model can interpolate between the current observation-derived representation and retrieved historical context on a stream-specific basis. The paper’s ablations explicitly compare gate versus additive fusion, indicating that fusion design is treated as a substantive modeling choice rather than a minor implementation detail (Shi et al., 8 Jun 2026).

When the PCMB exceeds its capacity $L$4, MemoryVLA++ uses redundancy-aware consolidation rather than simple truncation. For each stream, adjacent cosine similarities $L$5 are computed, the most similar adjacent pair is selected, and the pair is merged by averaging:

$L$6

after which $L$7 is deleted. The reported ablations compare token-merge versus FIFO consolidation and memory lengths of 4, 16, and 64, which indicates that retention strategy and memory budget are central hyperparameters of the framework (Shi et al., 8 Jun 2026).

A plausible implication is that the PCMB is intended to preserve both fine-grained visual traces and task-level semantics in a compressed episodic store. That interpretation aligns with the design choice to maintain separate perceptual and cognitive streams rather than collapsing memory into a single modality.

4. World model and imagination mechanism

MemoryVLA++ does not treat past memory as the sole source of temporal competence. It augments retrieved historical context with a world model that imagines future states in latent space. The world model is adapted from Stable Video Diffusion and trained on robot-manipulation videos with the diffusion MSE objective

$L$8

where

$L$9

During policy training, the world model is frozen. Given $S_t = [d_1,d_2,\dots,d_n]$0, it performs one or a few denoising steps and yields multi-scale UNet features $S_t = [d_1,d_2,\dots,d_n]$1, which are aggregated by a small FPN into latent tokens $S_t = [d_1,d_2,\dots,d_n]$2 with temporal embeddings $S_t = [d_1,d_2,\dots,d_n]$3 (Shi et al., 8 Jun 2026).

The imagination tokens are refined by spatial and temporal attention. Learnable queries $S_t = [d_1,d_2,\dots,d_n]$4 are initialized for each future step $S_t = [d_1,d_2,\dots,d_n]$5. Spatial attention is applied per step, producing $S_t = [d_1,d_2,\dots,d_n]$6, and temporal attention across the sequence yields

$S_t = [d_1,d_2,\dots,d_n]$7

These imagined latents are then integrated into the memory-augmented perceptual stream by cross-attention and another gate:

$S_t = [d_1,d_2,\dots,d_n]$8

The resulting full temporal-aware tokens are $S_t = [d_1,d_2,\dots,d_n]$9 (Shi et al., 8 Jun 2026).

This architecture operationalizes “imagination” as partially denoised latent future structure rather than explicit symbolic planning. That distinguishes it from the earlier scratchpad approach, in which a plan and progress toward subgoals were represented in natural language descriptions stored in the scratchpad (Haresh et al., 24 Feb 2026). The two approaches therefore target temporal reasoning through different representational media: textual subtask memory in one case, latent predictive modeling in the other.

5. Action generation, optimization, and empirical performance

Action prediction is handled by a diffusion action expert. Noisy actions $p(a_t,d_t \mid o_t,S_t,\ell)$0 at diffusion step $p(a_t,d_t \mid o_t,S_t,\ell)$1 are combined with the cognitive token through a cognition-attention block:

$p(a_t,d_t \mid o_t,S_t,\ell)$2

The resulting representation attends to the perceptual stream $p(a_t,d_t \mid o_t,S_t,\ell)$3, and a feed-forward network predicts $p(a_t,d_t \mid o_t,S_t,\ell)$4:

$p(a_t,d_t \mid o_t,S_t,\ell)$5

The policy objective is the mean-squared-error diffusion loss $p(a_t,d_t \mid o_t,S_t,\ell)$6. Training proceeds in two stages: the world model is trained first, typically for 20k–40k steps, and then frozen; policy training optimizes the diffusion expert end-to-end, including memory-fusion and imagination-integration layers, with $p(a_t,d_t \mid o_t,S_t,\ell)$7 and $p(a_t,d_t \mid o_t,S_t,\ell)$8 (Shi et al., 8 Jun 2026).

The reported simulation results span multiple benchmarks. On Libero, covering 5 suites and 130 tasks, MemoryVLA++ achieves 98.4% versus CogACT’s 93.2% for a gain of +5.2. On SimplerEnv with 4 tasks, it reports 73.9% versus 57.3% for +16.6. On Mikasa-Robo with 5 memory tasks, it reports 44.4% versus 29.4% for +15.0. On Calvin, which uses 5-step language tasks, the average completed steps are 4.29 versus 3.25, a gain of +1.04. On Libero-Plus with 7 OOD variations, zero-shot performance is 73.1% versus 67.9% (+5.2), and fine-tuned performance is 82.7% versus 79.6% (+3.1) (Shi et al., 8 Jun 2026).

The real-robot evaluation covers Franka, WidowX, and Dual-ARX5. For general tasks, defined as 6 short-horizon tasks, the reported result is 85% versus 76% (+9). For memory-dependent tasks, defined as 6 long-horizon tasks, the result is 83% versus 57% (+26). For imagination-dependent tasks, defined as 5 long-horizon conveyor/zipper tasks, the result is 77% versus 49% (+28). These numbers are presented as evidence that the combined memory-and-imagination design yields larger gains on temporally demanding tasks than on short-horizon manipulation (Shi et al., 8 Jun 2026).

6. Relation to scratchpad-augmented memory, limitations, and interpretation

The earlier scratchpad-augmented VLA work is directly relevant because it isolates the contribution of explicit memory to memory-dependent manipulation tasks. In that setting, a language scratchpad made it possible to memorize object positions, keep track of a plan, and record progress toward subgoals. On ClevrSkills-Mem, the stateless T-VLA baseline reported an average success rate of approximately 19%, while adding the scratchpad increased performance to approximately 68%, an improvement of +48.8 percentage points. The recurrent R-VLA baseline reported approximately 66%, increasing to approximately 77% with a scratchpad, a gain of +11 percentage points. On MemoryBench “Put-Block-Back,” T-VLA reported 0%, increasing to 40% with a scratchpad; a sim-eval setting with an oracle for the button press reported 100%. On a real-world pick-place-restore task with 20 randomized trials on xArm6, OpenVLA (7B finetuned) reported 0%, while the LoRA-based scratchpad variant reported 65%, with subtask completion rate 2.4 / 3 versus 3 / 3 human and mean final displacement 10.6 cm (Haresh et al., 24 Feb 2026).

These results matter for interpreting MemoryVLA++. The scratchpad system shows that explicit memory alone can substantially improve performance on non-Markovian manipulation. MemoryVLA++ retains that emphasis on memory, but adds two further commitments: first, that memory should be structured into perceptual and cognitive streams with learned retrieval and consolidation; second, that future-state imagination should be integrated into the policy representation before action denoising (Shi et al., 8 Jun 2026). This suggests that MemoryVLA++ can be read not as a replacement for memory-centric VLAs, but as a generalization of that agenda toward full temporal modeling.

The reported limitations are also specific. The PCMB may fail to retrieve the truly decisive past frame, and the latent world model may predict improbable futures when dynamics deviate from training. The framework adds approximately 30% inference latency, from 0.19 to 0.24 s, and approximately 5 GB of GPU usage over the baseline, while remaining within real-time control budgets. Proposed future extensions include dynamic memory slot allocation or hierarchical memory management, joint fine-tuning of the world model on policy data, incorporation of explicit physics priors or learned transition models, and multi-modal memory entries such as proprioceptive or force signals (Shi et al., 8 Jun 2026).

A recurring misconception in discussions of VLA temporal modeling is that “memory” names a single mechanism. The two papers indicate otherwise. One implements memory as an ever-growing natural-language buffer with update triggered by a designated token; the other implements memory as a learned bank of perceptual and cognitive token histories with timestep-aware retrieval and consolidation. Another misconception is that temporal modeling is reducible to past-context retention. MemoryVLA++ explicitly rejects that reduction by coupling memory with imagination, treating future-state inference as a coequal component of manipulation competence (Shi et al., 8 Jun 2026).