RoboMME: Robotic Memory Benchmark

• RoboMME is a benchmark that evaluates vision-language-action policies in long-horizon robotic manipulation by testing four cognitive memory types: temporal, spatial, object, and procedural.
• It features 16 well-defined tasks and 1,600 expert demonstrations to systematically analyze memory representations and integration strategies in robotics.
• The evaluation protocol measures success rates, computational efficiency, and interference robustness to guide the development of adaptive memory-augmented robotic systems.

RoboMME (Robotic Memory-augmented Manipulation Evaluation) is a large-scale standardized benchmark for evaluating and advancing vision-language-action (VLA) policies in long-horizon, history-dependent robotic manipulation scenarios. It provides a cognitively motivated taxonomy of memory types, a suite of diverse manipulation tasks, and a controlled evaluation protocol for systematically analyzing the capabilities and limitations of memory-augmented robotic policies. The benchmark addresses critical gaps in robot memory evaluation by enabling cross-comparison of memory representations, integration strategies, and robustness to interference across multi-task settings (Dai et al., 4 Mar 2026, Rathi, 21 Jun 2026).

1. Motivation and Foundational Principles

Long-horizon robotic manipulation frequently departs from Markovian assumptions, compelling agents to recall temporally distant events, occluded object locations, referential identities, and complex motion patterns. Previous benchmarks have inadequately addressed these requirements, often limiting themselves to single-episode or narrow memory types. RoboMME was developed to systematically probe four major cognitive memory types—temporal ("when"), spatial ("where"), object ("what"), and procedural ("how")—across a broad spectrum of manipulation tasks. The foundational observation is that no single policy or representation suffices across all history-dependent scenarios, necessitating a comprehensive, scalable evaluation framework (Dai et al., 4 Mar 2026).

2. Benchmark Structure and Task Taxonomy

RoboMME’s design incorporates a cognitively grounded taxonomy derived from classical psychology, operationalized through four task suites:

• Counting (Temporal memory): Tasks such as BinFill, PickXTimes, SwingXTimes, and StopCube necessitate event counting and sequence tracking.
• Permanence (Spatial memory): VideoUnmask, ButtonUnmask, VideoUnmaskSwap, and ButtonUnmaskSwap require memory of hidden object locations and tracking under dynamic occlusion and swaps.
• Reference (Object memory): Tasks like PickHighlight, VideoRepick, VideoPlaceButton, and VideoPlaceOrder focus on resolving referential identity across visual, action, and language cues.
• Imitation (Procedural memory): MoveCube, InsertPeg, PatternLock, and RouteStick demand faithful reproduction of demonstrated motion patterns.

Each suite consists of four tasks (16 in total), explicitly constructed such that success is unattainable without leveraging the intended form of memory. The dataset comprises 1,600 expert demonstrations across these tasks, stratified by difficulty, and each task is further evaluated using 50 held-out seeds for statistical rigor (Dai et al., 4 Mar 2026).

3. Policy Backbone and Memory Augmentation Mechanisms

RoboMME centers around the π₀.₅ policy—a two-expert diffusion-transformer architecture with vision-language and action modules, trained via multi-step flow-matching loss. Memory augmentation is realized in three principal forms:

• Symbolic memory: Represented as language subgoals (SimpleSG and GroundSG) generated by an oracle, Gemini-2.5, or Qwen3-VL. Integrated directly by token concatenation with language inputs.
• Perceptual memory: Implemented as learnable banks of visual patches. Key techniques include FrameSampling (even sampling with spatial max-pooling) and TokenDropping (saliency-based selection), with multi-modal rotary position encoding.
• Recurrent memory: Composed of RMT (Recurrent Memory Transformer) with persistent slots and TTT (Test-Time Training) with online-updated fast weights.

Integration strategies for perceptual and recurrent forms comprise: Context (prepend to VLM input), Modulator (cross-attend and inject via AdaLN), and Expert (dedicated transformer with joint action attention). The benchmark evaluates 14 resultant variants, reflecting all major combinations of representation and integration (Dai et al., 4 Mar 2026).

4. Evaluation Procedures and Metrics

RoboMME employs a multi-task imitation learning protocol, maintaining a fixed 512-token memory budget per policy (mirroring current-view token count). Training proceeds for 80,000 steps, with metrics aggregated over multiple checkpoints and random seeds. The principal evaluation metric is episode success rate, with computational efficiency measured in FLOPs per rollout. Baselines include π₀.₅ (no memory), π₀.₅ with symbolic memory (past actions), and memory-planning hybrids (SAM2Act+, MemER). Human performance, benchmarked via a video QA interface with oracle planning, sets an upper bound at 90.5% average success (Dai et al., 4 Mar 2026).

5. RoboMME-Interference: Robustness Under Distractor Sessions

To interrogate interference robustness—a critical property for deployment in multi-session, lifelong learning environments—RoboMME-Interference introduces controlled cross-session distractors in the history buffer. For each query, the relevant demonstration D_r is interleaved with k unrelated sessions (k ∈ {0,1,3,7}), each drawn from distinct task families and matched in memory size. The system must retrieve and utilize D_r from possibly long, noisy buffers.

Nine tasks (four imitation and five video-conditioned) are used, and memory system performance is recorded under graduated interference levels. Evaluation involves running memory-augmented π₀.₅ variants unmodified across all history conditions and measuring binary success with Wilson 95% confidence intervals.

Success Rate Decay Under Interference

System	No history	k=0	k=1	k=3	k=7
π₀.₅ (baseline)	17.3 %	–	–	–	–
FrameSamp-Modulator	18.2 %	45.3 %	38.4 %	30.0 %	19.3 %
TokenDrop-Modulator	17.1 %	35.3 %	30.9 %	23.6 %	19.8 %
FrameSamp-Context	17.8 %	26.7 %	18.7 %	18.7 %	17.8 %
FrameSamp-Expert	17.6 %	27.1 %	20.9 %	19.6 %	19.6 %
TokenDrop-Context	13.8 %	22.9 %	19.1 %	15.3 %	13.1 %
TokenDrop-Expert	16.9 %	27.3 %	20.2 %	15.6 %	14.0 %
TTT-Expert	16.0 %	18.0 %	16.4 %	16.2 %	15.6 %
TTT-Context	16.9 %	15.3 %	16.2 %	17.6 %	14.0 %

Key findings illustrate that FrameSamp-Modulator, under k=0 (no distractors), achieves +27.1 percentage points over its no-history performance, but with increasing interference (k=1,3,7), success decays to baseline levels. Similar degradations are observed in other variants. Recurrent (TTT) approaches show no meaningful gains at any k. Perceptual Modulator integration confers the highest benefit when the relevant session is surfaced but does not mitigate interference with increased distractor depth (Rathi, 21 Jun 2026).

6. Analysis of Key Results and Design Implications

Experimental results across RoboMME's task suite reveal that:

• Memory needs are task-dependent; no single representation or integration strategy is universally optimal.
• Symbolic memories (GroundSG+QwenVL) excel in temporal and short-horizon reference tasks (Counting: BinFill 95%, PickX 95%), but are insufficient for visuomotor or procedurally intensive scenarios.
• Perceptual memory (FrameSamp-Modulator) provides the most favorable trade-off in efficiency and overall success (44.5% average), and is especially important for motion-centric (PatternLock ~94%) and time-sensitive tasks (StopCube 42%).
• Current recurrent memory variants are suboptimal due to insufficient adaptation of backbone dynamics.
• Interference experiments demonstrate that performance deteriorates markedly as unrelated sessions are present in memory buffers, indicating that sequence concatenation and token dropping do not provide effective long-context recall.

This suggests that robust robot memory systems require learned retrieval mechanisms, hierarchical memory architectures, or adaptive gating/compression strategies—approaches analogous to those emerging in LLM toolkits. Notably, an interference-robust system should target >40% success at k=7, as achieved by Modulator at k=0, rather than regressing to ~20% (Rathi, 21 Jun 2026).

7. Broader Impact and Future Directions

RoboMME formalizes the evaluation of robot memory in long-horizon, non-Markovian settings and provides a reproducible “bar to beat” for the community. The benchmark’s open-source release, rigorous protocol, and comprehensive task coverage make it a pivotal foundation for future research. Immediate research trajectories include developing learned retrieval and session indexing modules, hierarchical and adaptive memory mechanisms, and unified frameworks that combine symbolic, perceptual, and recurrent memory. A plausible implication is that advances in interference-robust memory will underpin the deployment of reliable, generalist robotic agents capable of lifelong adaptation in unconstrained environments (Dai et al., 4 Mar 2026, Rathi, 21 Jun 2026).