MemER: Scaling Up Memory for Robot Control via Experience Retrieval

Abstract: Humans routinely rely on memory to perform tasks, yet most robot policies lack this capability; our goal is to endow robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing vision-language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we finetune Qwen2.5-VL-7B-Instruct and $\pi_{0.5}$ as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory. Videos and code can be found at https://jen-pan.github.io/memer/.

• The paper introduces MemER, a hierarchical framework that selects keyframes from past experiences to enhance long-horizon robotic control.
• The methodology combines a VLM-based high-level policy with a low-level robot policy, using keyframe clustering to maintain compact memory.
• Experiments demonstrate that MemER outperforms baselines with superior efficiency, robustness, and generalization across real-world tasks.

Scaling Memory for Long-Horizon Robotic Control: An Analysis of MemER

Introduction

MemER introduces a hierarchical vision-language-action (VLA) framework for robotic control that explicitly addresses the challenge of long-horizon memory in manipulation tasks. The framework is motivated by the observation that existing robot policies, even those based on large-scale VLMs, are limited by their inability to efficiently utilize and retrieve task-relevant information from extended observation histories. MemER leverages a high-level policy to select keyframes from past experiences, enabling efficient reasoning over long temporal dependencies while maintaining low-latency inference and compatibility with current VLA architectures.

Figure 1: MemER overview, illustrating the hierarchical memory retrieval mechanism for long-horizon robotic tasks.

Hierarchical Policy Architecture and Memory Construction

MemER decomposes the control policy into a high-level VLM-based policy and a low-level robot policy. The high-level policy, finetuned from Qwen2.5-VL-7B-Instruct, is responsible for both subtask prediction and keyframe nomination. At each timestep, it receives the most recent $N$ frames, the task instruction, and a compact set of selected keyframes from the episode. The policy outputs the next subtask and candidate keyframes, which are filtered and consolidated into episodic memory.

Figure 2: Architecture of MemER, showing the interaction between high-level and low-level policies and the flow of keyframes and subtasks.

Keyframe selection is performed via a 1D single-linkage clustering algorithm over the temporal indices of nominated frames. This process aggregates candidate keyframes across time, merges those within a fixed distance $d$, and selects the median frame from each cluster as the representative keyframe. This approach ensures that memory remains compact and relevant, avoiding the computational overhead and redundancy associated with naively conditioning on long observation histories.

Figure 3: Visualization of the keyframe clustering and selection process, demonstrating temporal aggregation and representative frame selection.

Implementation Details

High-Level Policy

• Model: Qwen2.5-VL-7B-Instruct, finetuned for subtask and keyframe prediction.
• Inputs: Recent context frames, selected keyframes, task instruction.
• Outputs: Current subtask, candidate keyframes.
• Training: 50 expert demonstrations per task, with semi-automatic keyframe annotation based on subtask boundaries and human-defined rules.
• Model Merging: Linear interpolation of pretrained and finetuned weights ($\alpha=0.8$) to preserve generalization and robustness.

Low-Level Policy

• Model: $\pi_{0.5}$, finetuned on the DROID dataset.
• Inputs: Current image, robot state, subtask.
• Outputs: Action chunk for robot control.
• Training: 50 long-horizon demonstrations and 10–15 intervention examples per task.

Deployment

• Scheduling: High-level policy at $\sim$1Hz, low-level policy at $\sim$2Hz, asynchronous execution for responsiveness.
• Inference: Subsampled image streams (2Hz), compact memory representation, low-latency closed-loop control.

Experimental Evaluation

MemER is evaluated on three real-world long-horizon tasks: Object Search, Counting Scoops, and Dust & Replace. Each task requires the robot to recall information spanning minutes, such as object locations, completed actions, and count tracking.

Figure 4: Example tasks used for evaluation, highlighting the complexity and memory requirements.

Main Results

MemER outperforms baselines that use no history, short history (8 frames), and long history (32 frames) across all tasks. Notably, the long history baseline, despite increased context, incurs significant inference latency and still underperforms compared to MemER. MemER achieves performance on par with a human high-level policy, with the most common failure mode attributable to low-level execution errors.

Figure 5: Comparative performance of MemER and baselines on long-horizon tasks, demonstrating superior task completion and optimal path adherence.

Modality and Model Merging Ablations

Experiments reveal that representing memory with images alone yields better performance than using text or a combination of text and images. The high-level policy tends to over-index on textual tokens, missing critical visual details. Model merging further improves or maintains performance across all tasks, validating the benefit of interpolating pretrained and finetuned weights.

Figure 6: (Left) Modality ablation results; (Right) model merging results, showing the impact of input modality and weight interpolation.

Multi-Task Generalization

Multi-task training enables MemER to generalize to novel object-task combinations, outperforming single-task variants when objects are swapped between tasks. This demonstrates the scalability and robustness of the memory retrieval mechanism across diverse domains.

Figure 7: (Left) Single- vs multi-task performance; (Right) cross-task object generalization, highlighting the benefits of multi-task training.

Keyframe Selection Algorithm

The keyframe selection algorithm is central to MemER's memory efficiency. It operates as follows:

def build_visual_memory(candidate_keyframes, merge_distance): indices = sorted([frame.index for frame in candidate_keyframes]) clusters = [] current_cluster = [indices[0]] for i in range(1, len(indices)): if indices[i] - indices[i-1] <= merge_distance: current_cluster.append(indices[i]) else: clusters.append(current_cluster) current_cluster = [indices[i]] clusters.append(current_cluster) selected_indices = [median(cluster) for cluster in clusters] selected_keyframes = [frames[i] for i in selected_indices] return selected_keyframes

This algorithm ensures that only the most informative frames are retained, keeping the memory footprint manageable even for extended episodes.

Limitations and Future Directions

MemER's episodic memory grows monotonically, lacking mechanisms for keyframe deletion or modification, which may be necessary for tasks requiring hours of recall. Throughput is bounded by VLM inference speed and scheduling, limiting applicability to high-frequency control scenarios. The current framework is restricted to visual memory; incorporating multimodal sensory data (e.g., tactile, audio) and extending to mobile manipulation or multi-room tasks are promising avenues for future research.

Conclusion

MemER presents a scalable, hierarchical approach to memory retrieval in robotic control, enabling efficient reasoning over long-horizon dependencies with minimal expert data. The framework's keyframe selection and consolidation strategy, combined with robust VLM priors and model merging, yield strong empirical results on complex manipulation tasks. The work highlights the importance of explicit memory mechanisms in robotics and sets the stage for further exploration of modifiable, multimodal, and spatially-aware memory architectures.