---
title: Multi-Scale Embodied Memory (MEM)
url: https://www.emergentmind.com/topics/multi-scale-embodied-memory-mem
type: topic
---

# Multi-Scale Embodied Memory (MEM)

Multi-Scale Embodied Memory (MEM) is an approach for mixed-modal long-horizon memory in robot policies that combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory [2603.03596]. In the MEM formulation, robotic memory is not represented in a single uniform format: dense recent visual history is used for short-term control and recovery from partial observability, while compact semantic summaries are used to remember task progress over many minutes. Together, these mechanisms enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich [2603.03596]. Within the broader embodied-memory literature, related systems use global memory with frontier map, landmark semantic memory, and visitation memory [2502.14254], Memory Snapshots and Frontier Snapshots [2411.17735], or a multi-index architecture searchable by meaning, space, and time [2606.03374].

## 1. Conceptual foundations

The central premise of MEM is that long-horizon, multi-stage robotic tasks require memory at multiple temporal and semantic scales. The MEM paper states that conventional end-to-end robotic learning typically gives a policy memory by feeding in a sequence of past observations, but that this breaks down for long-horizon real-world tasks because the information a robot needs to retain is heterogeneous [2603.03596]. Short-term, fine-grained memory is often visual and geometric: if the robot’s arm occludes an object while reaching, it needs to remember where that object was a moment ago; if a grasp slips, it may need to infer from recent visual dynamics that the previous grasp height or angle failed and should be changed. Long-term, semantic memory is often abstract and compact: in cooking or cleanup, the robot does not need raw images from ten minutes ago, but facts such as which ingredients have already been retrieved, whether soap was already applied, whether both sides of a sandwich were cooked, or whether a cabinet that was opened still needs to be closed [2603.03596].

This multi-scale view aligns with several adjacent embodied-memory systems, although they operationalize scale differently. MEIA introduces a Multimodal Environment Memory that combines environmental language memory and environmental image memory, but the paper explicitly characterizes it as a multimodal, partially multi-level embodied memory rather than a fully realized multi-scale memory system [2402.00290]. 3DLLM-Mem separates a limited-capacity working memory for the current observation from an expandable episodic memory for past observations and interactions, which is explicitly dual-system and strongly temporal in character [2505.22657]. This suggests that MEM belongs to a broader shift away from monolithic history buffers toward heterogeneous memory substrates specialized by timescale, modality, and abstraction level.

## 2. Hierarchical policy decomposition

MEM is integrated into a hierarchical vision-language-action pipeline with a high-level policy and a low-level policy. The overall target is a policy that predicts an action chunk given a task goal and a history of observations,
\[
\pi(a_{t:t+H} \mid o_{t-T:t}, g),
\]
but because conditioning directly on all past observations is infeasible, the paper factorizes the policy as
\[
\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),
\]
with \(K \ll T\) [2603.03596].

In this factorization, the high-level policy \(\pi_\text{HL}\) observes the current state, the task goal, and the current language memory \(m_t\), and predicts both the next subtask instruction \(l_{t+1}\) and the updated memory state \(m_{t+1}\). The low-level policy \(\pi_\text{LL}\) conditions on the recent observation window \(o_{t-K:t}\), the goal \(g\), and the next subtask instruction \(l_{t+1}\), and predicts an action chunk \(a_{t:t+H}\) [2603.03596]. The instantiated system is described as a \(\pi_0.5\) / pi0.5-like VLA system, specifically referred to as \(\pizerosix\)-MEM, initialized from a Gemma3-4B VLM backbone. Input observations include up to four camera streams, images at \(448\times448\) px per stream, and proprioceptive robot state. The action model uses discrete FAST action token prediction and a flow-matching action expert with 860M parameters, and the paper notes that gradients do not flow from the action expert into the VLM backbone [2603.03596].

The significance of this decomposition is that memory is not treated as passive conditioning context. Long-term language memory is maintained explicitly by the planner-like high-level module, whereas short-horizon visual memory is consumed by the control-oriented low-level module. This is a strong separation of roles: semantic progress tracking at one level, dense visuomotor adaptation at another.

## 3. Short-horizon video memory

MEM’s short-term memory is implemented by an efficient video ViT. Instead of encoding each past frame independently and passing all tokens to the VLA backbone, the paper modifies the original single-image ViT into a video encoder that patchifies all frames separately, performs standard spatial attention within each frame, augments every 4th layer with causal temporal attention across timesteps for corresponding spatial patches, and then drops tokens from past timesteps in upper layers, retaining only the representation for the current timestep [2603.03596]. Temporal information is therefore folded into the current-frame representation, so the VLA backbone sees about the same token count as in a standard single-frame model.

The formal attention construction introduces a temporal position embedding \(e(t)\) at layer \(l\):
\[
\hat{\mathbf{z}^{\,l-1}_{p,t} = \mathbf{z}^{l-1}_{p,t} + e(t),
\qquad e(0)=0,
\]
and reuses the pretrained ViT query, key, and value projections:
\[
\begin{aligned}
\mathbf{q}^{l, a}_{p, t} &= W^{l, a}_Q \,\text{LN}\big( \hat{\mathbf{z}^{\,l-1}_{p, t} \big), \\
\mathbf{k}^{l, a}_{p, t} &= W^{l, a}_K \,\text{LN}\big( \hat{\mathbf{z}^{\,l-1}_{p, t} \big), \\
\mathbf{v}^{l, a}_{p, t} &= W^{l, a}_V \,\text{LN}\big( \hat{\mathbf{z}^{\,l-1}_{p, t} \big).
\end{aligned}
\]
The encoder factorizes full spatiotemporal attention from
\[
\mathcal{O}(n^2 K^2)
\]
to
\[
\mathcal{O}(K n^2 + n K^2),
\]
where \(n\) is the number of spatial patches per frame and \(K\) is the number of timesteps [2603.03596].

This module is parameter-efficient: it adds no new learnable parameters compared to the original single-image ViT, changes only the attention pattern, adds fixed sinusoidal temporal embeddings, and can be initialized directly from pretrained ViT weights. In pretraining, the model uses 6 observations total, 5 past plus current, with 1-second stride. During post-training and inference, the horizon is expanded to up to 18 frames, spanning up to 54 seconds of observation memory [2603.03596]. The paper’s broader claim is that effective short-term memory in robotics must preserve geometry, appearance, and temporal structure, but only over a limited window where raw visual context remains computationally practical.

## 4. Long-horizon language memory

The long-term memory \(m_t\) is a natural-language summary of past semantic events. It is explicitly maintained by the high-level policy and updated over time [2603.03596]. The paper illustrates this with cumulative semantic updates such as moving from “I placed a plate in the cabinet and moved to the counter” to “I placed a plate in the cabinet, moved to the counter, and picked up a bowl.” The system is trained to compress and discard irrelevant detail rather than merely append everything; for example, “I put a light green bowl, a dark blue bowl and a bright yellow bowl into the top right cabinet” may be compressed to “I placed three bowls in the top right cabinet” [2603.03596].

Training supervision for this language memory is constructed automatically. Given a robot episode with subtask language annotations \(l_{0:T}\) and an indicator of whether each subtask execution succeeded or failed, the authors feed the subtask sequence and success/failure information to an off-the-shelf pretrained LLM, prompt it to summarize all previous information that remains relevant for future task execution, and use the resulting summary as the supervision target for the transition \(m_t \to m_{t+1}\) [2603.03596]. The memory update process therefore depends on language supervision derived from subtask-level annotations and an external LLM summarizer.

A central comparison is against a naive language memory that simply concatenates prior subtask instructions. The paper reports that this performs worse because of train-inference distribution shift: demonstrations used in training are near-optimal, so each subtask typically appears once, whereas at test time the robot may fail multiple times, yielding repeated subtask attempts such as
\[
\text{pick up bowl} \rightarrow \text{pick up bowl} \rightarrow \text{pick up bowl} \rightarrow \text{place bowl in cabinet}.
\]
MEM avoids this by updating memory only when semantically appropriate; failed attempts may not change \(m_t\) at all until success occurs [2603.03596]. This suggests that long-horizon language memory in embodied control is not merely a textual log, but a compression mechanism for persistent task state.

## 5. Tasks, adaptation, and empirical behavior

The MEM paper organizes its evaluation around three questions: whether memory can solve tasks requiring memory up to 15 minutes, whether memory improves in-context adaptation even in shorter tasks, and how MEM compares to prior memory architectures [2603.03596]. The long-horizon benchmarks include recipe setup and clean up kitchen. In recipe setup, the robot is given a natural-language recipe prompt specifying required ingredients and cookware, their initial locations, and target placements; it is trained on 42 recipes and evaluated on 5 recipes, with unseen kitchens and unseen objects. In clean up kitchen, the robot must wipe counters, dry them, throw towels away, put food items into the fridge, move dishes to cabinets, wash dishes with soap and water, and place washed dishes in the dish rack; episodes average about 8 subtasks [2603.03596].

The in-context adaptation tasks are designed to isolate short-term corrective memory. One task requires chopstick pick-up under out-of-distribution table height, so the policy must adjust grasp height after mis-grasps. Another requires opening a refrigerator whose hinge side is visually ambiguous; an episode is counted successful if the door opens in \(\le 4\) grasps [2603.03596]. The paper reports that with correction data, MEM policies learn to adjust grasp height after a failed chopstick pickup and to switch door-opening strategy after trying the wrong refrigerator-opening direction. The no-memory baseline cannot do this intelligently because it cannot remember which strategy was already attempted [2603.03596].

Across these evaluations, the paper reports several qualitative findings. MEM substantially improves long-horizon task performance, and both video memory and language memory are necessary. Without video memory, policies can get stuck indefinitely during tasks such as washing or wiping because they lack recent temporal context. Without language memory, policies are poorer at remembering completed recipe steps, tracking which objects have already been fetched or stowed, remembering to close doors or cabinets, and tracking cleanup progress over long intervals [2603.03596]. The paper also reports that Pool Memory can help on simple memory tasks but degrades on tasks needing richer long-term tracking, and that Proprio Memory helps when only robot-state history matters but fails when environment-state memory is needed. A Posttrain-only MEM variant performs noticeably worse than full MEM, suggesting that memory use itself must be learned during broad pretraining, not merely attached later. At the same time, MEM is reported to match state-of-the-art no-memory VLA performance on dexterous tasks that do not require memory [2603.03596].

## 6. Related architectures and open directions

MEM sits within a rapidly diversifying landscape of embodied-memory architectures. Some systems emphasize global allocentric structure, some foreground image-based episodic recall, and others make graph-based retrieval or affordance-aware selection primary.

| System | Memory organization | Embodied use |
|---|---|---|
| Mem2Ego [2502.14254] | frontier map, landmark semantic memory, visitation memory | ObjectNav via global-to-ego reasoning |
| 3D-Mem [2411.17735] | Memory Snapshots and Frontier Snapshots | exploration and embodied reasoning |
| 3DLLM-Mem [2505.22657] | working memory tokens and episodic memory | spatial-temporal reasoning and actions |
| eMEM [2606.03374] | observation, episode, gist, entity with SQLite, HNSW, and R-tree | searchable by meaning, space, and time |
| SpatialMem [2601.14895] | unified 3D memory with metric anchoring and hierarchical object nodes | language-guided navigation and object retrieval |

These systems illuminate complementary design points. Mem2Ego preserves geometry in a frontier map, stores landmark descriptions anchored to global coordinates, and projects global cues into the current panorama rather than reasoning over map text alone [2502.14254]. 3D-Mem stores explored regions as image-based Memory Snapshots and unexplored regions as Frontier Snapshots, explicitly coupling recall with information-seeking exploration [2411.17735]. eMEM unifies observation nodes, episode nodes, gist nodes, and entity nodes behind a multi-index architecture with SQL ITE for structured storage, hnswlib for semantic search, and an R-tree for spatial queries [2606.03374]. SpatialMem reconstructs metrically scaled indoor environments from casually captured egocentric RGB video, uses walls, doors, and windows as first-layer anchors, and links evidence patches, visual embeddings, and two-layer textual descriptions to 3D coordinates for compact storage and fast retrieval [2601.14895]. Affordance RAG, meanwhile, builds an Affordance-Aware Embodied Memory spanning affordance, instance, view, zone, area, and building levels, and reranks semantically matched candidates with affordance scores so that memory supports executability as well as recognition [2512.18987].

Other lines of work foreground mechanisms that MEM itself does not explicitly include. Memo interleaves periodic summarization tokens with policy inputs and can be interpreted as inducing a temporal hierarchy from raw recent observations to accumulated segment summaries [2510.19732]. MemCtrl uses a trainable memory head \(\mu\) as a gate to determine which observations or reflections to retain or discard during exploration, emphasizing online write-time pruning under bounded compute [2601.20831]. MEMENTO shows that personalized embodied assistance requires memory not only for object semantics but also for user patterns and multi-memory composition, with even GPT-4o showing a 30.5% performance drop on joint-memory tasks [2505.16348]. This suggests that future MEM systems may need stronger mechanisms for selective retrieval, pattern abstraction, and compositional reuse than episodic replay alone.

The MEM paper itself explicitly notes that future work should extend memory beyond an episode to weeks or months of deployment [2603.03596]. In light of the broader literature, a plausible implication is that a fuller multi-scale embodied memory would combine MEM’s mixed-modal control-oriented decomposition with explicit spatial grounding, hierarchical retrieval, write-time pruning, and longer-horizon consolidation. Such a system is not yet realized in the cited work, but the current literature already defines its main ingredients.

Source: https://www.emergentmind.com/topics/multi-scale-embodied-memory-mem