Multi-Scale Embodied Memory (MEM)

• Multi-Scale Embodied Memory (MEM) is an approach that integrates detailed visual short-term memory with compact language-based long-term memory for multi-stage robotic tasks.
• It employs a hierarchical policy decomposition that separates fine-grained visuomotor control from high-level semantic planning to enhance task execution and error recovery.
• MEM achieves efficient memory usage by compressing visual data via a video encoder and summarizing semantic progress with a language model over extended time horizons.

Multi-Scale Embodied Memory (MEM) is an approach for mixed-modal long-horizon memory in robot policies that combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory (Torne et al., 4 Mar 2026). In the MEM formulation, robotic memory is not represented in a single uniform format: dense recent visual history is used for short-term control and recovery from partial observability, while compact semantic summaries are used to remember task progress over many minutes. Together, these mechanisms enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich (Torne et al., 4 Mar 2026). Within the broader embodied-memory literature, related systems use global memory with frontier map, landmark semantic memory, and visitation memory (Zhang et al., 20 Feb 2025), Memory Snapshots and Frontier Snapshots (Yang et al., 2024), or a multi-index architecture searchable by meaning, space, and time (Rasheed et al., 2 Jun 2026).

1. Conceptual foundations

The central premise of MEM is that long-horizon, multi-stage robotic tasks require memory at multiple temporal and semantic scales. The MEM paper states that conventional end-to-end robotic learning typically gives a policy memory by feeding in a sequence of past observations, but that this breaks down for long-horizon real-world tasks because the information a robot needs to retain is heterogeneous (Torne et al., 4 Mar 2026). Short-term, fine-grained memory is often visual and geometric: if the robot’s arm occludes an object while reaching, it needs to remember where that object was a moment ago; if a grasp slips, it may need to infer from recent visual dynamics that the previous grasp height or angle failed and should be changed. Long-term, semantic memory is often abstract and compact: in cooking or cleanup, the robot does not need raw images from ten minutes ago, but facts such as which ingredients have already been retrieved, whether soap was already applied, whether both sides of a sandwich were cooked, or whether a cabinet that was opened still needs to be closed (Torne et al., 4 Mar 2026).

This multi-scale view aligns with several adjacent embodied-memory systems, although they operationalize scale differently. MEIA introduces a Multimodal Environment Memory that combines environmental language memory and environmental image memory, but the paper explicitly characterizes it as a multimodal, partially multi-level embodied memory rather than a fully realized multi-scale memory system (Liu et al., 2024). 3DLLM-Mem separates a limited-capacity working memory for the current observation from an expandable episodic memory for past observations and interactions, which is explicitly dual-system and strongly temporal in character (Hu et al., 28 May 2025). This suggests that MEM belongs to a broader shift away from monolithic history buffers toward heterogeneous memory substrates specialized by timescale, modality, and abstraction level.

2. Hierarchical policy decomposition

MEM is integrated into a hierarchical vision-language-action pipeline with a high-level policy and a low-level policy. The overall target is a policy that predicts an action chunk given a task goal and a history of observations,

$\pi(a_{t:t+H} \mid o_{t-T:t}, g),$

but because conditioning directly on all past observations is infeasible, the paper factorizes the policy as

$$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$

with $K \ll T$ (Torne et al., 4 Mar 2026).

In this factorization, the high-level policy $\pi_\text{HL}$ observes the current state, the task goal, and the current language memory $m_t$, and predicts both the next subtask instruction $l_{t+1}$ and the updated memory state $m_{t+1}$. The low-level policy $\pi_\text{LL}$ conditions on the recent observation window $o_{t-K:t}$, the goal $g$, and the next subtask instruction $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$0, and predicts an action chunk $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$1 (Torne et al., 4 Mar 2026). The instantiated system is described as a $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$2 / pi0.5-like VLA system, specifically referred to as $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$3-MEM, initialized from a Gemma3-4B VLM backbone. Input observations include up to four camera streams, images at $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$4 px per stream, and proprioceptive robot state. The action model uses discrete FAST action token prediction and a flow-matching action expert with 860M parameters, and the paper notes that gradients do not flow from the action expert into the VLM backbone (Torne et al., 4 Mar 2026).

The significance of this decomposition is that memory is not treated as passive conditioning context. Long-term language memory is maintained explicitly by the planner-like high-level module, whereas short-horizon visual memory is consumed by the control-oriented low-level module. This is a strong separation of roles: semantic progress tracking at one level, dense visuomotor adaptation at another.

3. Short-horizon video memory

MEM’s short-term memory is implemented by an efficient video ViT. Instead of encoding each past frame independently and passing all tokens to the VLA backbone, the paper modifies the original single-image ViT into a video encoder that patchifies all frames separately, performs standard spatial attention within each frame, augments every 4th layer with causal temporal attention across timesteps for corresponding spatial patches, and then drops tokens from past timesteps in upper layers, retaining only the representation for the current timestep (Torne et al., 4 Mar 2026). Temporal information is therefore folded into the current-frame representation, so the VLA backbone sees about the same token count as in a standard single-frame model.

The formal attention construction introduces a temporal position embedding $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$5 at layer $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$6: $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$7 and reuses the pretrained ViT query, key, and value projections: $$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$8 The encoder factorizes full spatiotemporal attention from

$$\pi(a_{t:t+H},\,l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_\text{LL}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\, \pi_\text{HL}(l_{t+1}, m_{t+1} \mid o_t, m_t, g),$$9

to

$K \ll T$0

where $K \ll T$1 is the number of spatial patches per frame and $K \ll T$2 is the number of timesteps (Torne et al., 4 Mar 2026).

This module is parameter-efficient: it adds no new learnable parameters compared to the original single-image ViT, changes only the attention pattern, adds fixed sinusoidal temporal embeddings, and can be initialized directly from pretrained ViT weights. In pretraining, the model uses 6 observations total, 5 past plus current, with 1-second stride. During post-training and inference, the horizon is expanded to up to 18 frames, spanning up to 54 seconds of observation memory (Torne et al., 4 Mar 2026). The paper’s broader claim is that effective short-term memory in robotics must preserve geometry, appearance, and temporal structure, but only over a limited window where raw visual context remains computationally practical.

4. Long-horizon language memory

The long-term memory $K \ll T$3 is a natural-language summary of past semantic events. It is explicitly maintained by the high-level policy and updated over time (Torne et al., 4 Mar 2026). The paper illustrates this with cumulative semantic updates such as moving from “I placed a plate in the cabinet and moved to the counter” to “I placed a plate in the cabinet, moved to the counter, and picked up a bowl.” The system is trained to compress and discard irrelevant detail rather than merely append everything; for example, “I put a light green bowl, a dark blue bowl and a bright yellow bowl into the top right cabinet” may be compressed to “I placed three bowls in the top right cabinet” (Torne et al., 4 Mar 2026).

Training supervision for this language memory is constructed automatically. Given a robot episode with subtask language annotations $K \ll T$4 and an indicator of whether each subtask execution succeeded or failed, the authors feed the subtask sequence and success/failure information to an off-the-shelf pretrained LLM, prompt it to summarize all previous information that remains relevant for future task execution, and use the resulting summary as the supervision target for the transition $K \ll T$5 (Torne et al., 4 Mar 2026). The memory update process therefore depends on language supervision derived from subtask-level annotations and an external LLM summarizer.

A central comparison is against a naive language memory that simply concatenates prior subtask instructions. The paper reports that this performs worse because of train-inference distribution shift: demonstrations used in training are near-optimal, so each subtask typically appears once, whereas at test time the robot may fail multiple times, yielding repeated subtask attempts such as

$K \ll T$6

MEM avoids this by updating memory only when semantically appropriate; failed attempts may not change $K \ll T$7 at all until success occurs (Torne et al., 4 Mar 2026). This suggests that long-horizon language memory in embodied control is not merely a textual log, but a compression mechanism for persistent task state.

5. Tasks, adaptation, and empirical behavior

The MEM paper organizes its evaluation around three questions: whether memory can solve tasks requiring memory up to 15 minutes, whether memory improves in-context adaptation even in shorter tasks, and how MEM compares to prior memory architectures (Torne et al., 4 Mar 2026). The long-horizon benchmarks include recipe setup and clean up kitchen. In recipe setup, the robot is given a natural-language recipe prompt specifying required ingredients and cookware, their initial locations, and target placements; it is trained on 42 recipes and evaluated on 5 recipes, with unseen kitchens and unseen objects. In clean up kitchen, the robot must wipe counters, dry them, throw towels away, put food items into the fridge, move dishes to cabinets, wash dishes with soap and water, and place washed dishes in the dish rack; episodes average about 8 subtasks (Torne et al., 4 Mar 2026).

The in-context adaptation tasks are designed to isolate short-term corrective memory. One task requires chopstick pick-up under out-of-distribution table height, so the policy must adjust grasp height after mis-grasps. Another requires opening a refrigerator whose hinge side is visually ambiguous; an episode is counted successful if the door opens in $K \ll T$8 grasps (Torne et al., 4 Mar 2026). The paper reports that with correction data, MEM policies learn to adjust grasp height after a failed chopstick pickup and to switch door-opening strategy after trying the wrong refrigerator-opening direction. The no-memory baseline cannot do this intelligently because it cannot remember which strategy was already attempted (Torne et al., 4 Mar 2026).

Across these evaluations, the paper reports several qualitative findings. MEM substantially improves long-horizon task performance, and both video memory and language memory are necessary. Without video memory, policies can get stuck indefinitely during tasks such as washing or wiping because they lack recent temporal context. Without language memory, policies are poorer at remembering completed recipe steps, tracking which objects have already been fetched or stowed, remembering to close doors or cabinets, and tracking cleanup progress over long intervals (Torne et al., 4 Mar 2026). The paper also reports that Pool Memory can help on simple memory tasks but degrades on tasks needing richer long-term tracking, and that Proprio Memory helps when only robot-state history matters but fails when environment-state memory is needed. A Posttrain-only MEM variant performs noticeably worse than full MEM, suggesting that memory use itself must be learned during broad pretraining, not merely attached later. At the same time, MEM is reported to match state-of-the-art no-memory VLA performance on dexterous tasks that do not require memory (Torne et al., 4 Mar 2026).

6. Related architectures and open directions

MEM sits within a rapidly diversifying landscape of embodied-memory architectures. Some systems emphasize global allocentric structure, some foreground image-based episodic recall, and others make graph-based retrieval or affordance-aware selection primary.

System	Memory organization	Embodied use
Mem2Ego (Zhang et al., 20 Feb 2025)	frontier map, landmark semantic memory, visitation memory	ObjectNav via global-to-ego reasoning
3D-Mem (Yang et al., 2024)	Memory Snapshots and Frontier Snapshots	exploration and embodied reasoning
3DLLM-Mem (Hu et al., 28 May 2025)	working memory tokens and episodic memory	spatial-temporal reasoning and actions
eMEM (Rasheed et al., 2 Jun 2026)	observation, episode, gist, entity with SQLite, HNSW, and R-tree	searchable by meaning, space, and time
SpatialMem (Zheng et al., 21 Jan 2026)	unified 3D memory with metric anchoring and hierarchical object nodes	language-guided navigation and object retrieval

These systems illuminate complementary design points. Mem2Ego preserves geometry in a frontier map, stores landmark descriptions anchored to global coordinates, and projects global cues into the current panorama rather than reasoning over map text alone (Zhang et al., 20 Feb 2025). 3D-Mem stores explored regions as image-based Memory Snapshots and unexplored regions as Frontier Snapshots, explicitly coupling recall with information-seeking exploration (Yang et al., 2024). eMEM unifies observation nodes, episode nodes, gist nodes, and entity nodes behind a multi-index architecture with SQL ITE for structured storage, hnswlib for semantic search, and an R-tree for spatial queries (Rasheed et al., 2 Jun 2026). SpatialMem reconstructs metrically scaled indoor environments from casually captured egocentric RGB video, uses walls, doors, and windows as first-layer anchors, and links evidence patches, visual embeddings, and two-layer textual descriptions to 3D coordinates for compact storage and fast retrieval (Zheng et al., 21 Jan 2026). Affordance RAG, meanwhile, builds an Affordance-Aware Embodied Memory spanning affordance, instance, view, zone, area, and building levels, and reranks semantically matched candidates with affordance scores so that memory supports executability as well as recognition (Korekata et al., 22 Dec 2025).

Other lines of work foreground mechanisms that MEM itself does not explicitly include. Memo interleaves periodic summarization tokens with policy inputs and can be interpreted as inducing a temporal hierarchy from raw recent observations to accumulated segment summaries (Gupta et al., 22 Oct 2025). MemCtrl uses a trainable memory head $K \ll T$9 as a gate to determine which observations or reflections to retain or discard during exploration, emphasizing online write-time pruning under bounded compute (Dorbala et al., 28 Jan 2026). MEMENTO shows that personalized embodied assistance requires memory not only for object semantics but also for user patterns and multi-memory composition, with even GPT-4o showing a 30.5% performance drop on joint-memory tasks (Kwon et al., 22 May 2025). This suggests that future MEM systems may need stronger mechanisms for selective retrieval, pattern abstraction, and compositional reuse than episodic replay alone.

The MEM paper itself explicitly notes that future work should extend memory beyond an episode to weeks or months of deployment (Torne et al., 4 Mar 2026). In light of the broader literature, a plausible implication is that a fuller multi-scale embodied memory would combine MEM’s mixed-modal control-oriented decomposition with explicit spatial grounding, hierarchical retrieval, write-time pruning, and longer-horizon consolidation. Such a system is not yet realized in the cited work, but the current literature already defines its main ingredients.