MM-Mem: Evolving Memory in AI Agents

• MM-Mem is a family of methods that treats memory as an active entity, with explicit policies for formation, update, retrieval, and revision to improve long-horizon modeling.
• It integrates multimodal, embodied, and reinforcement learning approaches to manage memory evolution and enhance agent performance.
• Implementations show gains in retrieval accuracy and efficiency by optimizing the memory cycle of construction, evolution, and credit assignment.

Searching arXiv for papers and uses of “MM-Mem” to ground the article. arXiv search query: "MM-Mem" In recent arXiv usage, “MM-Mem” appears less as a single canonical architecture than as a family resemblance across memory-centric systems for long-horizon modeling, agent memory management, multimodal reasoning, embodied control, and explicit editable memory substrates. This suggests that the term is best treated as a shorthand for methods that elevate memory from a passive buffer to an operational object with explicit policies for formation, update, retrieval, compression, reflection, or revision (Ma et al., 13 Jan 2026, Lu et al., 18 Feb 2026, Torne et al., 4 Mar 2026, Ding et al., 8 May 2026, Li, 23 Jun 2026). Within that broad usage, the recurring technical question is how a model or agent should represent past information, decide what persists, retrieve only what is useful, and maintain performance under long horizons, distribution shift, and limited context.

1. Terminological scope and research lines

The contemporary literature uses closely related “MM-Mem”-style ideas in several partially overlapping senses.

Research line	Typical memory object	Representative mechanism
Long-horizon LLM agents	External memory store	INSERT/UPDATE/DELETE/SKIP or ADD/UPDATE/DELETE/NONE
Multimodal and embodied agents	Retrieved multimodal evidence or state-conditioned guidance	Reliability scoring, video memory, Soft-Mem
Explicit editable memory substrates	Associative matrices or transaction memories	[memo](https://www.emergentmind.com/topics/memo), forget, retrieve, rollback, trace
MemComputing	Dynamical memory as time non-locality	Continuous-time nonlinear systems

A common misconception is that MM-Mem denotes one standardized module. The surveyed works point in the opposite direction: “MM-Mem” is used generically for trainable memory-manager agents, multimodal memory layers, embodied memory compilers, version-aware memory algebras, and even broader MemComputing usage in which memory means time non-locality rather than stored text (Ma et al., 13 Jan 2026, Lin et al., 19 Mar 2026, Ding et al., 8 May 2026, Li, 23 Jun 2026, Zhang et al., 2023).

What unifies these lines is not a fixed implementation but a shared design stance. Memory is treated as an active computational layer with its own state, operations, and supervision signals. In agent settings this usually means an external store that persists beyond the context window; in explicit-memory architectures it means first-class memo and forget operations; in multimodal systems it means retaining visual, textual, or latent evidence at different granularities; and in MemComputing it means a physical dynamical system whose evolution depends on history (Lu et al., 18 Feb 2026, Torne et al., 4 Mar 2026, Li, 23 Jun 2026, Zhang et al., 2023).

2. Memory as construction, evolution, and retrieval

A central synthesis across the literature is that memory is not a single operation but a cycle. MemMA names this the “memory cycle effect,” decomposing memory into construction, retrieval, and utilization, while emphasizing both forward dependency and backward dependency between these stages (Lin et al., 19 Mar 2026). Mem-T makes the same point algorithmically by separating memory formation, memory evolution, and multi-turn retrieval within a hierarchical database (Yue et al., 30 Jan 2026).

In Mem-T, the memory state is explicitly typed:

$$\mathcal{M}_t = \langle \mathcal{M}^{\text{work}}_t,\ \mathcal{M}^{\text{fact}}_t,\ \mathcal{M}^{\text{exp}}_t,\ \mathcal{M}^{\text{raw}}_t\rangle.$$

Working memory stores a short-term summary of the current session; factual memory stores declarative long-term facts; experiential memory stores procedural strategies; raw memory stores timestamped past content (Yue et al., 30 Jan 2026). This typed decomposition is conceptually close to MemBench’s distinction between factual memory and reflective memory, where factual memory concerns explicit attributes and event details, while reflective memory concerns higher-level preferences or emotions inferred from many low-level observations (Tan et al., 20 Jun 2025).

Construction policies in this family typically extract candidate memories from the current chunk, and evolution policies decide whether each candidate should be added, updated, deleted, or ignored. Retrieval is then modeled as a policy over searches rather than a one-shot lookup. Mem-T formalizes retrieval actions as Search(r, key, topk) over different memory modules plus Finish, and executes multi-turn retrieval conditioned on previously retrieved evidence (Yue et al., 30 Jan 2026). MemMA analogously uses a Meta-Thinker to diagnose answerability and direct iterative retrieval through a Query Reasoner (Lin et al., 19 Mar 2026).

This suggests a general MM-Mem principle: memory quality depends jointly on what enters the store, how entries are reconciled over time, and how queries traverse the resulting structure. Systems that optimize only one stage often inherit bottlenecks from the others.

3. Learning memory operations and credit assignment

A major obstacle for MM-Mem-style systems is sparse and delayed supervision. Fine-Mem states the problem directly: long-horizon memory managers make many local operations, yet reward often arrives only from final downstream task performance, causing reward sparsity and poor credit assignment (Ma et al., 13 Jan 2026). Mem-T makes the same diagnosis for end-to-end optimization of memory construction and retrieval over long horizons (Yue et al., 30 Jan 2026).

Fine-Mem addresses this with two dense reward components. The Chunk-level Step Reward provides immediate supervision by asking chunk-specific question answering over the current memory after each step, while Evidence-Anchored Reward Attribution redistributes global reward back to the steps that produced the memory items actually used as evidence (Ma et al., 13 Jan 2026). The per-step reward is

$$r_t = r_{\text{EARA}}^{(t)} + r_{\text{fmt}}^{(t)} + w_1\, r_{\text{chunk}}^{(t)} + w_2\, r_{\text{comp}},$$

combining evidence-attributed global reward, formatting reward, chunk-level reward, and compression reward (Ma et al., 13 Jan 2026). Empirically, Fine-Mem reaches an average score of 0.663 versus 0.619 for Mem-$\$$ on Memalpha, and 0.664 versus 0.592 on MemoryAgentBench, corresponding to gains of +4.4 points and +7.2 points respectively (Ma et al., 13 Jan 2026).

Mem-T densifies retrieval supervision differently. Its MoT-GRPO algorithm constructs a memory operation tree whose nodes are retrieval actions paired with reasoning context, then backpropagates node-wise rewards using evidence density and descendant task performance (Yue et al., 30 Jan 2026). This turns terminal QA reward into dense supervision over intermediate search decisions. It then performs hindsight credit assignment for earlier memory-construction operations, rewarding writes that either processed ground-truth evidence or were later retrieved on successful trajectories (Yue et al., 30 Jan 2026). On LoCoMo, Mem-T reaches F1 58.65 and surpasses frameworks such as A-Mem and Mem0 by up to $14.92\%$; it also reduces inference tokens per query by $\sim24.45\%$ relative to GAM without sacrificing performance (Yue et al., 30 Jan 2026).

MemMA adds a complementary backward path. Its “in-situ self-evolving memory construction” synthesizes probe QA pairs after each session, verifies the provisional memory, and converts failures into repair actions before the memory is finalized (Lin et al., 19 Mar 2026). With a LightMem backend, MemMA improves ACC from 75.66 to 81.58, and with a Single-Agent backend from 52.60 to 84.87 (Lin et al., 19 Mar 2026). MetaMem generalizes the same theme from another angle: it learns a self-evolving “meta-memory” of knowledge-utilization rules through self-reflection and symbolic ADD/DEL/MOD operations, improving over strong baselines by over 3.6% (Xin et al., 27 Jan 2026).

Across these works, MM-Mem is not merely storage engineering. It is increasingly a reinforcement-learning and self-reflection problem in which memory operations themselves become the object of optimization.

4. Multimodal and embodied MM-Mem

The multimodal strand of MM-Mem expands the memory object beyond text. MMA, the Multimodal Memory Agent, treats each retrieved memory item as a reliability-weighted signal and computes a dynamic confidence score

$$\mathcal{C}(M_i) = \Big[w'_s S(M_i) + w'_t T(M_i) + w'_c C_{\text{con}}(M_i)\Big]_0^1,$$

where the three components are source reliability, temporal decay, and conflict-aware network consensus (Lu et al., 18 Feb 2026). This allows the agent to reweight evidence and abstain when support is insufficient. On FEVER, MMA matches baseline accuracy while reducing variance by 35.2%; on MMA-Bench it reaches 41.18% Type-B accuracy in Vision mode, whereas the baseline collapses to 0.0% under the same protocol (Lu et al., 18 Feb 2026).

MEM, “Multi-Scale Embodied Memory,” addresses a different but related problem in robot control: long-horizon tasks require both short-term dense perceptual memory and long-term abstract semantic memory (Torne et al., 4 Mar 2026). Its architecture combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory updated by a high-level policy. The factorization

$$\pi(a_{t:t+H}, l_{t+1}, m_{t+1} \mid o_{t-T:t}, m_t, g) \approx \pi_{\text{LL}}(a_{t:t+H} \mid o_{t-K:t}, l_{t+1}, g)\; \pi_{\text{HL}}(l_{t+1}, m_{t+1} \mid o_t, m_t, g)$$

makes explicit the split between low-level action conditioned on recent video and high-level subtask and memory updates conditioned on current observation, goal, and long-term language memory (Torne et al., 4 Mar 2026). The resulting policies support tasks that span up to fifteen minutes and enable in-context adaptation of manipulation strategies (Torne et al., 4 Mar 2026).

MemCompiler provides perhaps the clearest embodied formulation of MM-Mem as runtime memory delivery rather than storage. It rejects Ahead-of-time Monolithic Memory Injection and replaces it with State-Conditioned Memory Compilation, in which a Memory Compiler consumes a structured Brief State and compiles only step-relevant memory into executable guidance (Ding et al., 8 May 2026). Guidance is emitted through a text channel and a latent Soft-Mem channel that preserves perceptual information not expressible in text (Ding et al., 8 May 2026). Across AlfWorld, EmbodiedBench, and ScienceWorld, MemCompiler improves over no-memory across open-source backbones by up to +129% and reduces per-step latency by 60% (Ding et al., 8 May 2026).

Taken together, these systems indicate that multimodal MM-Mem has at least three recurrent ingredients: typed evidence, explicit reliability or state tracking, and separation of human-readable guidance from non-linguistic latent memory.

5. Explicit editable memory and version-aware operation layers

Another research line treats memory as an explicitly editable substrate rather than an opaque vector store. In the MeMo framework, memory consists of correlation matrix memories (CMMs), with the core associative matrix

$C = \sum_i k_i v_i^{\top},$

and primitive procedures memo(S,y), forget(S,y), and retrieve(S) (Li, 23 Jun 2026). The paper “Towards Version-aware Operations and Transaction Memories for Multi-layer MeMo” argues that knowledge updates should be represented not as retraining but as ordered transactions of primitive memory edits (Li, 23 Jun 2026).

This version-aware layer introduces high-level operations such as replace, obsolete, keep-history, rollback, and trace, compiled into ordered lists of primitive edits over sequence-token pairs (Li, 23 Jun 2026). Two auxiliary memories are added: a Version CMM (V-CMM), which maps version transitions to transaction handles, and a Transaction CMM (T-CMM), which stores reusable change contents and inverse programs (Li, 23 Jun 2026). The resulting system supports direct sequence-level edits, structured diff-level inputs, rollback via inverse transactions, and traceability from current answers back to the transactions that introduced them (Li, 23 Jun 2026).

A plausible implication is that this line extends MM-Mem from “memory management” to “memory governance.” The memory substrate is not only retrieved from; it is revised, versioned, audited, and rolled back.

A more distant but conceptually related usage appears in MemComputing, where MM-Mem in the broader literature refers to computing with memory as time non-locality (Zhang et al., 2023). Digital MemComputing Machines are continuous-time nonlinear dynamical systems with memory whose solutions correspond to equilibria, and the cited hardware work implements such systems using standard electronic components rather than exotic devices (Zhang et al., 2023). Here memory is not a database at all but a dynamical property of the computational medium. This widens the semantic range of MM-Mem: in one tradition it means explicit editable associative memory; in another it means physical memory embedded in system dynamics.

6. Evaluation, failure modes, and open issues

MemBench provides the most systematic benchmark-oriented vocabulary for MM-Mem. It evaluates memory mechanisms along effectiveness, efficiency, capacity, and recall, and explicitly separates factual memory from reflective memory as well as participation scenarios from observation scenarios (Tan et al., 20 Jun 2025). Its comparative results show that RetrievalMemory scales far more robustly than FullMemory or RecentMemory when histories grow from 10k to 100k tokens, while the latter suffer from context-window and recency failures (Tan et al., 20 Jun 2025). This suggests that larger raw context alone is not an adequate substitute for a memory mechanism.

The literature also documents several recurrent failure modes. In MMA, consensus can become too conservative in sparse conversational memory, even though it is beneficial in dense, conflicting settings; the paper also identifies the “Visual Placebo Effect,” in which the mere presence of visual input causes overconfident errors (Lu et al., 18 Feb 2026). Fine-Mem notes the cost of auxiliary QA construction and the limitations of BM25-based retrieval and text-only memory (Ma et al., 13 Jan 2026). MemMA identifies session-boundary assumptions, probe-generation cost, and the fact that planner guidance is advisory rather than a hard constraint (Lin et al., 19 Mar 2026). MemCompiler shows that monolithic memory injection can degrade lightweight executors below the no-memory baseline (Ding et al., 8 May 2026).

Privacy is a distinct open issue. MRMMIA studies membership inference attacks on chat-agent memory stores and shows that memory membership can be inferred even under black-box access (Chen et al., 27 May 2026). In white-box settings, the attack attains TPR@FPR1% of 96.3% on PerLTQA, 87.5% on LoCoMo, and 98.8% on MSC for Mem0; corresponding MemGPT results are 96.1%, 86.4%, and 99.0% (Chen et al., 27 May 2026). The paper’s prompt-based defense leaves AUC essentially unchanged, indicating that output policy alone is insufficient as a defense (Chen et al., 27 May 2026).

The broader trajectory of the field suggests that MM-Mem is evolving toward systems that are simultaneously more structured and more adaptive: typed stores, explicit update algebras, state-aware retrieval, dense reward shaping, multimodal evidence calibration, probe-based repair, and privacy-aware evaluation. What remains unsettled is not whether memory matters, but which combination of representation, control, supervision, and auditability yields the most reliable long-horizon behavior.