Multimodal Memory Agent (MMA)

• Multimodal Memory Agents are AI systems that use explicit, updatable memory stores across text and visual data to support long-horizon planning and decision-making.
• They employ layered architectures and hybrid retrieval methods, iteratively reasoning over stored episodic, semantic, and visual memories to improve task performance.
• Empirical studies highlight significant gains in accuracy and efficiency while also identifying challenges in multimodal evidence integration and memory reliability.

Searching arXiv for papers on multimodal memory agents to ground the article in the current literature. {"query":"multimodal memory agent long-term multimodal memory arXiv", "max_results": 10, "sort_by": "submittedDate"} Multimodal Memory Agent (MMA) denotes a class of long-horizon AI agents that maintain explicit, updatable memory over multimodal experience rather than relying solely on a model’s parametric weights or raw context replay. Across recent formulations, an MMA is characterized by an external memory store or hierarchy, memory-control policies for writing and maintenance, retrieval mechanisms that operate over text and visual evidence, and a reasoning layer that conditions decisions on retrieved memory during question answering, action selection, or recommendation (Liu et al., 28 May 2026). The term also names a specific reliability-aware framework that augments retrieval-augmented generation with dynamic memory confidence estimation and abstention (Lu et al., 18 Feb 2026). In the broader literature, MMAs appear in long-video reasoning, personalized dialogue, GUI control, tool selection, recommendation, translation, and human–human interaction settings, where persistent memory must support selective retention, temporal consistency, multimodal grounding, and downstream use under extended horizons (Yeo et al., 2 Dec 2025, Feng et al., 7 Feb 2026, Zeng et al., 18 May 2026, Lu et al., 9 Jul 2025, Zhu et al., 8 Jun 2026).

1. Concept and Scope

The general MMA paradigm formalized in WorldMemArena defines a Multimodal Memory Agent as an embodied or GUI-driven agent powered by a LLM that observes a partially visible, evolving world through text, images, tool feedback, and related channels; selects actions; receives feedback; and maintains an explicit, updatable memory store for long-horizon planning and decision-making (Liu et al., 28 May 2026). Within this formulation, the agent must write only memory items that support future decisions, maintain and revise memory as facts change, retrieve minimal and relevant evidence at decision time, and faithfully use that evidence to produce correct answers or actions.

This general definition encompasses multiple concrete memory architectures. TeleMem presents an MMA as a unified long-term and multimodal memory system for coherent user profiles and long-term video-grounded reasoning (Chen et al., 12 Dec 2025). WorldMM defines an MMA for long-video reasoning that maintains episodic, semantic, and visual memories with an adaptive retrieval agent (Yeo et al., 2 Dec 2025). NS-Mem defines an MMA as a long-term neuro-symbolic memory framework with episodic, semantic, and logic-rule layers (Jiang et al., 16 Mar 2026). M2A defines an MMA as a dual-layer hybrid memory system for personalized multimodal interaction (Feng et al., 7 Feb 2026). MementoGUI treats long-horizon GUI control as an online memory-control problem with working and episodic memory governed by learned operators (Zeng et al., 18 May 2026).

A recurrent distinction in the literature separates MMAs from simpler long-context or text-only memory systems. TeleMem identifies deficiencies in standard RAG, specifically unreliable updating and refinement of stored memories, schema-driven hallucinations, inefficient write operations, and minimal support for multimodal reasoning (Chen et al., 12 Dec 2025). WorldMemArena similarly argues that existing benchmarks often reduce memory to static recall and collapse visual observations into captions, which obscures failures in writing, maintenance, retrieval, or use (Liu et al., 28 May 2026). This suggests that the MMA concept is defined as much by explicit memory lifecycle management as by multimodal representation.

2. Core Architectural Patterns

A common architectural pattern in MMA research decomposes memory into multiple layers or stores with distinct operational roles. TeleMem separates profile memory from event memory and augments them with a multimodal memory module for video-derived captions and object memory (Chen et al., 12 Dec 2025). M2A couples an immutable RawMessageStore with an editable SemanticMemoryStore, allowing different retrieval granularities and provenance linking from semantic entries back to raw message ranges (Feng et al., 7 Feb 2026). WorldMM uses three complementary memories: episodic memory for factual events across multiple temporal scales, semantic memory for evolving conceptual knowledge, and visual memory for feature embeddings and timestamp-indexed frames (Yeo et al., 2 Dec 2025). NS-Mem employs a three-layer architecture comprising episodic, semantic, and logic layers, linked by cross-layer edges (Jiang et al., 16 Mar 2026).

Another recurring pattern is separation between memory management and task execution. In M2A, ChatAgent decides when to retrieve or update memory, while MemoryManager translates these requests into retrieval or write operations over the dual-layer memory bank (Feng et al., 7 Feb 2026). In MementoGUI, MementoCore sits alongside a frozen GUI action model and implements four memory-control operators—step processing, working-memory compression, episodic writing, and episodic selection—without finetuning the backbone action model (Zeng et al., 18 May 2026). In ToolMem, the memory module stores past tool-use experiences, while the agent uses retrieved capability memories to predict tool performance and select a tool for the current task (Xiao et al., 8 Oct 2025).

A third shared design choice is iterative or agentic reasoning over retrieved memory rather than one-shot prompting. M3-Agent alternates between [Search] and [Answer] actions through multi-turn iterative reasoning over its multimodal memory graph (Long et al., 13 Aug 2025). TeleMem uses a ReAct-style reasoning loop in which an LLM interleaves tool calls such as video.retrieval, video.rag, and video.qa until it emits finish (Chen et al., 12 Dec 2025). WorldMM’s retrieval agent iteratively chooses a memory source and temporal granularity until sufficient evidence is gathered (Yeo et al., 2 Dec 2025). These designs indicate that MMA architectures generally treat memory as an active substrate for control, not merely as a passive retrieval database.

3. Memory Representation, Writing, and Maintenance

Memory representation in MMAs ranges from free-form summaries to structured graphs and multimodal embeddings. TeleMem’s narrative dynamic extraction generates two role-specific free-form summaries per dialogue turn, $p_t^{user}$ and $p_t^{bot}$, stored as JSON entries with turn id, speaker, text, and embedding (Chen et al., 12 Dec 2025). Its event memory pipeline summarizes turns into snippets, retrieves related past entries, clusters their union, and uses an LLM decision prompt to emit add, delete, update, or no-op actions before writing consolidated entries back to event memory. M2A’s semantic entries contain $C_{\text{text}}$, $C_{\text{caption}}$, $C_{\text{image}}$, and pointers to raw message ranges, and each entry is indexed by dense text, sparse BM25, and cross-modal image representations (Feng et al., 7 Feb 2026).

WorldMM constructs episodic memory as knowledge graphs at multiple temporal scales $\mathcal{T}=\{t_0,t_1,\dots,t_N\}$, semantic memory as a continually updated knowledge graph, and visual memory as the union of segment-level feature embeddings and timestamp-indexed raw frames (Yeo et al., 2 Dec 2025). NS-Mem formalizes its memory system as

$$\mathcal{M}=(\mathcal{L}_{epi},\mathcal{L}_{sem},\mathcal{L}_{logic},\mathcal{E}),$$

with episodic nodes $e=(t,\mathbf{d},\mathbf{v}_e)$, semantic nodes $s=(\text{type},\text{attrs},\mathbf{v}_s)$, and logic nodes $$\mathcal{N}=(\text{id},c,\mathbf{I},\mathcal{G},\mathcal{F})$$ that encode procedural knowledge as neuro-symbolic structures (Jiang et al., 16 Mar 2026).

Writing policies vary substantially. TeleMem performs batched writing over $p_t^{bot}$0 turns: summarize, retrieve top-$p_t^{bot}$1 related entries, globally cluster the union, then consolidate with temporally ordered decision prompts (Chen et al., 12 Dec 2025). MementoGUI frames writing as online control: at each timestep, the step processor outputs a write salience score $p_t^{bot}$2, event summary $p_t^{bot}$3, ROI box $p_t^{bot}$4, and retrieval trigger $p_t^{bot}$5; if $p_t^{bot}$6, an entry containing summary, ROI, crop, and visual embedding is appended to working memory (Zeng et al., 18 May 2026). TaskMem formulates writing as a memorization policy $p_t^{bot}$7 over a sliding-window state $p_t^{bot}$8, with the action either generating a new memory entry or skipping writing (Zou et al., 29 May 2026). Omni-SimpleMem admits an event into memory only if a modality-specific novelty filter exceeds a threshold, creating a Multimodal Atomic Unit with summary, embedding, pointer, timestamp, modality tag, and graph links (Liu et al., 1 Apr 2026).

Maintenance mechanisms address redundancy, staleness, and conflict. TeleMem’s clustering-and-consolidation pipeline reduces duplication and supports delete and update operations (Chen et al., 12 Dec 2025). NS-Mem reinforces, weakens, prunes, or creates semantic nodes according to similarity thresholds $p_t^{bot}$9 and $C_{\text{text}}$0, and incrementally updates logic node indices through an exponential moving average (Jiang et al., 16 Mar 2026). WorldMM updates semantic memory via

$C_{\text{text}}$1

with $C_{\text{text}}$2 and $C_{\text{text}}$3 determined by an LLM-guided conflict-resolution prompt (Yeo et al., 2 Dec 2025). M2A supports CREATE, DELETE, and replacement-style updates over its semantic layer while preserving provenance to raw logs (Feng et al., 7 Feb 2026). The reliability-focused MMA paper treats stale or conflicting retrieved items as a first-class problem and introduces temporal decay and consensus penalties at retrieval time rather than only during memory writing (Lu et al., 18 Feb 2026).

4. Retrieval and Reasoning Mechanisms

Retrieval in MMAs is typically hybrid, multi-stage, or adaptive. TeleMem embeds all memory entries in a common vector store and retrieves top-$C_{\text{text}}$4 items, usually $C_{\text{text}}$5, from each store; for video-grounded queries, a ReAct agent combines text lookup with video.qa over raw clips (Chen et al., 12 Dec 2025). M2A uses tri-path hybrid retrieval, combining dense text similarity, BM25 sparse matching, and visual similarity, then fuses ranks with Reciprocal Rank Fusion: $C_{\text{text}}$6 (Feng et al., 7 Feb 2026). Omni-SimpleMem similarly merges dense FAISS retrieval with BM25 and augments the candidate set through knowledge-graph expansion, followed by pyramid retrieval under a token budget (Liu et al., 1 Apr 2026).

Long-video MMAs often require adaptive selection across temporal scales and modalities. WorldMM’s retrieval agent $C_{\text{text}}$7 iteratively emits either STOP or a pair $C_{\text{text}}$8, where $C_{\text{text}}$9, allowing the system to choose both a memory source and a query granularity (Yeo et al., 2 Dec 2025). Episodic retrieval uses Personalized PageRank over multi-scale graphs, followed by cross-scale LLM reranking. Semantic retrieval uses PPR over the semantic graph, and visual retrieval uses cosine similarity over video features plus timestamp-based frame access when a temporal range has been identified.

In neuro-symbolic MMAs, retrieval is explicitly type-aware. NS-Mem first classifies a query into factual, constraint, or character types, then performs neural discovery with

$C_{\text{caption}}$0

followed by type-aware reranking and, when needed, deterministic symbolic query functions such as getProcedureWithEvidence, queryStepSequence, and aggregateCharacterBehaviors (Jiang et al., 16 Mar 2026). This architecture is designed to support analytical and deductive reasoning beyond vector similarity alone.

Reliability-aware retrieval constitutes a distinct strand within MMA research. In the paper titled “MMA: Multimodal Memory Agent,” each retrieved item $C_{\text{caption}}$1 receives a dynamic reliability score derived from source reliability, temporal decay, and network consensus (Lu et al., 18 Feb 2026). Temporal decay is defined as

$C_{\text{caption}}$2

and network consensus is computed over the retrieved neighborhood using similarity-weighted agreement or contradiction. Retrieved evidence is then reweighted in the prompt, and the system abstains with “Unknown” if aggregated support is insufficient. This formulation extends MMA beyond memory retention and retrieval into epistemic control.

5. Application Domains and Representative Systems

The MMA literature spans several application classes, each emphasizing different memory functions.

System	Domain	Distinctive memory design
TeleMem	Long-term dialogue and video-grounded role-play	Narrative profile memory, structured event writing, object memory, ReAct multimodal read (Chen et al., 12 Dec 2025)
MementoGUI	Long-horizon GUI agents	Learned working-memory compression and episodic selection with ROI-level evidence (Zeng et al., 18 May 2026)
WorldMM	Long-video QA	Episodic, semantic, and visual memories with adaptive retrieval (Yeo et al., 2 Dec 2025)
NS-Mem	Multimodal reasoning	Episodic, semantic, and logic rule layers with symbolic query functions (Jiang et al., 16 Mar 2026)
M2A	Personalized multimodal interaction	Dual-layer RawMessageStore plus SemanticMemoryStore with tri-path retrieval (Feng et al., 7 Feb 2026)
ToolMem	Tool-using agents	Capability memory over past tool-use experiences (Xiao et al., 8 Oct 2025)
MMEACR	Recommendation	Dual-track reasoning memory and embedding memory (Cong et al., 8 Jul 2026)
ViDove	Translation and subtitling	Shared short-term and long-term multimodal memory across agents (Lu et al., 9 Jul 2025)

In long-video reasoning, M3-Agent, WorldMM, and TeleMem represent three distinct MMA design lines. M3-Agent uses an entity-centric multimodal memory graph with episodic and semantic entries and a reinforcement-learned controller that issues search or answer actions (Long et al., 13 Aug 2025). WorldMM organizes memory by function and temporal scale (Yeo et al., 2 Dec 2025). TeleMem focuses on memory consolidation efficiency and multimodal tool use within long-term role-play settings (Chen et al., 12 Dec 2025).

In human-centered interaction, M2A addresses evolving personalization by coupling raw logs with editable semantic observations (Feng et al., 7 Feb 2026), while H2HMem provides a benchmark for agents that passively observe dyadic and multi-party conversations with text and image streams and must answer recall, reasoning, and application queries from the final memory store (Zhu et al., 8 Jun 2026). This suggests a widening MMA scope from single-user assistant memory toward multi-party discourse memory with anaphora, deixis, and conflict.

In control settings, MementoGUI frames memory as online control over long-horizon interface interactions (Zeng et al., 18 May 2026). ToolMem treats memory as accumulated knowledge about tool strengths and weaknesses, enabling retrieval-conditioned tool performance prediction and selection (Xiao et al., 8 Oct 2025). MMEACR adapts the MMA idea to recommendation through User and Item Memory Agents that maintain persistent textual narratives, alongside a decoupled embedding memory for fine-grained multimodal matching (Cong et al., 8 Jul 2026). ViDove distributes a shared multimodal memory across specialized agents—Visual, Auditory, Translation, Proofreader, and Editor—in a translation pipeline (Lu et al., 9 Jul 2025).

6. Evaluation, Empirical Findings, and Open Issues

A central empirical theme is that better storage does not automatically yield better downstream performance. WorldMemArena explicitly reports that better memory writing and storage do not guarantee better performance, that multimodal memory still struggles to fully use visual evidence, that systems are unstable across domains, and that harness memory remains flexible but costly and less reliable (Liu et al., 28 May 2026). Its four-stage lifecycle—write, maintain, retrieve, use—provides stage-level metrics such as Memory Recall, update handling, interference rejection, Recall@K, NDCG@K, and final QA quality, making it possible to localize memory failures rather than collapsing them into end-task accuracy.

Representative benchmark results show gains from richer memory architectures but also persistent bottlenecks. TeleMem reports 86.33% QA accuracy on the ZH-4O ultra-long role-play benchmark, with +19 percentage points over Mem0’s 70.20%, 43% fewer tokens, and $C_{\text{caption}}$3 faster end-to-end memory read/write throughput (Chen et al., 12 Dec 2025). WorldMM reports 69.5% average accuracy across five ultra-long video QA benchmarks, an absolute +8.4% over the prior best memory-based baseline, with ablations showing that multi-scale episodic memory improves temporal grounding, semantic memory boosts HabitInsight and RelationMap by up to +23%, visual memory lifts EntityLog and EventRecall by up to +4%, and adaptive multi-turn retrieval yields +9% on multi-step queries (Yeo et al., 2 Dec 2025). NS-Mem reports an average 4.35% improvement in overall reasoning accuracy over pure neural memory systems, with gains of up to 12.5% on constrained reasoning queries (Jiang et al., 16 Mar 2026). TaskMem reports VQA accuracy gains of 6.3%, 7.0%, and 5.3% on streaming reformulations of VideoMME, EgoLife, and EgoTempo, respectively (Zou et al., 29 May 2026).

Benchmarking studies reveal recurring weaknesses. H2HMem reports that advanced agents exhibit substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions (Zhu et al., 8 Jun 2026). In its reported results, recall tasks outperform cross-modal tasks, temporal reasoning is relatively high, whereas multimodal causal reasoning, reference evolution tracking, and conflict detection are weak; error analysis attributes roughly 45% of errors to modal misalignment, about 33% to speaker-related mistakes, about 15% to temporal confusion, and about 5% to hallucinations or other causes (Zhu et al., 8 Jun 2026). WorldMemArena likewise observes that multimodal external systems may achieve high recall yet lower end-to-end QA, indicating poor downstream use of visual evidence (Liu et al., 28 May 2026).

A distinct controversy concerns multimodal evidence itself. The reliability-aware MMA paper identifies the “Visual Placebo Effect,” defined as a failure mode in which the mere presence of an image induces overconfident answers even when no reliable evidence exists (Lu et al., 18 Feb 2026). On MMA-Bench, the baseline reaches 0.0% Type-B accuracy in Vision mode under the reported protocol, while MMA reaches 41.18%; yet in Type-D unknowable scenarios, MMA’s CoRe score drops from +0.69 in Text mode to –0.38 in Vision mode, which the paper interprets as evidence that visual input can override reliability filters (Lu et al., 18 Feb 2026). This finding complicates a common assumption that adding visual context necessarily improves grounding.

Across these studies, several research directions recur. WorldMemArena argues for mutable memory rather than append-only logs, retrieval-aware writing, first-class multimodal indexing, and evaluation beyond final QA (Liu et al., 28 May 2026). H2HMem recommends stronger vision-language fusion, adaptive memory pruning, explicit graph and index operations, coreference resolution, and temporal summarization (Zhu et al., 8 Jun 2026). NS-Mem highlights hierarchical procedure decomposition, integration with external knowledge graphs, and dynamic graph neural networks as plausible extensions (Jiang et al., 16 Mar 2026). A plausible implication is that MMA research is converging on a view of memory as an explicitly managed computational substrate whose quality depends jointly on representation, update policy, retrieval policy, and evidence-grounded use, rather than on any single memory module in isolation.