Multimodal Long-Term Conversational Memory

• Multimodal long-term conversational memory is a system that encodes, stores, and retrieves visual, auditory, and textual data to maintain extended context in dialogues.
• The methodology involves using hybrid multimodal encoders, dynamic memory buffers, and retrieval-augmented generation to support context-aware dialogue.
• Empirical evaluations across diverse benchmarks show benefits in context retention while highlighting challenges like memory degradation and token efficiency.

Multimodal long-term conversational memory refers to the computational modeling, storage, retrieval, and utilization of verbal, visual, and auditory information over extended conversational horizons, enabling artificial agents to maintain context, recall past events, reason across modalities, and adapt to evolving discourse dynamics. This capability is central for the next generation of multimodal LLMs (MLLMs), agentic assistants, and socially intelligent systems, with applications spanning multiparty dialogue, embodied agents, long-form video understanding, and real-world interactive AI (Bei et al., 7 Jan 2026, Jang et al., 31 May 2025, Chen et al., 12 Dec 2025, Long et al., 13 Aug 2025).

1. Problem Definition and Functional Framework

Multimodal long-term conversational memory is formally structured as a dynamic memory system $$\mathcal S = \langle \mathcal M, f_\theta, \Phi, \mathcal R\rangle$$ operating over sequences of multimodal observations $\mathcal O = \{o_1, \dots, o_T\}$, each $o_t = \langle v_t, s_t\rangle$ with $v_t$ visual (image, video frame) and $s_t$ textual (utterance) content. The memory state $\mathcal M_t$ accumulates atomic multimodal units $m_i$ comprising raw assets, cross-modal descriptions, and high-dimensional latent codes; these are evolved by an update operator $\Phi$ (which might add, merge, or forget entries) and retrieved via $\mathcal R$ in response to queries $q_\tau$. The MLLM then conditions its output on the working context $C_\tau$, retrieved memory $\mathcal M_{\rm ret}$, and the current query, i.e.,

$$y_\tau = \mathrm{MLLM}(C_\tau \oplus \mathcal M_{\rm ret} \oplus q_\tau)$$

where $\oplus$ denotes multi-modal concatenation (Bei et al., 7 Jan 2026). Memory organization must support efficient extraction, cross-modal reasoning, and knowledge management—tasks not satisfied by simple buffer-based context or text-only retrieval.

2. Datasets and Benchmarks

The field has produced a suite of multimodal long-term conversational datasets that probe memory over diverse modalities and extended dialog horizons:

Dataset	Modalities	# Dialogues / Sessions	Notable Features
MeMo (Tsfasman et al., 2024)	Audio, Video, Text	31 hours / 45-min sessions	Memory-annotated multiparty Zoom calls; first-party recall reports (immediate and delayed), facial/body keypoints, gaze, prosody
Mem-Gallery (Bei et al., 7 Jan 2026)	Images, Text	240 conversations (3,962 rounds)	Multi-session, QA with annotated evidence clues, rich visual–textual dependencies, task taxonomy for extraction/reasoning/management
M³C (Jang et al., 31 May 2025)	Audio, Images, Text	54,000 3-session episodes	Multi-party, structured memory linking, explicit modality alignment
MPChat (Ahn et al., 2023)	Images, Text	15,000 Reddit dialogues	Persona pairs (episodic memories) as image–sentence sets
LoCoMo (Maharana et al., 2024)	Images, Text	50 conversations (up to 35 sessions)	Very long-term (300+ turns), benchmark for QA, event summarization, MM dialogue generation
MMRC (Xue et al., 17 Feb 2025)	Images, Text	5,120 open-world	Manual QA annotation, 6 core memory abilities, analysis of recall drop over ~22 turns
M3-Bench (Long et al., 13 Aug 2025)	Video, Audio, Text	1,020 videos	Focuses on embodied/robot agents, delayed and compositional QA

These corpora enable the systematic evaluation of memory retention, multimodal fusion, and memory-driven adaptation over durations ranging from several minutes (single sessions) to months (dozens of sessions, thousands of tokens) (Bei et al., 7 Jan 2026, Maharana et al., 2024).

3. Model Architectures and Memory Organization

Prevailing systems for multimodal long-term conversational memory use hybrid architectures comprising:

• Multimodal Encoders: $f_\theta$ maps $(v, s, a)$ to joint embeddings, with backbones such as CLIP, ViT, Qwen2-VL, or BLIP-2. Audio is processed via adapters on pre-trained encoders (CLAP) (Jang et al., 31 May 2025, Chen et al., 12 Dec 2025, Long et al., 13 Aug 2025).
• Memory Store: A persistent buffer $M = \{m_1, ..., m_T\}$ grows over time, with $m_i$ containing cross-modal information (summaries, raw multimodal events, entity IDs, linking metadata) (Jang et al., 31 May 2025, Chen et al., 12 Dec 2025). Some systems organize $M$ as an entity-centric memory graph $M = (V, E)$ for deeper reference and flexible retrieval (Long et al., 13 Aug 2025).
• Memory Update/Consolidation: Batch-clustering, deduplication, and consolidation strategies compress redundant entries and allow for dynamic forgetting and merging (as in TeleMem's structured writing pipeline) (Chen et al., 12 Dec 2025). Session-boundary summarization is standard; memory linking graphs record dependencies and support pointer-based expansion on retrieval (Jang et al., 31 May 2025).
• Retrieval Mechanisms: Cosine similarity over joint multimodal embeddings is widely used (Jang et al., 31 May 2025, Chen et al., 12 Dec 2025). Retrieval-augmented generation (RAG) and ReAct-style reasoner loops combine nearest neighbor search with stepwise LLM reasoning over atomic or clustered facts (Maharana et al., 2024, Chen et al., 12 Dec 2025).
• Control Policies: Models deploy contextual gating (e.g., [RET_IMG], [RET_AUD] tokens), autonomous retrieval decision modules, and explicit action spaces for memory access during generation (Jang et al., 31 May 2025, Long et al., 13 Aug 2025).

Memory systems must scale to thousands of multimodal units; vector stores (FAISS or similar) index high-dimensional embeddings, supporting efficient approximate search even as memory grows sublinearly via clustering (Chen et al., 12 Dec 2025).

4. Evaluation Dimensions and Empirical Findings

Benchmarking frameworks such as Mem-Gallery, MMRC, M3-Bench, and LoCoMo decompose memory-oriented tasks into three core axes (Bei et al., 7 Jan 2026, Xue et al., 17 Feb 2025, Long et al., 13 Aug 2025, Maharana et al., 2024):

• Memory Extraction/Test-Time Adaptation: Factual retrieval (FR), visual search (VS), and test-time learning (TTL) score whether agents can recall explicit events or adapt to novel examples. State-of-the-art multimodal RAG methods (MuRAG, UniversalRAG) attain $>$0.67 F1 on extraction, outperforming text-only memory by $+11.9\%$ (Bei et al., 7 Jan 2026). However, test-time adaptation to new multimodal facts remains challenging.
• Memory Reasoning: Temporal reasoning (TR), visual-centric reasoning (VR), and multi-entity aggregation (MR) require integrating multiple modalities and events over long horizons. State-of-the-art reasoning performance remains limited ($\sim$0.58 F1), with error modes dominated by context truncation, temporal misattribution, and cross-modal confusion (Bei et al., 7 Jan 2026, Maharana et al., 2024).
• Memory Knowledge Management: Knowledge resolution (KR), contradiction/conflict detection (CD), and answer refusal (AR) probe agents' ability to update memory and abstain when necessary. All methods remain weak on KR and CD ($\sim$0.37–0.46 F1), indicating substantial room for improvement in dynamic memory revision (Bei et al., 7 Jan 2026). Explicit note-taking mechanisms yield significant gains on answer refusal and factual recall (Xue et al., 17 Feb 2025).

Results from MMRC further show that memory recall, information extraction, and image management abilities degrade by $\approx60\%$ from shortest to longest dialogs (4–22 turns), evidencing a systemic long-term memory decay (Xue et al., 17 Feb 2025). Conversational memory models built on session summaries and extracted "observations" show moderate mitigation, while full-context or static pooling is insufficient (Maharana et al., 2024).

5. Memory-Driven Dialogue Systems: Capabilities and Limitations

Advanced agent architectures demonstrate that persistent multimodal memory unlocks several capabilities:

• Entity-centric and cross-modal tracking: Systems such as M3-Agent build memory graphs linking entities across modalities (e.g., person–face–voice equivalence via high-confidence edges) and use them for retrieval and conflict resolution (Long et al., 13 Aug 2025).
• Iterative, memory-augmented reasoning: Closed-loop action policies combining retrieval and stepwise reasoning achieve robust performance on long-horizon, multi-hop, and delayed QA (Long et al., 13 Aug 2025, Chen et al., 12 Dec 2025).
• Multi-session, multi-party adaptation: Linking memory across sessions enables appropriate referencing of prior plans or experiences, supporting dynamic group interactions and personalized dialogue (Jang et al., 31 May 2025, Tsfasman et al., 2024).
• Handling of non-textual modalities: Proper encoding and memory retrieval for images, audio, and, in latest systems, video and object memories enable richer, more human-like situational awareness and adaptation (Chen et al., 12 Dec 2025, Jang et al., 31 May 2025).

Limitations persist:

• Token inefficiency and memory crowding: Naïvely concatenating full multimodal history severely degrades performance due to excessive context ("token crowding"), highlighting the need for principled memory management (Bei et al., 7 Jan 2026).
• Long-term memory degradation: Even with advanced methods, recall of facts, images, and reasoning chains declines steeply beyond ten sessions or several thousand tokens (Xue et al., 17 Feb 2025, Maharana et al., 2024).
• Scarce audio/video immersion: Most systems rely on captions as proxies for non-textual events, exposing failure modes in settings with ambiguous or subtle perceptual cues (Jang et al., 31 May 2025, Chen et al., 12 Dec 2025).
• Knowledge revision and contradiction detection: All benchmarks show that knowledge updates and contradiction resolution lag behind factual recall, leaving DKG/LLM agents prone to propagating outdated or conflicting memories (Bei et al., 7 Jan 2026, Xue et al., 17 Feb 2025).

6. Design Trends and Future Directions

Recent advances point to several convergent trends and open challenges:

• Structured multimodal representations: Graph/schematic memory—encoding entities, event types, and relationships—surpass flat-bag-of-embeddings, especially for reasoning and knowledge management (Bei et al., 7 Jan 2026, Long et al., 13 Aug 2025).
• Scalable and efficient retrieval: Batch-building, clustering, and hybrid parametric/non-parametric stores (profile facts vs. episodic logs) accelerate memory operations and reduce token overhead by up to 43% (Chen et al., 12 Dec 2025).
• Explicit memory policies: Gating, linking, and scoring (memory-importance/forgetting) facilitate selective retrieval and adaptive context windows, reducing both recall errors and inference cost (Jang et al., 31 May 2025).
• Integrated learning and updating: Joint multimodal pretraining, reinforcement learning with reward from LLM-based evaluators, and continual memory calibration are critical for robust, long-horizon adaptation (Long et al., 13 Aug 2025, Maharana et al., 2024).
• Evaluation expansion: Benchmarks are broadening to include real-world video, audio immersion, coordinated multi-agent conversation, and human-in-the-loop co-reference and memory validation (Long et al., 13 Aug 2025, Bei et al., 7 Jan 2026).

Future challenges include the development of schema-based and event-centric memory, advanced cross-modal reasoning modules, efficient dynamic forgetting, and extending beyond images and text to full multimodal (video, audio, embodied) memory representations. Approaches integrating adaptive context truncation, active retrieval, and memory importance scoring—possibly optimized via reinforcement learning—constitute active research frontiers (Chen et al., 12 Dec 2025, Long et al., 13 Aug 2025).

7. Relation to Human Conversational Memory and Socio-Cognitive Implications

Conversational memory research in humans reveals strong serial position effects, selective recall, and functional motivations for remembering (self-relevance, relationship-building). Datasets such as MeMo, which leverage first-party memory reports after both short and long delays, provide insights into which multimodal events are encoded and retained, offering a bridge between cognitive memory science and computational modeling (Tsfasman et al., 2024). Longitudinal analyses confirm that conversational memory patterns evolve as social relationships strengthen, with increases in group cohesion linked to shifts in what is retained and recalled. A plausible implication is that next-generation multimodal memory systems will benefit from personalization and session-aware adaptation, mirroring human memory plasticity and social dynamics (Tsfasman et al., 2024).

---

References:

• (Tsfasman et al., 2024) Introducing MeMo: A Multimodal Dataset for Memory Modelling in Multiparty Conversations
• (Ahn et al., 2023) MPCHAT: Towards Multimodal Persona-Grounded Conversation
• (Xue et al., 17 Feb 2025) MMRC: A Large-Scale Benchmark for Understanding Multimodal LLM in Real-World Conversation
• (Maharana et al., 2024) Evaluating Very Long-Term Conversational Memory of LLM Agents
• (Jang et al., 31 May 2025) Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions
• (Chen et al., 12 Dec 2025) TeleMem: Building Long-Term and Multimodal Memory for Agentic AI
• (Bei et al., 7 Jan 2026) Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
• (Long et al., 13 Aug 2025) Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory