Multi-Modal Multi-Turn Memory Benchmark

• Multi-modal multi-turn memory benchmarks are evaluation frameworks that measure an AI’s ability to retain and integrate information from diverse modalities over multiple dialogue turns.
• They challenge systems to perform long-range recall, cross-modal grounding, and retrieval of fragmented data across sequential interactions.
• These benchmarks drive innovations in memory-augmented architectures, guiding improvements in real-world agentic AI deployments.

The multi-modal multi-turn memory benchmark is a formal evaluation framework that assesses the ability of AI systems, especially LLMs and vision-LLMs (VLMs), to acquire, retain, and utilize information distributed across multiple modalities (typically text, images, and sometimes video/audio) over extended, multi-turn interactions. These benchmarks are designed to probe long-range dialogue memory, cross-modal grounding, contextual reasoning, and dynamic retrieval across sequential or fragmented sources, with applications spanning dialogue, agentic assistance, embodied AI, instruction following, and robotic planning.

1. Motivation and Scope of Multi-Modal Multi-Turn Memory Benchmarks

Multi-modal multi-turn memory benchmarks arose to fill several key gaps left by traditional single-turn, text-only, or local-context vision-language evaluations. Real-world agent deployments—virtual assistants, embodied agents, wearable-device interfaces, and collaborative systems—must resolve references and integrate information spread across long temporal horizons and diverse modalities. Text-based benchmarks fail to assess visual memory, and static multimodal datasets overlook long-term accumulation, sequence reasoning, and memory-based adaptation.

Core motivations include:

• Contextual linkage across turns: Many benchmarks, such as MMMB (Tong et al., 15 Oct 2025), MT-Video-Bench (Pan et al., 20 Oct 2025), and Mem-Gallery (Bei et al., 7 Jan 2026), enforce information dependencies across multiple dialogue rounds, images, or videos.
• Cross-modal retrieval and grounding: Robust evaluation of when and how an agent retrieves relevant visual or textual evidence, especially when crucial signals appear only in one channel (see DMV-Bench (Tang et al., 25 Jun 2026) for visual-only cues).
• Realistic fragmentation and distributed memory: Benchmarks like SMMBench (Chai et al., 15 May 2026) and M³Exam (Huang et al., 5 Jun 2026) go further, distributing evidence over separate sources or sessions, requiring agents to perform non-trivial retrieval and composition.

These benchmarks thus systematically characterize the agent's ability to operate over long and heterogeneous multi-modal trajectories, explicitly exposing deficits in memory retention, reasoning, and retrieval.

2. Benchmark Constructions and Design Dimensions

Multi-modal multi-turn memory benchmarks differ in scenario, modality coverage, data generation, and evaluation protocol.

Benchmark Construction Dimensions:

• Modalities: Most focus on text-plus-image (e.g., MMMB (Tong et al., 15 Oct 2025), Mem-Gallery (Bei et al., 7 Jan 2026), ContextQFormer/TMDialog (Lei et al., 29 May 2025)); some include audio (InteractiveOmni (Tong et al., 15 Oct 2025)), video (MT-Video-Bench (Pan et al., 20 Oct 2025)), tables and charts (M³Exam (Huang et al., 5 Jun 2026)), or real-world robotic sensor data (RoboMemArena (Lei et al., 11 May 2026)).
• Multi-turn Dialogue Lengths: Dialogue sequences typically range from 3 to 20+ turns (TMDialog, MMMT-IF (Epstein et al., 2024), Mem-Gallery), or longer in multi-session settings (Mem-Gallery, H2HMem (Zhu et al., 8 Jun 2026), M³Exam).
• Data Generation: Datasets are either fully synthetic (MTIF (Shen et al., 6 Sep 2025), MT-Video-Bench), programmatically constructed and annotated (MMMB, SMMBench), or based on human-authored conversations (Mem-Gallery, H2HMem, M³Exam).
• Task Categories: Benchmarks typically decompose evaluation into targeted subtasks, including factual retrieval, cross-modal grounding, memory reasoning, conflict detection, action prediction, and instruction following (see Section 3).

Example Data/Task Structure (from MMMB (Tong et al., 15 Oct 2025)):

Task Type	Memory Required	Example Question
Text Memory	Recall of past text	"What did the user say in turn 3?"
Image Memory	Recall of past image detail	"What color was the car in turn 3?"
Mixed Memory	Joint text+image integration	"In the scene from turn 2, what did...?"

Annotation: Many benchmarks enforce strict programmatic answer checking via closed-form/multiple-choice (MMMB, SMMBench) or code-execution-based metrics (MMMT-IF (Epstein et al., 2024)). Others use semantic or LLM-as-judge rubrics (H2HMem, Mem-Gallery).

3. Evaluation Taxonomy: Task Structure and Metrics

Evaluation frameworks classify subtasks and corresponding metrics as follows:

A. Memory Access and Retrieval

• Factual/Textual Retrieval: Direct recall of specific details from past turns (Mem-Gallery: F1/EM/BLEU-1; MMMB: accuracy).
• Visual/Multimodal Retrieval: Locate or identify visual objects/entities previously observed (Mem-Gallery: VS; DMV-Bench TSR; H2HMem: UPR, CRR).

B. Memory Reasoning and Composition

• Temporal/Multi-turn Reasoning: Cross-episode/event ordering or logical relationships (MT-Video-Bench: MR, TR; RoboMemArena: sequence, counting).
• Cross-modal Integration: Synthesis of evidence from multiple sources/modalities (SMMBench: Multi-Hop QA; M³Exam: MR, FM; H2HMem: CRR, MCR).
• Conflict Detection and Knowledge Resolution: Recognition and resolution of contradictions or corrections (Mem-Gallery: KR, CD; H2HMem: CD, KR).

C. Adaptive Memory and Application

• Instruction Following: Adherence to cumulative or hidden constraints (MMMT-IF: PIF score; CAMVR/MTIF: IFSR).
• Test-time Adaptation/Action Prediction: Application of new memory or function-call generation (SMMBench: function call; Mem-Gallery: TTL).

Metrics:

• Lexical: F1, BLEU, EM.
• Retrieval: Recall@K, Precision@K, Hit@K, Task Success Rate (RoboMemArena, DMV-Bench).
• Semantic: LLM-as-judge ratings (0-1 scale).
• Instruction Compliance: PIF, PIF-N-K metrics (MMMT-IF).
• Efficiency: Index build time, memory bank size vs. retention (DMV-Bench, M³Exam).

Performance degradation is typically reported as function of turn distance, number of images to recall, or cross-session reach (MMMB, Mem-Gallery, DMV-Bench).

4. Representative Benchmarks and Key Findings

4.1. Memory Span and Modal Alignment

MMMB (Tong et al., 15 Oct 2025):

• Probes memory retention over 15-turn dialogues with image/text state evolution. InteractiveOmni-4B achieves 70.71% (text), 30.39% (image), 59.68% (mixed) final-turn accuracy—strongly outperforming open-source comparators.

Mem-Gallery (Bei et al., 7 Jan 2026):

• Multi-session, multi-modal memory with 3,962 turns and 1,711 QA pairs. MuRAG (multimodal RAG) achieves F1≈0.70 overall; visual search F1 surpasses 0.88, highlighting the necessity of explicit cross-modal retrieval.

H2HMem (Zhu et al., 8 Jun 2026):

• Human-human, multi-party benchmarks with rich speaker and visual context. Weighted LLM-Judge scores top out at 0.5757 (A-Mem baseline), with recall substantially exceeding precision and cross-modal and multi-party scenarios remaining particularly difficult.

4.2. Memory Drift, Long-Horizon Reasoning, and Source Distribution

M³Exam (Huang et al., 5 Jun 2026):

• 5,150 QA over 239 multi-session user-agent dialogues, with artifact diversity (text, images, charts, PDFs). M³Proctor improves accuracy by ~13%, and reduces computational cost by 70%+ relative to naive multimodal retrieval via modality-aware chunking and staged retrieval.

SMMBench (Chai et al., 15 May 2026):

• For source-distributed memory reasoning (evidence over ≥2 sources), retrieval-augmented methods (HMRAG) reach only 0.4933 overall accuracy (golden evidence reference: 0.7473). Function-call prediction is notably poor (0.11 EM), and over 60% of failures are due to missing distributed evidence.

DMV-Bench (Tang et al., 25 Jun 2026):

• In a strictly visual e-commerce setting with L₂-leakage contract, the DualMem architecture achieves >80% long-horizon success (J=5–15) by leveraging parallel visual/verbal codes, dramatically outperforming monolithic or caption-based memories.

4.3. Instruction Following and Memory-Reasoning Coupling

MMMT-IF (Epstein et al., 2024):

• With 990 turns requiring global instruction compliance, the PIF metric reveals performance drops from 0.81 (turn 1) to 0.64 (turn 20) for frontier models. Repetition (PIF-4-4) lowers full compliance to ≈11%. Appending all instructions at the end of context boosts PIF by >20 points, demonstrating that retrieval—not only execution—dominates task difficulty.

CAMVR/MTIF (Shen et al., 6 Sep 2025):

• Dynamic memory mechanisms (VCMU + AVFG) improve multi-turn instruction-following success rates by ~3%–4% over strong baselines, with ablation studies confirming the benefit of explicit cross-modal memory over conventional attention.

5. Architectural and Methodological Innovations

Emerging memory systems have incorporated specialized mechanisms to address long-horizon, cross-modal retrieval and reasoning:

• External, Slot-Based, and Dual-Coding Memories: External FIFO/buffered memories (ContextQFormer (Lei et al., 29 May 2025), CAMVR (Shen et al., 6 Sep 2025)), dual coding (DMV-Bench (Tang et al., 25 Jun 2026)), and keyframe banks (PrediMem in RoboMemArena (Lei et al., 11 May 2026)) focus on persistent, query-tunable embedding banks across interaction histories.
• Memory-Augmented RAG approaches: RAG, MuRAG, and HMRAG, which retrieve over fused or graph-based multi-modal embeddings (Mem-Gallery (Bei et al., 7 Jan 2026), SMMBench (Chai et al., 15 May 2026)).
• Hierarchical and Agentic Memory: Hierarchical note-taking, reflection-based, and episodic memory systems (H2HMem, Mem-Gallery) explicitly manage and index events, actors, and modalities.
• Modality-aware Retrieval and Cost-Aware Cascading: M³Proctor uses a modality-detection cascade to minimize raw artifact loading and reduce token cost without accuracy loss (Huang et al., 5 Jun 2026).

Ablation and scaling studies (DMV-Bench, RoboMemArena) confirm that models with explicit visual codes or predictive/cached memory outperform those using generic text-augmented retrieval, particularly as bank size and horizon grow.

6. Failure Modes, Bottlenecks, and Future Directions

Despite advances, multi-modal multi-turn memory benchmarks consistently reveal the following bottlenecks:

• Cross-modal and source-distributed alignment remains unsolved: Modal misalignment, missing cross-source composition, and recency/authority confusion dominate error profiles (>40–60%, SMMBench, H2HMem).
• Instruction retrieval and execution decouple: Most failures in long-horizon compliance are rooted in instruction retrieval, not combinatorial execution logic (MMMT-IF).
• Efficiency and scaling: Explicit memory/graph/fusion systems significantly increase storage and retrieval costs (3–4× over text-only; see Mem-Gallery, M³Exam).
• Human-level gap persists: Top-performing LLMs and agentic memory architectures are 20–40 points below human accuracy or compliance in most metrics.
• Need for structured, cross-modal, and conflict-aware memories: Multiple works (H2HMem, Mem-Gallery) advocate joint graph memory, timestamp/authority tracking, and dynamic summarization for scalable performance.

Future research directions focus on memory modules that jointly encode, retrieve, and reason over persistent multi-modal facts, improved cross-source and cross-session linking, conflict detection modules, and efficient memory compression pipelines to approach agent deployment constraints.

7. References to Foundational Benchmarks and Methods

• MMMB: Multi-modal Multi-turn Memory Benchmark—Text+image dialogue, closed-form QA, memory span analysis (Tong et al., 15 Oct 2025).
• H2HMem: Human-to-Human Multimodal Memory Benchmark—Dyadic/multi-party, images+text, recall and reasoning breakdown (Zhu et al., 8 Jun 2026).
• CRAG-MM: Retrieval-augmented, egocentric memory, web+image KG, RAG protocols (Wang et al., 30 Oct 2025).
• ContextQFormer/TMDialog: Query-based memory module, FIFO turn memory, context fusion (Lei et al., 29 May 2025).
• RoboMemArena: Robotic manipulation, keyframe-based memory, dual-system architecture, real-world tasks (Lei et al., 11 May 2026).
• MT-Video-Bench: Video dialogue, perceptivity/interactivity tasks, memory accuracy and cross-scene drop (Pan et al., 20 Oct 2025).
• SMMBench: Source-distributed multimodal samples, conflict and preference composition, action call accuracy (Chai et al., 15 May 2026).
• Mem-Gallery: Multi-session dialogue, memory reasoning/adaptation, MuRAG/UniversalRAG/agentic memory comparison (Bei et al., 7 Jan 2026).
• M³Exam: Persona-driven multimodal multi-session QA, modality-aware retrieval, efficiency benchmarks (Huang et al., 5 Jun 2026).
• MMMT-IF: Programmatic instruction-following metric, multi-turn image QA with scattered instructions (Epstein et al., 2024).
• DMV-Bench: Chain-of-session visual memory, no text cues, dual-code visual+verbal memory design (Tang et al., 25 Jun 2026).

These benchmarks and architectures provide rigorous, granular measurement of model progress, surface latent failure modes, and direct practical innovation toward memory-centric, real-world agentic AI.