---
title: 'MMMB: Multi-Modal Multi-Turn Memory Benchmark'
url: https://www.emergentmind.com/topics/multi-modal-multi-turn-memory-benchmark-mmmb-0994feff-10b6-4a2c-89c8-d47e63d82d8c
type: topic
---

# MMMB: Multi-Modal Multi-Turn Memory Benchmark

A Multi-Modal Multi-Turn Memory Benchmark (MMMB) is a formal evaluation protocol designed to probe the long-term memory capabilities of multimodal large language models (MLLMs) in open-ended, visually grounded, multi-turn conversational tasks. The primary objective is to quantify an MLLM’s ability to encode, retrieve, and reason over accrued information—textual and visual—across extended, contextually rich dialog sequences. MMMB isolates memory-dependent performance by focusing on tasks where correct answers depend on either factual extraction, reasoning, or management of evolving multimodal information, often under severe context-length and distraction constraints. This enables diagnostic comparison across architectures, memory-augmentation strategies, and attention mechanisms.

## 1. Formal Definition and Core Objectives

An MMMB is constructed as a suite of multi-turn, multimodal dialogues, each composed of a sequence of turns $t_1,...,t_T$, where each turn includes a user prompt (text), one or more images, and, where relevant, an assistant response. Dialogues are sampled to span a range of memory dependencies:

- **Text Memory**: Correct responses depend solely on textual history.
- **Image Memory**: Correct responses depend on images (current or referenced).
- **Mixed Memory**: Correct responses require joint reasoning over historical text and images.

A canonical MMMB protocol evaluates the model’s answer to a designated “final question” in each dialogue, with the ground-truth answer strictly dependent on recalling or integrating information from previous turns. The evaluation metric is typically exact-match accuracy:
\[
\text{Accuracy} = \frac{1}{|D|} \sum_{k} \mathbb{1}[\hat{y}_k = y_k]
\]
where $D$ is the set of dialogues, $y_k$ the reference answer, and $\hat{y}_k$ the model’s output for the $k$-th dialogue’s final turn [2510.13747].

For deeper analysis, MMMB also measures **forgetting curves**—accuracy as a function of (a) the number of historical items/entities to be recalled, and (b) the distance in turns between critical evidence and the evaluation point [2510.13747]. This quantifies context-length resilience and modality-conditioned degradation.

## 2. Dataset Construction and Characteristics

Modern MMMBs employ carefully synthesized and/or real-world, human-verified dialogues to ensure dense multimodal dependencies and realistic memory demands. Key dataset characteristics include:

| Aspect          | Value/Range                       |
|-----------------|----------------------------------|
| Dialogues       | 300–5,000+ (e.g., MMMB: 300 [2510.13747]; MMRC: 5,120 [2502.11903]) |
| Turns/dialogue  | 2–22 (typical max: 15–22)        |
| Images/dialogue | 2–5+                             |
| Modalities      | Text, Images (optionally Audio/Video) |

Each dialogue group is annotated for *memory type* (“text,” “image,” “mixed”) and curated such that the dependence structure is explicit and unambiguous. Images and text are sampled or composed so that memory-related questions cannot be answered from the last turn or image alone; rather, historical context is required. Systems like Mem-Gallery [2601.03515] further annotate each QA pair with precise supporting turns (“clues”) to enable fine-grained retrieval analysis.

Recent benchmarks (e.g., Mem-Gallery, MMMT-IF [2409.18216], MMRC [2502.11903]) introduce added complexity such as:
- Cross-session dependencies
- Interleaved global instructions
- Factual updates and corrections 
- Realistic image quality variation (as in CRAG-MM [2510.26160])
- Instruction following under programmatic metrics (PIF) [2409.18216]

## 3. Evaluation Protocols and Metrics

MMMBs are evaluated using deterministic or lightly randomized protocols to ensure replicability and diagnostic power:

- **Primary Metric:** Final-turn exact-match accuracy: ratio of correct answers to total test dialogues.
- **Decomposed Memory Metrics:** Accuracy stratified by memory type (Text, Image, Mixed), memory burden (#entities to recall), and turn distance from supporting evidence.
- **Programmatic Metrics:** For instruction-following variants, the $\mathrm{PIF}$ (Programmatic Instruction Following) score quantifies the fraction of historical instructions satisfied by the response:
\[
\mathrm{PIF}(X,Y) = \frac{|\{\text{instructions in context } X \text{ that } Y \text{ satisfies}\}|}{|\{\text{instructions in context } X\}|}
\]
where $X$ is the context and $Y$ is the response [2409.18216].
- **Robustness Metrics:** $\mathrm{PIF\!-\!N\!-\!K}$ measures how consistently multiple samples from a model satisfy all instructions.
- **Auxiliary Metrics:** BLEU-1 (for n-gram overlap), F1/Precision/Recall (token-level), LLM-based scoring for open-ended responses (Mem-Gallery).

Advanced protocols may require:
- Early-stop decisions in multi-turn runs (e.g., for two consecutive failures: CRAG-MM).
- Memory “abstention” (answer refusal) for unreachable or unsupported queries (high AR precision).

## 4. Baseline Model Results and Observed Limitations

Benchmarking MLLMs on MMMB-style tasks consistently reveals substantial memory degradation as the context grows, and profound differences between memory architectures.

| Model                  | Text Mem | Image Mem | Mixed Mem | Average Accuracy |
|------------------------|----------|-----------|-----------|------------------|
| GPT-4o-mini            | 70.00    | 29.41     | 58.06     | 51.33            |
| Gemini-2.5-Flash       | 75.76    | 40.19     | 70.97     | 60.84            |
| InteractiveOmni-4B     | 70.71    | 30.39     | 59.68     | 52.47            |
| Qwen2.5-VL-7B          | 35.35    | 13.73     | 27.42     | 25.10            |

All models experience pronounced decline as either the number of items to be recalled increases or the temporal distance between evidence and query grows. Open-source models (Qwen, InternVL, etc.) underperform proprietary systems by large margins, especially on image-memory and mixed-modality tasks [2510.13747]. Multimodal retrieval-augmented systems (MuRAG, UniversalRAG) achieve the best overall scores, but still fail to exceed $0.70$ F1/accuracy on the most demanding mixed-memory cases [2601.03515].

Common failure modes include:
- Early forgetting of image features (especially with token-limited or naively concatenated memory)
- Inability to identify and reconcile updates/corrections across sessions or turns
- Hallucinated or refused answers under uncertainty
- Catastrophic context-window overflow in “unstructured” approaches

Emerging strategies such as structured external note-taking (MMRC) [2502.11903] and explicit memory annotation/retrieval augment (Mem-Gallery, MuRAG) are shown to increase performance, particularly for information extraction and memory recall subtasks.

## 5. Comparative Analysis with Related Benchmarks

The MMMB paradigm extends prior single-turn and unimodal memory benchmarks in scope and difficulty:

- **MMRC [2502.11903]:** Focuses on six abilities (IE, CR, IU, IM, MR, AR) in unconstrained human–MLLM dialogue. It diagnoses memory recall decline, error propagation, and refusal reluctance in open-ended, real-user contexts.
- **CRAG-MM [2510.26160]:** Targets retrieval-augmented generation (RAG) in egocentric, wearable scenarios; mixes image/text-web RAG with multi-turn support but emphasizes retrieval over sustained memory.
- **Mem-Gallery [2601.03515]:** Systematic, multi-session, annotated memory evaluation with dense visual and textual interdependencies and retrieval-based organizational protocol analysis.
- **MMMT-IF [2409.18216]:** Specializes in multi-turn multimodal instruction following, introducing the PIF metric for execution-verified adherence under scattered instruction sets.

A distinguishing feature of MMMB (in the sense of [2510.13747]) is the isolation of memory-dependent answerability apart from single-turn recognition or reasoning, and its inclusion of explicit memory-type annotation and “forgetting curve” diagnostics.

## 6. Open Challenges and Design Recommendations

Persistent limitations of current MLLMs include:

- **Multimodal Retention:** Text-only or caption-centric systems lag direct visual retrieval by $>10\%$ accuracy across several benchmarks; raw image embeddings are essential [2601.03515].
- **Memory Organization:** Flat (retrieval-based) memory structures often outperform hierarchical or complex structured memories due to better noise management [2601.03515].
- **Reasoning and Knowledge Management:** Dynamic update detection and multi-entity/temporal reasoning remain bottlenecks, with typical F1/accuracy $<0.5$ even in best systems [2601.03515].
- **Robustness to Instruction Spread:** Retrieval of distributed constraints (instructions) is a major failure point; performance (PIF) recovers dramatically when all instructions are re-appended to the context [2409.18216].

Future MMMB design should incorporate:
- Explicit representation of raw, high-dimensional modalities (video, audio).
- Modularization of memory layers (short, long, and semantic-term buffers).
- Relevance-aware, possibly learned retrieval filtering to minimize token wastage and maximize Precision@K.
- Revision operators to recognize and update contradicting facts or evolving entities over long horizons.
- Benchmarks for cross-domain transfer and continual update robustness in real-world conversational flows [2510.26160].
- Programmatic instruction-following metrics and robustness evaluation under repeat sampling [2409.18216].

## 7. Significance and Research Impact

MMMBs form a critical infrastructure for diagnosing, quantifying, and accelerating progress in memory-augmented multimodal models. They provide a standardized, controlled setting to compare architectures, training strategies, and retrieval mechanisms under pressure from context-length, modality, and reasoned-update requirements.

By revealing sharp regime changes and failure points—e.g., token-limit drop-off, hallucinated updates, fact-overwrite collisions—MMMBs guide the field toward development of more robust, contextually aware, and memory-consistent agents.

Researchers leveraging MMMB protocols and datasets are now able to reproducibly evaluate incremental advances and investigate systemic sources of long-range memory failure, driving rapid iteration in both fundamental model architecture and external memory augmentation [2409.18216][2502.11903][2510.13747][2601.03515].

Source: https://www.emergentmind.com/topics/multi-modal-multi-turn-memory-benchmark-mmmb-0994feff-10b6-4a2c-89c8-d47e63d82d8c