---
title: 'MRBench Dataset: A Multimodal Evaluation Benchmark'
url: https://www.emergentmind.com/topics/mrbench-dataset
type: topic
---

# MRBench Dataset: A Multimodal Evaluation Benchmark

MRBench is a large-scale, human-annotated benchmark specifically designed to evaluate Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) systems. It addresses the need for rigorous, domain-diverse, and challenging evaluation of generative models that synthesize interleaved text and image outputs, leveraging both retrieval and multimodal generation in open-domain, academic, and lifestyle contexts. Unlike previous benchmarks, MRBench systematically measures the ability of models to retrieve, ground, and accurately compose both textual and visual information in complex, multi-step QA scenarios, encompassing a wide range of real-world use cases [2502.04176].

## 1. Benchmark Composition and Dataset Structure

MRBench comprises 4,346 documents, 14,190 images, and 4,800 QA pairs, spanning three principal domains (Web, Academia, Lifestyle) and three ascending difficulty levels:

| Domain     | Subset          | #Docs | #Images | #QA Pairs | Mean Images/Doc |
|------------|-----------------|-------|---------|-----------|-----------------|
| Web        | MRAMG-Wit       | 639   | 639     | 600       | 1.0             |
|            | MRAMG-Wiki      | 538   | 538     | 500       | 1.0             |
|            | MRAMG-Web       | 1,500 | 1,500   | 750       | 1.0             |
| Academia   | MRAMG-Arxiv     | 101   | 337     | 200       | 3.3             |
| Lifestyle  | MRAMG-Recipe    | 1,528 | 8,569   | 2,360     | 5.6             |
|            | MRAMG-Manual    | 40    | 2,607   | 390       | 65.2            |

Multi-image QA instances are pervasive: 948 questions reference exactly two images (e.g., schematic comparison), and 862 questions require synthesis across three or more images (e.g., recipe step sequences, technical multi-panel figures). With an overall average of 3.27 images per document, the benchmark uniquely tests integration and ordering of visual evidence in answers [2502.04176].

## 2. Data Sources, Annotation, and QA Construction

MRBench aggregates and refines data from curated sources:

- **Level 1 (Web):** Draws on Wit (Wikipedia-based image-text), WikiWeb2M, and WebQA img-posFacts resources, with additional context expansion through GPT-4o to ensure caption-image alignment.
- **Level 2 (Academic):** 110 manually filtered arXiv LaTeX/PDF papers (2023–2024); technical questions and answers authored by graduate-level annotators.
- **Level 3 (Lifestyle):** RecipeQA evaluation set (step-by-step with images), and manuals from ManualsLib and Kaggle, parsed and cleaned via MinerU and human curation.

Annotation follows a semi-automated, multi-stage process:
1. **Question Generation:** 
    - GPT-4o is used for initial QA construction (Web, Recipe), with expert questions for Arxiv and Manual.
2. **Answer Creation:** 
    - Chain-of-Thought (CoT) GPT-4o prompting for draft generation, image placeholder assignment.
    - Manual answer curation and ordering for technical and multi-image settings.
3. **Quality Control:** 
    - Three-stage review combining annotator, GPT-4o, and senior expert intervention.
    - Cross-annotator agreement approaches 100% on question validity, answer groundedness, and image selection.

QA pairs are encoded as interleaved lists of text and image blocks, preserving the order and semantics required for faithful multimodal answer generation [2502.04176].

## 3. Formal Task Definition

The MRAMG task requires, given a natural language query $q$ and a large corpus $\mathcal{D}$ (each document consisting of text $T_i$ and image $I_i$ fragments), to:
1. Retrieve the top-$k$ relevant documents $\mathcal{D}_q^*$
2. Generate an answer $\mathcal{A}$ as an ordered sequence of text and image references

Formally,
\[
\mathcal{A} = \mathcal{F}(q, \mathcal{D}_q^*, \mathcal{M})
\]
where $\mathcal{F}$ is the generation framework and $\mathcal{M}$ a text or multimodal model (LLM or MLLM).

For probabilistic models:
\[
\mathcal{A} = \arg\max_{\mathcal{A}} P(\mathcal{A}\mid q, \mathcal{D}_q^*)
\]

This structure supports both pipeline (retrieval → generation) and fully end-to-end approaches [2502.04176].

## 4. Evaluation Metrics and Protocols

Evaluation in MRBench leverages both statistical and LLM-based metrics, targeting both retrieval and generation components.

**Retrieval Metrics:**
- **Context Recall@k:** Fraction of queries for which all gold document text is in the top-$k$ retrieved results.
- **Image Recall@k:** Fraction of gold-referenced images among the top-$k$.

**Generation Metrics:**
- **Image Precision, Recall, F₁:** $|\text{images}_\text{pred}\cap\text{images}_\text{gold}|$ over predicted and gold sets
- **Image Ordering Score:** Weighted edit distance between predicted and ground-truth image sequences.
- **ROUGE-L:** LCS-based overlap for textual segments [Lin 2004].
- **BERTScore:** Contextualized semantic similarity [Zhang et al. 2019].
- **LLM-based metrics (scored 1–5 by GPT-4o):**
  1. Image Relevance
  2. Image Effectiveness
  3. Image Position Score
  4. Comprehensive Multimodal Answer Quality

Bleu-1 and further metrics are available for fine-grained analysis, with evaluation code distributed for reproducibility [2502.04176].

## 5. Baseline Systems and Comparative Results

MRBench reports performance across 11 models under three answer generation paradigms:
- **A) Rule-based insertion:** e.g., DeepSeek-V3, which demonstrates competitive performance on Web data (Comp. = 80.18).
- **B) Direct MLLM:** Models such as Gemini-1.5-Pro (Web Comp. = 92.13, Paper = 85.90).
- **C) LLM + placeholders:** Combining retrieved evidence with LLM generation.

Closed-source models (GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro) systematically outperform open-source alternatives, especially on complex, multi-image answers. For example, Gemini-1.5-Pro achieves comprehensive scores of 94.35 (Web), 87.60 (Paper), and 83.10 (Lifestyle), yielding an overall ≈88.7 [2502.04176]. Precise multi-image orderings remain challenging (GPT-4o bests others in Web Ord. = 43.5). Rule-based methods are viable for simple cases but degrade as question and context complexity increase.

## 6. Usage Considerations, Best Practices, and Pitfalls

**Recommended pipeline:**
- **Retrieval:** Embed with BGE-M3; retrieve top-10 by cosine similarity, aggregate associated images.
- **Preprocessing:** Chunk using SentenceSplitter (256 tokens); use standardized `<PIC>` placeholders for image blocks in LLM settings.
- **Generation:** In LLM modes, include surrounding context sentences. For MLLMs, CLIP-based ranking selects images ($N\le8$ images). Tune BLEU/BGE thresholds in rule-based settings.

Pitfalls include over-insertion of irrelevant images (monitor Image Precision), image misordering (train or instruct with ordering score), and context window truncation (chunking or longer-context models). Text-only QA is rare and only present in certain Lifestyle datasets. All data and code are released under CC BY 4.0 (https://huggingface.co/MRAMG), facilitating adoption and benchmarking [2502.04176].

## 7. Significance and Impact on Multimodal Retrieval-Augmented Generation

MRBench establishes the foundational evaluation benchmark for the MRAMG task: fully integrated text + image question answering grounded in retrieval from large, diverse domains. It differs from prior benchmarks by combining
- high multimodal density,
- diverse multi-image and multi-step scenarios,
- rigorous statistical and LLM-based evaluation,
- broad coverage (Web, scientific, real-world “how-to”).

This design enables robust tracking of MRAG and MLLM advances on fine-grained, realistic multimodal synthesis tasks. The benchmark exposes current model deficiencies—such as failure on answer grounding, image selection, and document-order tracking—and provides the structure required for future methodological progress in retrieval-augmented multimodal generative systems [2502.04176].

Source: https://www.emergentmind.com/topics/mrbench-dataset