---
title: 'CRAG-MM: Multi-modal QA Benchmark'
url: https://www.emergentmind.com/topics/crag-mm-benchmark
type: topic
---

# CRAG-MM: Multi-modal QA Benchmark

CRAG-MM, short for **“CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmark,”** is a benchmark for evaluating **multi-modal retrieval-augmented generation** systems that answer factual questions grounded in an **image** together with **externally retrieved knowledge**, and, in the multi-turn setting, also grounded in **conversation history** [2510.26160]. It is explicitly designed for **wearable AI** and **egocentric assistant** scenarios, where a system must answer questions about entities in view under realistic first-person imaging conditions rather than under curated web-image assumptions. The benchmark paper reports **6,462 single-turn QA instances**, **1,956 multi-turn conversations**, and **7,943 images** in total, while the abstract summarizes these at a coarser level as **6.5K** single-turn triplets, **2K** multi-turn conversations, and **6.2K** egocentric images [2510.26160].

## 1. Origins and benchmark scope

CRAG-MM extends the design philosophy of the original **CRAG** benchmark from text-centric factual RAG to a multimodal, retrieval-dependent, and conversational setting. The original CRAG introduced a **Comprehensive RAG Benchmark** with **4,409 question-answer pairs**, **five domains**, **eight question types**, and mock APIs over web and knowledge-graph sources, emphasizing realism, dynamism, long-tail entities, and trust-aware evaluation [2406.04744]. CRAG-MM carries that orientation into a setting where the primary grounding signal is visual and frequently egocentric rather than textual.

The benchmark paper defines two task formulations. In the single-turn case, an MM-RAG QA system takes an image \(I\) and a question \(Q\), and outputs an answer \(A\) using both internal model knowledge and externally retrieved information. In the multi-turn case, the system additionally uses previous turns’ questions and answers as context [2510.26160]. This makes CRAG-MM neither a conventional VQA benchmark nor a pure retrieval benchmark. Its object of evaluation is an end-to-end system that must combine perception, retrieval, synthesis, and calibrated abstention.

CRAG-MM is centered on factual questions rather than open-ended captioning. The benchmark paper states that about **89%** of its conversations focus on factual questions that require external information for trustworthy answers, which places it closer to RAG evaluation than to generic multimodal chat [2510.26160]. A plausible implication is that CRAG-MM operationalizes multimodal QA primarily as a grounded information-access problem.

## 2. Dataset composition and annotation design

The dataset contains **6,248 egocentric images** and **1,695 normal/public images**, for **7,943** images overall [2510.26160]. The egocentric portion was collected with **smart glasses**, and the benchmark deliberately includes difficult wearable-style imagery. The paper states that **15% of egocentric images** were requested under imperfect conditions: **low-light**, **blurred**, **truncated**, **occluded**, and **rotated**. The detailed counts are **300** low-light images, **215** blurred, **401** truncated, **124** occluded, and **167** rotated, alongside **6,736** images categorized as normal [2510.26160]. These artifacts are not incidental; they are a formal benchmark dimension.

CRAG-MM spans **13 domains**: **Animal**, **Book**, **Food**, **General Object Recognition**, **Local**, **Math & Science**, **Plants & Gardening**, **Shopping**, **Sports & Games**, **Style & Fashion**, **Text Understanding**, **Vehicle**, and **Other** [2510.26160]. The single-turn QA distribution is dominated by domains such as **Local** (**937**), **Plants & Gardening** (**930**), **Vehicle** (**860**), and **Food** (**771**), while the multi-turn set uses the first turn’s domain label and covers all 13 domains as well [2510.26160].

The benchmark defines **six question types**. Their single-turn counts are **Simple-recognition** (**462**), **Simple-knowledge** (**2,614**), **Multi-hop** (**928**), **Comparison** (**977**), **Aggregation** (**772**), and **Reasoning** (**709**) [2510.26160]. The benchmark is therefore not dominated by image-only recognition; the largest category is **Simple-knowledge**, and the paper states that **52%** of questions are complex and require **multi-source information synthesis** [2510.26160].

The multi-turn portion contains **1,956 conversations**, each with **2–6 turns** and an **average length of 4.9 turns**. Each conversation spans **1–3 domains**, and about **38%** involve **domain shifts** [2510.26160]. This makes dialogue context a substantive evaluation axis rather than a superficial add-on.

Single-turn QA was created from two sources. In the **KG-based** pipeline, the authors selected relation templates from knowledge graphs and paired sampled entities with those templates, including explicit multi-hop constructions of the form \((e_1,r_1,e_2)\) and \((e_2,r_2,e_3)\). In the **web-content-based** pipeline, annotators wrote plausible wearable-device questions, formulated standalone web-search queries, searched the web, and recorded ground-truth answers [2510.26160]. Multi-turn conversations were seeded from web contents, expanded using **Llama-3.2-90B-Vision-Instruct**, then revised by annotators to remove implausible or ambiguous turns and to ensure that each turn had a single indisputable answer [2510.26160].

## 3. Task structure and retrieval environment

CRAG-MM is organized into three tasks, each exposing a different retrieval condition and thus a different MM-RAG operating regime [2510.26160].

| Task | Retrieval resources | Primary capability |
|---|---|---|
| **Task 1: Single-source augmentation** | Image search mock API over image-based KG | Image-grounded factual QA with single-source retrieval |
| **Task 2: Multi-source augmentation** | Image search mock API + web search mock API | Cross-source retrieval and synthesis |
| **Task 3: Multi-turn QA** | Multi-source retrieval + conversation history | Conversational grounding and context tracking |

The image-side retrieval resource is a **mock image-based KG** containing **68K images** and **26K entities** [2510.26160]. It is constructed to include seed coverage from images used for question generation, open-web candidate data, **positive examples**, and **hard negatives** consisting of visually similar images from different entities. The original query images are then removed to make the retrieval setting more realistic [2510.26160]. The image encoder is **CLIP ViT-L/14@336px**, and the backend index is **ChromaDB** [2510.26160].

The web-side resource contains **800K webpage URLs** and **2.7M chunks** [2510.26160]. For each question, annotators created a standalone search query; the benchmark additionally generated negative queries using only the ground-truth entity or similar but wrong entities, searched them with **Brave Search API**, stored up to **20 URLs per query**, and pooled up to **50 distinct webpages per question** into the searchable repository [2510.26160].

The retrieval environment is intentionally noisy. The paper reports an image-search relevant-to-irrelevant ratio of roughly **1:20** and a web-search ratio of roughly **1:2** [2510.26160]. At the same time, the retrieval resources are not deliberately impossible: the image KG covers **93.4%** of query entities, the top-50 web search results are estimated to contain the ground-truth facts for **88.6%** of questions, querying the image KG with the full original image yields **51.9% recall**, and manual cropping improves that to **58.0%** [2510.26160]. This combination of nontrivial coverage and systematic noise is one of the benchmark’s defining design choices.

## 4. Evaluation methodology and scoring

CRAG-MM uses **truthfulness** as its primary benchmark metric. The benchmark paper describes single-turn scoring in prose as **correct = 1**, **missing = 0**, and **incorrect = -1**, averaged over examples; for multi-turn QA, if a system gives **two consecutive wrong or missing answers**, the conversation is **early stopped** and all remaining turns are counted as missing [2510.26160]. The benchmark tables additionally report **Acc.**, **Miss.**, **Hallu.**, and **Early Stop.** [2510.26160].

A competition report writes this benchmark-level aggregation explicitly as
\[
\text{Trustfulness}=\frac{1}{M}\sum_{i=1}^{M}s_i,
\]
with \(s_i \in \{+1,0,-1\}\) according to whether an answer is correct, refused/missing, or incorrect [2510.14330]. Another CRAG-MM solution paper describes a four-way rubric—**Perfect**, **Acceptable**, **Missing**, and **Incorrect**—with scores **1.0**, **0.5**, **0.0**, and **-1.0**, respectively [2507.21520]. Taken together, these reports indicate that the benchmark’s automatic leaderboard metric is strongly trustfulness-oriented, while manual evaluation preserves a finer distinction between fully correct and acceptable answers.

Automatic judging is performed by **GPT-4o**, using the question, ground-truth answer, and model prediction, with predictions truncated to **75 tokens** [2510.26160]. Against manual evaluation, the paper reports **99.1% accuracy** for the auto-evaluator overall, with **94.3 F1** on accurate answers, **78.2 F1** on incorrect answers, and **100.0 F1** on missing answers [2510.26160]. The benchmark paper does **not** define a separate official retrieval-only leaderboard metric; retrieval quality is instead analyzed through coverage and recall studies rather than through a standalone retrieval score [2510.26160].

This scoring design materially changes system incentives. Since an incorrect answer incurs a negative score while abstention is neutral, CRAG-MM rewards selective refusal when support is weak. This suggests that calibration is a first-class benchmark target rather than a secondary deployment concern.

## 5. Empirical difficulty, baselines, and challenge behavior

CRAG-MM is difficult for both straightforward baselines and strong industry systems. In the official benchmark paper, the best **single-turn** straightforward result is **GPT-5 Mini** with **22.5** truthfulness on **Task 1** and **31.5** on **Task 2**; the best **multi-turn** straightforward result is **42.5** on **Task 3**, again from **GPT-5 Mini** [2510.26160]. Among industry systems tested with native tools, **GPT-5** reaches **32.2** single-turn truthfulness and **45.0** multi-turn truthfulness, but still shows substantial hallucination rates: **30.5** single-turn and **26.1** multi-turn [2510.26160].

The benchmark is especially hard on low-quality images, visual-only recognition, less popular entities, and compositional question types. The paper reports that all state-of-the-art systems degrade sharply on low-quality images, with truthfulness drops of up to **46%**; **low-light** and **occluded** images are the hardest, with best truthfulness only **20%** and **24%** [2510.26160]. Performance also drops by up to **37%** when text clues are absent and recognition must rely on visual information alone, and **21%** of questions involve **torso-to-tail entities**, which are explicitly harder [2510.26160]. In multi-turn evaluation, systems struggle on questions requiring conversation history; even the best system averages only **3.2 successful turns** out of **4.9**, whereas the best straightforward system averages **2.7** [2510.26160].

CRAG-MM hosted **KDD Cup 2025**, attracting about **1K participants** and **5K submissions** [2510.26160]. The winning team improved over the straightforward system using the same base model by **+28%** on single-turn and **+18%** on multi-turn evaluation [2510.26160]. Competition papers make clear what kinds of methods the benchmark favors. The **BlackPearl** system used **data augmentation**, **RAG**, **reranking**, and **multi-task fine-tuning**, reaching automatic-evaluation ranks of **3rd**, **3rd**, and **1st** across the three tasks, then **2nd** on Task 3 after human evaluation [2507.21520]. The **CRUISE** system prioritized a lightweight router, query-aware retrieval, dual-path generation, and post-hoc verification, achieving **3rd place in Task 1** [2507.20136]. The **y3h2** system reached **5th place** by suppressing hallucinations with detectors trained on hidden states and attention heads of **Llama-3.2-11B-Vision-Instruct**, showing that aggressive hallucination filtering can improve truthfulness even when it sacrifices many correct answers [2510.14330].

These results indicate that CRAG-MM does not reward raw answer coverage alone. It rewards systems that retrieve selectively, filter evidence, exploit conversation state when needed, and refuse when justification is weak.

## 6. Position in the benchmark landscape, limitations, and nomenclature

CRAG-MM occupies a distinct place among multimodal and RAG benchmarks. Unlike **REAL-MM-RAG**, which evaluates page-level retrieval over long enterprise documents with multimodal pages, rephrased queries, and ranking metrics such as **NDCG@5**, CRAG-MM evaluates end-to-end **factual image-grounded QA** with retrieval APIs and multi-turn dialogue [2502.12342]. REAL-MM-RAG is thus primarily a multimodal retrieval benchmark, whereas CRAG-MM is an MM-RAG QA benchmark whose retrieval layer is embedded inside a broader answer-generation task.

The acronym **CRAG** is also overloaded in unrelated literatures. **“Mixture-of-Workflows for Self-Corrective Retrieval-Augmented Generation”** evaluates agentic chemical search and multimodal NMR retrieval, but it does **not** define an official benchmark called **CRAG-MM** [2502.19629]. Likewise, **“CRAG: Can 3D Generative Models Help 3D Assembly?”** introduces a 3D assembly-and-generation method and corresponding evaluation settings, but not a benchmark named CRAG-MM [2602.22629]. Within the RAG benchmark lineage, the direct precursor is the original **CRAG** benchmark, not those unrelated acronym uses [2406.04744].

The official CRAG-MM paper also leaves several boundaries explicit. The benchmark is **English-only**, web retrieval is **text-based only**, and image search is simplified into **KG-search** rather than full open-world image retrieval [2510.26160]. The paper does **not** provide a formal freshness taxonomy comparable to the temporal dynamism categories of the original CRAG benchmark, even though current information and prices are part of the motivating examples [2510.26160]. Nor does it define a separate retrieval-only leaderboard metric. These omissions do not negate the benchmark’s breadth, but they delimit what it evaluates directly.

CRAG-MM’s significance lies in the conjunction of its design choices: **wearable-centered egocentric images**, **explicit retrieval infrastructure**, **multi-source and multi-turn task structure**, **hard negatives**, **long-tail entities**, **image-quality perturbations**, and **truthfulness-oriented scoring** [2510.26160]. In that sense, it serves less as a conventional VQA dataset than as a systems benchmark for grounded multimodal assistance under realistic retrieval conditions.

Source: https://www.emergentmind.com/topics/crag-mm-benchmark