CRAG-MM: Multimodal RAG Benchmark

• CRAG-MM is a multimodal benchmark that evaluates retrieval-augmented generation systems in wearable-device settings with a focus on factual truthfulness and hallucination control.
• It comprises 6.5K single-turn queries and 2K multi-turn dialogues across 13 diverse domains, incorporating realistic egocentric image conditions.
• The evaluation protocol uses truthfulness scoring to penalize hallucinations, guiding improvements in system architectures and retrieval strategies.

Searching arXiv for the benchmark and challenge papers to ground the article. CRAG-MM is a benchmark and associated KDD Cup 2025 competition for evaluating multi-modal, retrieval-augmented generation systems in realistic wearable-device settings. It formalizes question answering over images, an image-based knowledge graph, webpage retrieval, and multi-turn dialogue, with an explicit emphasis on truthfulness and hallucination control rather than raw answer frequency. The benchmark contains a diverse set of 6.5K single-turn triplets and 2K visual-based multi-turn conversations across 13 domains, including 6.2K egocentric images designed to mimic captures from wearable devices, and its evaluation shows that straightforward RAG approaches achieve only 32% and 43% truthfulness on single- and multi-turn QA, whereas state-of-the-art industry solutions have similar quality at 32% and 45% (Wang et al., 30 Oct 2025).

1. Origins and conceptual scope

CRAG-MM extends the original CRAG benchmark, which was introduced as a factual question answering benchmark of 4,409 question-answer pairs with mocked web search and knowledge graph access, five domains, eight question categories, varying entity popularity, and temporal dynamism ranging from years to seconds. That earlier benchmark was explicitly designed around trustworthy QA, with hallucination-sensitive evaluation and realistic multi-source retrieval, and it laid the groundwork for the KDD Cup 2024 challenge (Yang et al., 2024).

The multimodal extension shifts the setting from text-only RAG to wearable, first-person, visually grounded MM-RAG. In CRAG-MM, the system input is an image $I$ and a question $Q$, or, in the multi-turn setting, an image together with previous dialogue turns; the output is a short factual answer. The benchmark is centered on smart-glasses scenarios in which users ask about entities in view, while the underlying retrieval environment provides both image-KG and webpage search (Wang et al., 30 Oct 2025).

This scope matters because CRAG-MM is not merely a visual question answering dataset with fixed context. Its central object is the end-to-end MM-RAG system: visual recognition, query rewriting, retrieval over heterogeneous sources, answer generation, abstention, and, in Task 3, dialogue-state maintenance.

2. Dataset construction and task tracks

The benchmark comprises 7,943 images, of which 6,248 are egocentric and 1,695 are normal images. The single-turn portion contains 6,462 questions, and the multi-turn portion contains 1,956 conversations with an average length of 4.9 turns. The 13 domains are Animal, Book, Food, General Object Recognition, Local, Math & Science, Plants & Gardening, Shopping, Sports & Games, Style & Fashion, Text Understanding, Vehicle, and Other (Wang et al., 30 Oct 2025).

Question design is deliberately heterogeneous. CRAG-MM defines six question types: Simple-recognition, Simple-knowledge, Multi-hop, Comparison, Aggregation, and Reasoning. Approximately 52% of questions are complex, meaning multi-hop, comparison, aggregation, or reasoning. The dataset also encodes head–torso–tail entity popularity, with about 21% of questions involving torso-to-tail entities, and it incorporates information dynamism through a mix of static and web-dependent questions (Wang et al., 30 Oct 2025).

Egocentric realism is a first-class design variable. Images are annotated across Normal, Low-light, Blurred, Truncated, Occluded, and Rotated conditions. The released counts are 6,736 normal-quality images, 300 low-light, 215 blurred, 401 truncated, 124 occluded, and 167 rotated. About 15% of egocentric images were deliberately collected under imperfect conditions, so the benchmark stresses not only retrieval difficulty but also perceptual degradation (Wang et al., 30 Oct 2025).

Task	Inputs and sources	Objective
Task 1: Single-source augmentation	Image $I$, question $Q$, image-based mock KG	Basic MM-RAG with image-KG grounding
Task 2: Multi-source augmentation	Image $I$, question $Q$, image-KG + web search	Multi-source synthesis under noisy retrieval
Task 3: Multi-turn QA	Image, current turn, dialogue history, image-KG + web	Contextual multi-turn MM-RAG

Task 1 restricts external grounding to the image-based mock KG. Task 2 adds webpage retrieval. Task 3 further introduces dialogue history, with later turns that may or may not still depend on the image. About 38% of conversations involve domain shifts, so Task 3 is not reducible to repeated single-turn answering over a fixed visual entity set (Wang et al., 30 Oct 2025).

3. Retrieval environment and evaluation protocol

The image-based mock KG contains 68K images for 26K entities. The web corpus contains about 800K webpage URLs and 2.7M text chunks. Both are exposed through standardized search APIs. The image search index is built with CLIP ViT-L/14@336px embeddings via ChromaDB; the web corpus is also indexed through the provided retrieval stack so that all participants operate over the same corpus and APIs (Wang et al., 30 Oct 2025).

Retrieval is intentionally imperfect. The image KG covers 93.4% of entities referenced in questions, but the correct entity recall is only 51.9% when the full query image is used and 58.0% with manual cropping. For web retrieval, the top 50 results contain the necessary ground-truth facts for 88.6% of questions. These numbers formalize an important property of the challenge: the system is not being tested on an oracle retrieval channel, but on a realistic noisy approximation (Wang et al., 30 Oct 2025).

The benchmark’s primary metric is Truthfulness. For single-turn QA, each answer receives $+1$ for Perfect, $0$ for Missing, and $-1$ for Incorrect, and the overall score is the mean:

$$\text{Truthfulness} = \frac{1}{N} \sum_{i=1}^N s_i,$$

where $Q$0. Accuracy, Missing, and Hallucination are reported alongside Truthfulness. For multi-turn QA, the scoring is conversation-level, and if the system produces two consecutive wrong or missing answers, all later turns are treated as missing; this early-stop rule operationalizes conversational breakdown (Wang et al., 30 Oct 2025).

Label	Score
Perfect	$Q$1
Missing	$Q$2
Incorrect	$Q$3

Automatic evaluation is performed with GPT-4o as an LLM judge. The reported overall judge accuracy is 99.1%, with average F1 across accurate, incorrect, and missing buckets of 90.8%. Human evaluation is used for manual validation of top systems and private-test assessment, but the automatic judge is accurate enough to support large-scale benchmarking and public-leaderboard iteration (Wang et al., 30 Oct 2025).

4. Baselines, benchmark difficulty, and competition outcomes

The benchmark paper evaluates MM-LLM-only baselines, straightforward MM-RAG baselines, and industry systems. The MM-LLM-only setting already reveals the tension between visual recognition and factual grounding: even strong closed systems produce substantial hallucination rates. Straightforward MM-RAG improves performance, but only modestly relative to the challenge’s difficulty, yielding 32% truthfulness on single-turn QA and 43% on multi-turn QA, while state-of-the-art industry solutions reach similar quality at 32% and 45% (Wang et al., 30 Oct 2025).

The task-wise baselines clarify where gains come from. Task 1, which uses only the image KG, provides only a modest improvement over MM-LLM-only prompting. Task 2, which adds web search and query rewriting, is where the largest single-turn gains appear. Task 3 benefits from the same retrieval stack, but dialogue history introduces error propagation and early-stop effects, so improvements in retrieval alone do not eliminate conversational fragility (Wang et al., 30 Oct 2025).

The KDD Cup 2025 CRAG-MM Challenge attracted about 1K participants and 5K submissions. The winning solution improved baseline performance by 28%, which the benchmark paper highlights as evidence of early impact. In the competition-oriented comparison, a straightforward Llama-3.2-11B-Vision-Instruct RAG baseline has negative single-turn truthfulness and low multi-turn truthfulness, whereas the winning system substantially improves truthfulness by sharply lowering hallucination at the cost of higher missing rates (Wang et al., 30 Oct 2025).

The broader significance is continuous with the original CRAG benchmark. In the text-only setting, most advanced LLMs achieved $Q$4 accuracy on CRAG, and adding RAG in a straightforward manner improved the accuracy only to 44%, while state-of-the-art industry RAG solutions answered only 63% of questions without any hallucination (Yang et al., 2024). CRAG-MM inherits that same design principle: realistic retrieval makes the benchmark substantially harder than closed-book or fixed-context QA.

5. Dominant solution strategies in the CRAG-MM Challenge

The leading systems did not converge on a single architecture, but they did converge on a common objective: suppress hallucinations under a scoring rule in which incorrect answers are worse than abstentions. Three representative lines of attack are particularly illustrative: curriculum-guided RL over a VLLM backbone, verification-centric MM-RAG, and internal-representation-based hallucination filtering (Zhang et al., 14 Aug 2025, Chen et al., 27 Jul 2025, Nakamizo et al., 16 Oct 2025).

System family	Core mechanism	Reported outcome
Curriculum-guided RL + RAG	GPT-4.1 distillation, VisualRFT/GRPO, curriculum over easy/hard examples, web search for Tasks 2/3	1st place in Task 1; 3rd place in Task 3
Verification-centric MM-RAG	Lightweight query router, query-aware retrieval and summarization, dual-pathways generation, post-hoc verification	3rd place in Task 1
Internal-state hallucination filtering	Logistic regression probes on hidden states and attention heads, ensemble thresholding, “I don’t know” fallback	5th place overall

The Dianping-Trust-Safety system uses Llama 3.2–11B-Vision-Instruct as the backbone, GPT-4.1-generated supervision, and VisualRFT with Grouped Relative Policy Optimization. Its defining contribution is a three-stage curriculum over easy and hard examples to prevent RL collapse into an “answer nothing” regime. Officially, it achieved 1st place in Task 1 with a significant lead of 52.38% in manual evaluation and 3rd place in Task 3 (Zhang et al., 14 Aug 2025).

The CRUISE system adopts a multi-stage verification-centric design. It uses a LLaMA-3.2-1B-Instruct query router to predict whether external information is needed and whether a question is real-time, then a query-aware retrieval module with a one-sentence image summary generated by LLaMA-3.2-11B-Vision-Instruct, a MAD-based reranking threshold, dual-pathways answer generation, self-consistency checking, and a Chain-of-Verification post-hoc verifier. This system achieved 3rd place in Task 1 and illustrates the value of explicit retrieval-quality gating and confidence-controlled abstention (Chen et al., 27 Jul 2025).

The y3h2 system is notable because it does not rely on external retrieval at all. Instead, it trains logistic-regression hallucination detectors on hidden states and attention head outputs of meta-llama/Llama-3.2-11B-Vision-Instruct, ensembles 65 selected probes, and accepts an answer only when the predicted probability of non-hallucination exceeds 0.65; otherwise it outputs “I don’t know.” This hallucination-filtering strategy placed 5th on the final leaderboard, demonstrating that, under CRAG-MM scoring, confidence estimation and refusal can be as important as retrieval itself (Nakamizo et al., 16 Oct 2025).

6. Misconceptions, limitations, and adjacent research directions

A common misconception is that adding more retrieval automatically improves performance. The challenge evidence is more nuanced. Straightforward MM-RAG improves over MM-LLM-only baselines, but remains far from solved, and some successful systems become highly conservative to suppress hallucinations. One Task 1 solution explicitly stopped using image search in its final system because the base VLLM was not good at object detection or precise region selection, and the image search results introduced instability and noise; another reported that naive RAG could increase hallucination rate and produce strongly negative truthfulness unless verification was added (Zhang et al., 14 Aug 2025, Chen et al., 27 Jul 2025).

The benchmark itself also has acknowledged limitations. The web corpus is a static snapshot rather than a live web index; image search is restricted to the mock KG rather than unrestricted web images; egocentric collection is shaped by privacy and safety constraints; and domain coverage, while broad, is not exhaustive. These constraints are deliberate for reproducibility and fairness, but they mean that CRAG-MM is best understood as a controlled benchmark for realistic wearable MM-RAG rather than a full simulation of open-world deployment (Wang et al., 30 Oct 2025).

Related evaluation work points toward likely future extensions. MRAG-Suite proposes difficulty-based and ambiguity-aware filtering together with MM-RAGChecker, a claim-level diagnostic tool for multimodal RAG, showing that difficult and ambiguous queries produce substantial accuracy drops and elevated hallucination rates (Ji, 29 Sep 2025). A plausible implication is that future CRAG-MM variants may move beyond answer-level truthfulness toward finer-grained evidence attribution, ambiguity handling, and modality-specific grounding diagnostics.

The acronym itself is overloaded across arXiv. “CRAG” can denote the original Comprehensive RAG benchmark (Yang et al., 2024), “Clustered Retrieved Augmented Generation” (Akesson et al., 2024), and an open-source reproduction of “Corrective Retrieval Augmented Generation” (Yalavarthi, 17 Mar 2026); it also appears in unrelated areas such as 3D assembly in “CRAG: Can 3D Generative Models Help 3D Assembly?” (Jiang et al., 26 Feb 2026). Within the challenge literature, however, “CRAG-MM” refers specifically to the multimodal, multi-turn extension of Comprehensive RAG introduced for KDD Cup 2025 (Wang et al., 30 Oct 2025).