CRAG-MM: Multi-modal QA Benchmark

• The paper introduces CRAG-MM, a benchmark for evaluating multi-modal retrieval-augmented generation systems under realistic egocentric imaging conditions and conversational context.
• It comprises 6,462 single-turn QA instances, 1,956 multi-turn conversations, and 7,943 images, including challenging low-light, blurred, and occluded cases to test system robustness.
• CRAG-MM assesses end-to-end system capabilities by integrating image perception, retrieval synthesis, and dialogue history, while emphasizing truthfulness and calibrated abstention in scoring.

CRAG-MM, short for “CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmark,” is a benchmark for evaluating multi-modal retrieval-augmented generation systems that answer factual questions grounded in an image together with externally retrieved knowledge, and, in the multi-turn setting, also grounded in conversation history (Wang et al., 30 Oct 2025). It is explicitly designed for wearable AI and egocentric assistant scenarios, where a system must answer questions about entities in view under realistic first-person imaging conditions rather than under curated web-image assumptions. The benchmark paper reports 6,462 single-turn QA instances, 1,956 multi-turn conversations, and 7,943 images in total, while the abstract summarizes these at a coarser level as 6.5K single-turn triplets, 2K multi-turn conversations, and 6.2K egocentric images (Wang et al., 30 Oct 2025).

1. Origins and benchmark scope

CRAG-MM extends the design philosophy of the original CRAG benchmark from text-centric factual RAG to a multimodal, retrieval-dependent, and conversational setting. The original CRAG introduced a Comprehensive RAG Benchmark with 4,409 question-answer pairs, five domains, eight question types, and mock APIs over web and knowledge-graph sources, emphasizing realism, dynamism, long-tail entities, and trust-aware evaluation (Yang et al., 2024). CRAG-MM carries that orientation into a setting where the primary grounding signal is visual and frequently egocentric rather than textual.

The benchmark paper defines two task formulations. In the single-turn case, an MM-RAG QA system takes an image $I$ and a question $Q$, and outputs an answer $A$ using both internal model knowledge and externally retrieved information. In the multi-turn case, the system additionally uses previous turns’ questions and answers as context (Wang et al., 30 Oct 2025). This makes CRAG-MM neither a conventional VQA benchmark nor a pure retrieval benchmark. Its object of evaluation is an end-to-end system that must combine perception, retrieval, synthesis, and calibrated abstention.

CRAG-MM is centered on factual questions rather than open-ended captioning. The benchmark paper states that about 89% of its conversations focus on factual questions that require external information for trustworthy answers, which places it closer to RAG evaluation than to generic multimodal chat (Wang et al., 30 Oct 2025). A plausible implication is that CRAG-MM operationalizes multimodal QA primarily as a grounded information-access problem.

2. Dataset composition and annotation design

The dataset contains 6,248 egocentric images and 1,695 normal/public images, for 7,943 images overall (Wang et al., 30 Oct 2025). The egocentric portion was collected with smart glasses, and the benchmark deliberately includes difficult wearable-style imagery. The paper states that 15% of egocentric images were requested under imperfect conditions: low-light, blurred, truncated, occluded, and rotated. The detailed counts are 300 low-light images, 215 blurred, 401 truncated, 124 occluded, and 167 rotated, alongside 6,736 images categorized as normal (Wang et al., 30 Oct 2025). These artifacts are not incidental; they are a formal benchmark dimension.

CRAG-MM spans 13 domains: Animal, Book, Food, General Object Recognition, Local, Math & Science, Plants & Gardening, Shopping, Sports & Games, Style & Fashion, Text Understanding, Vehicle, and Other (Wang et al., 30 Oct 2025). The single-turn QA distribution is dominated by domains such as Local (937), Plants & Gardening (930), Vehicle (860), and Food (771), while the multi-turn set uses the first turn’s domain label and covers all 13 domains as well (Wang et al., 30 Oct 2025).

The benchmark defines six question types. Their single-turn counts are Simple-recognition (462), Simple-knowledge (2,614), Multi-hop (928), Comparison (977), Aggregation (772), and Reasoning (709) (Wang et al., 30 Oct 2025). The benchmark is therefore not dominated by image-only recognition; the largest category is Simple-knowledge, and the paper states that 52% of questions are complex and require multi-source information synthesis (Wang et al., 30 Oct 2025).

The multi-turn portion contains 1,956 conversations, each with 2–6 turns and an average length of 4.9 turns. Each conversation spans 1–3 domains, and about 38% involve domain shifts (Wang et al., 30 Oct 2025). This makes dialogue context a substantive evaluation axis rather than a superficial add-on.

Single-turn QA was created from two sources. In the KG-based pipeline, the authors selected relation templates from knowledge graphs and paired sampled entities with those templates, including explicit multi-hop constructions of the form $(e_1,r_1,e_2)$ and $(e_2,r_2,e_3)$. In the web-content-based pipeline, annotators wrote plausible wearable-device questions, formulated standalone web-search queries, searched the web, and recorded ground-truth answers (Wang et al., 30 Oct 2025). Multi-turn conversations were seeded from web contents, expanded using Llama-3.2-90B-Vision-Instruct, then revised by annotators to remove implausible or ambiguous turns and to ensure that each turn had a single indisputable answer (Wang et al., 30 Oct 2025).

3. Task structure and retrieval environment

CRAG-MM is organized into three tasks, each exposing a different retrieval condition and thus a different MM-RAG operating regime (Wang et al., 30 Oct 2025).

Task	Retrieval resources	Primary capability
Task 1: Single-source augmentation	Image search mock API over image-based KG	Image-grounded factual QA with single-source retrieval
Task 2: Multi-source augmentation	Image search mock API + web search mock API	Cross-source retrieval and synthesis
Task 3: Multi-turn QA	Multi-source retrieval + conversation history	Conversational grounding and context tracking

The image-side retrieval resource is a mock image-based KG containing 68K images and 26K entities (Wang et al., 30 Oct 2025). It is constructed to include seed coverage from images used for question generation, open-web candidate data, positive examples, and hard negatives consisting of visually similar images from different entities. The original query images are then removed to make the retrieval setting more realistic (Wang et al., 30 Oct 2025). The image encoder is CLIP ViT-L/14@336px, and the backend index is ChromaDB (Wang et al., 30 Oct 2025).

The web-side resource contains 800K webpage URLs and 2.7M chunks (Wang et al., 30 Oct 2025). For each question, annotators created a standalone search query; the benchmark additionally generated negative queries using only the ground-truth entity or similar but wrong entities, searched them with Brave Search API, stored up to 20 URLs per query, and pooled up to 50 distinct webpages per question into the searchable repository (Wang et al., 30 Oct 2025).

The retrieval environment is intentionally noisy. The paper reports an image-search relevant-to-irrelevant ratio of roughly 1:20 and a web-search ratio of roughly 1:2 (Wang et al., 30 Oct 2025). At the same time, the retrieval resources are not deliberately impossible: the image KG covers 93.4% of query entities, the top-50 web search results are estimated to contain the ground-truth facts for 88.6% of questions, querying the image KG with the full original image yields 51.9% recall, and manual cropping improves that to 58.0% (Wang et al., 30 Oct 2025). This combination of nontrivial coverage and systematic noise is one of the benchmark’s defining design choices.

4. Evaluation methodology and scoring

CRAG-MM uses truthfulness as its primary benchmark metric. The benchmark paper describes single-turn scoring in prose as correct = 1, missing = 0, and incorrect = -1, averaged over examples; for multi-turn QA, if a system gives two consecutive wrong or missing answers, the conversation is early stopped and all remaining turns are counted as missing (Wang et al., 30 Oct 2025). The benchmark tables additionally report Acc., Miss., Hallu., and Early Stop. (Wang et al., 30 Oct 2025).

A competition report writes this benchmark-level aggregation explicitly as

$\text{Trustfulness}=\frac{1}{M}\sum_{i=1}^{M}s_i,$

with $s_i \in \{+1,0,-1\}$ according to whether an answer is correct, refused/missing, or incorrect (Nakamizo et al., 16 Oct 2025). Another CRAG-MM solution paper describes a four-way rubric—Perfect, Acceptable, Missing, and Incorrect—with scores 1.0, 0.5, 0.0, and -1.0, respectively (Zhang et al., 29 Jul 2025). Taken together, these reports indicate that the benchmark’s automatic leaderboard metric is strongly trustfulness-oriented, while manual evaluation preserves a finer distinction between fully correct and acceptable answers.

Automatic judging is performed by GPT-4o, using the question, ground-truth answer, and model prediction, with predictions truncated to 75 tokens (Wang et al., 30 Oct 2025). Against manual evaluation, the paper reports 99.1% accuracy for the auto-evaluator overall, with 94.3 F1 on accurate answers, 78.2 F1 on incorrect answers, and 100.0 F1 on missing answers (Wang et al., 30 Oct 2025). The benchmark paper does not define a separate official retrieval-only leaderboard metric; retrieval quality is instead analyzed through coverage and recall studies rather than through a standalone retrieval score (Wang et al., 30 Oct 2025).

This scoring design materially changes system incentives. Since an incorrect answer incurs a negative score while abstention is neutral, CRAG-MM rewards selective refusal when support is weak. This suggests that calibration is a first-class benchmark target rather than a secondary deployment concern.

5. Empirical difficulty, baselines, and challenge behavior

CRAG-MM is difficult for both straightforward baselines and strong industry systems. In the official benchmark paper, the best single-turn straightforward result is GPT-5 Mini with 22.5 truthfulness on Task 1 and 31.5 on Task 2; the best multi-turn straightforward result is 42.5 on Task 3, again from GPT-5 Mini (Wang et al., 30 Oct 2025). Among industry systems tested with native tools, GPT-5 reaches 32.2 single-turn truthfulness and 45.0 multi-turn truthfulness, but still shows substantial hallucination rates: 30.5 single-turn and 26.1 multi-turn (Wang et al., 30 Oct 2025).

The benchmark is especially hard on low-quality images, visual-only recognition, less popular entities, and compositional question types. The paper reports that all state-of-the-art systems degrade sharply on low-quality images, with truthfulness drops of up to 46%; low-light and occluded images are the hardest, with best truthfulness only 20% and 24% (Wang et al., 30 Oct 2025). Performance also drops by up to 37% when text clues are absent and recognition must rely on visual information alone, and 21% of questions involve torso-to-tail entities, which are explicitly harder (Wang et al., 30 Oct 2025). In multi-turn evaluation, systems struggle on questions requiring conversation history; even the best system averages only 3.2 successful turns out of 4.9, whereas the best straightforward system averages 2.7 (Wang et al., 30 Oct 2025).

CRAG-MM hosted KDD Cup 2025, attracting about 1K participants and 5K submissions (Wang et al., 30 Oct 2025). The winning team improved over the straightforward system using the same base model by +28% on single-turn and +18% on multi-turn evaluation (Wang et al., 30 Oct 2025). Competition papers make clear what kinds of methods the benchmark favors. The BlackPearl system used data augmentation, RAG, reranking, and multi-task fine-tuning, reaching automatic-evaluation ranks of 3rd, 3rd, and 1st across the three tasks, then 2nd on Task 3 after human evaluation (Zhang et al., 29 Jul 2025). The CRUISE system prioritized a lightweight router, query-aware retrieval, dual-path generation, and post-hoc verification, achieving 3rd place in Task 1 (Chen et al., 27 Jul 2025). The y3h2 system reached 5th place by suppressing hallucinations with detectors trained on hidden states and attention heads of Llama-3.2-11B-Vision-Instruct, showing that aggressive hallucination filtering can improve truthfulness even when it sacrifices many correct answers (Nakamizo et al., 16 Oct 2025).

These results indicate that CRAG-MM does not reward raw answer coverage alone. It rewards systems that retrieve selectively, filter evidence, exploit conversation state when needed, and refuse when justification is weak.

6. Position in the benchmark landscape, limitations, and nomenclature

CRAG-MM occupies a distinct place among multimodal and RAG benchmarks. Unlike REAL-MM-RAG, which evaluates page-level retrieval over long enterprise documents with multimodal pages, rephrased queries, and ranking metrics such as NDCG@5, CRAG-MM evaluates end-to-end factual image-grounded QA with retrieval APIs and multi-turn dialogue (Wasserman et al., 17 Feb 2025). REAL-MM-RAG is thus primarily a multimodal retrieval benchmark, whereas CRAG-MM is an MM-RAG QA benchmark whose retrieval layer is embedded inside a broader answer-generation task.

The acronym CRAG is also overloaded in unrelated literatures. “Mixture-of-Workflows for Self-Corrective Retrieval-Augmented Generation” evaluates agentic chemical search and multimodal NMR retrieval, but it does not define an official benchmark called CRAG-MM (Callahan et al., 26 Feb 2025). Likewise, “CRAG: Can 3D Generative Models Help 3D Assembly?” introduces a 3D assembly-and-generation method and corresponding evaluation settings, but not a benchmark named CRAG-MM (Jiang et al., 26 Feb 2026). Within the RAG benchmark lineage, the direct precursor is the original CRAG benchmark, not those unrelated acronym uses (Yang et al., 2024).

The official CRAG-MM paper also leaves several boundaries explicit. The benchmark is English-only, web retrieval is text-based only, and image search is simplified into KG-search rather than full open-world image retrieval (Wang et al., 30 Oct 2025). The paper does not provide a formal freshness taxonomy comparable to the temporal dynamism categories of the original CRAG benchmark, even though current information and prices are part of the motivating examples (Wang et al., 30 Oct 2025). Nor does it define a separate retrieval-only leaderboard metric. These omissions do not negate the benchmark’s breadth, but they delimit what it evaluates directly.

CRAG-MM’s significance lies in the conjunction of its design choices: wearable-centered egocentric images, explicit retrieval infrastructure, multi-source and multi-turn task structure, hard negatives, long-tail entities, image-quality perturbations, and truthfulness-oriented scoring (Wang et al., 30 Oct 2025). In that sense, it serves less as a conventional VQA dataset than as a systems benchmark for grounded multimodal assistance under realistic retrieval conditions.