---
title: 'CRAG-MM: Benchmark for MM-RAG Evaluation'
url: https://www.emergentmind.com/topics/crag-mm
type: topic
---

# CRAG-MM: Benchmark for MM-RAG Evaluation

CRAG-MM (Comprehensive Retrieval-Augmented Generation Benchmark – Multi-Modal Multi-Turn) is the first comprehensive public benchmark specifically designed to evaluate multi-modal, multi-turn Retrieval-Augmented Generation (RAG) methods, targeting the vision-and-language QA demands of wearable AI scenarios such as smart glasses and AI pins. It emphasizes robust evaluation of MM-RAG models under realistic, challenging conditions, including egocentric images, long-tail entities, varied image qualities, multi-source retrieval, and extended conversational continuity [2510.26160].

## 1. Motivation and Scope

CRAG-MM was introduced to address a critical gap in evaluation resources for MM-RAG systems, especially those supporting on-the-go fact-seeking in wearable contexts [2510.26160]. Unlike static VQA benchmarks, CRAG-MM focuses on scenarios where:

- The primary visual input is egocentric—captured from a first-person perspective—with inherent degradation (blur, occlusion, low-light).
- Questions require grounding in external structured (image-KG) and unstructured (web) sources; answers are virtually impossible from pixels alone.
- Dialogues naturally unfold over multiple turns, exhibiting topic drift, accumulated context, and dependency resolution.

This suite is aligned with next-generation AI assistant use-cases, as exemplified by KDD Cup 2025, which served as the deployment venue and drew widespread academic and industry participation [2507.21520][2507.20136][2510.14330].

## 2. Dataset Composition and Stratified Challenge Design

CRAG-MM contains:

- 7,943 images: 6,248 (79%) are egocentric (from wearable devices, e.g., Ray-Ban Meta smart glasses), designed to include 15% with deliberate degradation (blur, occlusion, truncation, rotation, low-light).
- 6,462 single-turn (image, question, answer) triplets and 1,956 multi-turn conversations (each 2–6 turns, mean 4.9) spanning 13 domains including shopping, text understanding, plants, and vehicles.
- Entity buckets sampled for head, mid, and tail frequency to probe retrieval and grounding performance on long-tail concepts.

Annotations and splits are stratified across four orthogonal axes: image quality, question type (six categories: text recognition, lookup, multi-hop, comparison, aggregation, reasoning), entity popularity, and dialog complexity (38% of multi-turn sessions involve domain shift; user abandonment after two failed turns is simulated).

## 3. Tasks and Retrieval Infrastructure

CRAG-MM operationalizes three progressively complex tasks:

| Task                       | External Retrieval APIs            | Input Context                                 | Complexity                      |
|----------------------------|------------------------------------|-----------------------------------------------|----------------------------------|
| Single-Source Augmentation | Image-KG (68K images, 26K entities)| I, Q, R_I (K visually similar images + metadata) | One question, one retrieval source |
| Multi-Source Augmentation  | Image-KG + Web search (800K chunks, Brave) | I, Q, R_I, R_W (up to 50 web hits/query)    | Adds noisy web context, multi-source |
| Multi-Turn Conversation    | Same as above                      | I, Q_t, R_I, R_W, H_{<t} (history)            | Requires sustained context, topic drift |

Retrieval employs CLIP ViT-L/14@336px for image embeddings and ChromaDB; simulated web retrieval introduces ∼1:20 relevant/irrelevant ratio (image) and 1:2 (web) to reflect practical noise [2510.26160].

Formally, for single-turn with image-KG:
$$
A = \text{MM-LLM}(I, Q, R_I), \quad R_I = \{ (i_k, m_k) \}_{k=1}^K \leftarrow \text{ImageSearch}(I)
$$
The pipeline extends to include web search and conversation history in subsequent tasks.

## 4. Evaluation Protocol and Metrics

CRAG-MM enforces a strict truthfulness metric:
- Each answer labeled as Perfect (+1), Missing/Refusal ("I don't know", 0), or Hallucinated/Incorrect (–1).
- Truthfulness for $N$ queries: 
$$
\text{Truth} = \frac{\#\text{perfect} \cdot 1 + \#\text{incorrect} \cdot (–1) + \#\text{missing} \cdot 0}{N}
$$
- Multi-turn sessions are truncated after two non-positive turns (mimicking real user abandonment). Per-session truthfulness averages across valid turns; final scores average across sessions [2510.26160].
- Leaderboard and competition rankings (KDD Cup 2025) use this metric; final placement for top teams involves human correction of LLM-grader outputs [2507.21520].

## 5. Baseline, SOTA Results, and Leading Approaches

CRAG-MM results highlight significant challenges:

- Baseline vision-only MM-LLM: Truth ≈ 18% (single-turn), 30% (multi-turn).
- Task 1 (image-KG): Truth ≈ 22%.
- Task 2 (image-KG + web): Straightforward RAG approaches including commercial SOTA (e.g., GPT-5, Gemini, Claude Sonnet with proprietary retrieval) yield ≈ 32% (single-turn), 43–45% (multi-turn), with hallucination rates >25%.
- Winning team (KDD Cup 2025, Llama-3.2-11B fine-tuned + distillation): +28% (single-turn) and +18% (multi-turn) truth improvement over baseline, with further hallucination reduction at the expense of increased "I don't know" responses.

Advanced solutions utilize multi-stage pipelines: query classification, retrieval diversification, reranking via fine-tuned LLMs, and integrated verification (e.g. Chain-of-Verification, self-consistency checks) [2507.21520][2507.20136][2510.14330]. Hallucination-focused strategies achieve >90% reduction in false positives but induce a 40–60% drop in coverage, trading off recall for precision [2510.14330][2507.20136].

## 6. Impact, Insights, and Limitations

CRAG-MM's competitive environment (≈1,000 participants, 5,000 submissions at KDD Cup 2025) catalyzed rapid method development. Key experimental insights include:

- Truthfulness degrades substantially (up to 46% drop) on low-quality egocentric images, exposing the limits of current visual encoding.
- Long-tail entities are consistently under-recognized and under-retrieved, emphasizing the need for improved entity linking and retriever coverage.
- Multi-hop, aggregation, and comparison queries remain major failure points, evidencing open problems in cross-modal reasoning.
- Session robustness is constrained: >25% of multi-turn sessions are truncated by SOTA models, with average session lengths at 3.2/4.9 turns [2510.26160].

A key contribution of CRAG-MM is anchoring a reproducible, publicly available test suite with fine-grained scoring and retrieval APIs, enabling precise ablation and benchmarking of future methods.

## 7. Future Directions and Challenges

CRAG-MM's design foregrounds unresolved issues in MM-RAG:

- Robustness to challenging, real-world egocentric input and long-tail entity exposures.
- Mitigation of hallucinated outputs without sacrificing answer coverage—moving beyond aggressive abstention-only strategies.
- End-to-end architectures capable of sustained, stateful dialogue over noisy multi-modal contexts.
- Improved retrieval fusion and reasoning techniques for both structured and unstructured evidence.
- Practical integration with latency and hardware constraints constrained by mobile/wearable device requirements.

Planned development includes refining the retrieval infrastructure, extending domain and image coverage, and stimulating advancement on these fronts by annual community competition [2510.26160][2507.21520][2507.20136].

---

**References:**  
[2510.26160], [2510.14330], [2507.21520], [2507.20136]

Source: https://www.emergentmind.com/topics/crag-mm