Meta CRAG-MM Challenge Overview

• Meta CRAG-MM Challenge is a benchmark for multi-modal, wearable factual QA integrating egocentric images, external retrieval, and dialogue.
• The challenge emphasizes retrieval-augmented generation with explicit penalties for hallucination and conservative abstention strategies.
• Early competition results from over 1K participants highlight significant performance gaps, underscoring the need for improved confidence estimation and evidence fusion.

The Meta CRAG-MM Challenge is a KDD Cup 2025 competition built on CRAG-MM, a benchmark for multi-modal, multi-turn retrieval-augmented generation in wearable-style settings. It targets systems that answer factual questions about images—especially egocentric images from smart glasses—by combining visual understanding with external retrieval over an image-based knowledge graph and web corpora. CRAG-MM extends the earlier CRAG benchmark from text-centric factual QA to multi-modal, dialogue-based, and wearable-oriented scenarios, while preserving its central concern: truthful answering under explicit hallucination penalties (Wang et al., 30 Oct 2025, Yang et al., 2024).

1. Lineage and problem setting

CRAG-MM inherits the design philosophy of the original Comprehensive RAG Benchmark, which was introduced to evaluate realistic factual QA under web retrieval, mock knowledge graphs, temporal dynamism, long-tail entities, and explicit hallucination costs (Yang et al., 2024). The multi-modal extension is motivated by wearable devices such as smart glasses, where users ask information-seeking questions about entities in view, but the visual input is egocentric, noisy, and often insufficient without external knowledge (Wang et al., 30 Oct 2025).

This setting differs from conventional VQA in two ways. First, most questions are knowledge-intensive rather than purely perceptual. Second, the benchmark is explicitly organized around retrieval, not only recognition: the system must identify what is in the image, decide whether external evidence is needed, retrieve from controlled APIs and corpora, and answer without fabricating unsupported facts. The challenge therefore sits at the intersection of VLM inference, MM-RAG, tool use, calibration, and conversational state tracking (Wang et al., 30 Oct 2025).

The competition scale also made CRAG-MM a community benchmark rather than a one-off shared task. The benchmark paper reports that KDD Cup 2025 attracted about 1K participants and 5K submissions, and that winning solutions improved baseline performance by 28%, indicating early but measurable progress on the benchmark’s truthfulness-oriented objective (Wang et al., 30 Oct 2025).

2. Benchmark composition and task structure

CRAG-MM contains 7,943 images, including 6,248 egocentric images and 1,695 public images, together with 6,462 single-turn image-question-answer triplets and 1,956 multi-turn conversations averaging 4.9 turns. It spans 13 domains and was explicitly constructed to reflect real-world wearable usage, including five image-quality issues, six question types, varying entity popularity, differing information dynamism, and multi-turn conversational context (Wang et al., 30 Oct 2025).

The domain coverage includes Animal, Book, Food, General Object Recognition, Local, Math & Science, Plants & Gardening, Shopping, Sports & Games, Style & Fashion, Text Understanding, Vehicle, and Other. Question types are Simple-recognition, Simple-knowledge, Multi-hop, Comparison, Aggregation, and Reasoning. The image-quality stressors are Low-light, Blurred, Truncated, Occluded, and Rotated (Wang et al., 30 Oct 2025).

A substantial fraction of the benchmark is deliberately retrieval-dependent. The benchmark paper states that about 89% of questions require external knowledge, and that roughly 52% are complex in the sense of being Multi-hop, Comparison, Aggregation, or Reasoning questions. This shifts the benchmark away from closed-book VQA and toward explicit MM-RAG system design (Wang et al., 30 Oct 2025).

Task	Interaction setting	Retrieval scope
Task 1: Single-source augmentation	Single-turn image QA	Image-KG API only
Task 2: Multi-source augmentation	Single-turn image QA	Image-KG API and web search API
Task 3: Multi-turn conversations	Dialogue over images	Image-KG API, web search API, and conversation history

The retrieval environment is correspondingly large. The benchmark paper reports an image KG with 68K images and 26K entities, and a web corpus with 800K webpage URLs and 2.7M text chunks (Wang et al., 30 Oct 2025). Challenge descriptions in solution papers further specify an image search API that returns visually similar images plus structured metadata, and a web search API that returns up to 50 webpages with URL, title, snippet, timestamp, and full HTML, indexed with CLIP for images and bge-large-en-v1.5 for text (Jiang et al., 7 Aug 2025).

3. Evaluation regime and the centrality of abstention

The challenge is designed so that unsupported answering is actively harmful. One solution paper describes the automatic grading setup as assigning labels Correct, Partially correct, Incorrect, and Missing/refusal, with the aggregate score reported as

$$\text{Trustfulness}=\frac{1}{N}\sum_{i=1}^{N}s_i,\quad s_i\in\{-1,0,+1\},$$

where correct answers receive $+1$, partially correct or refusal answers receive $0$, and incorrect answers receive $-1$ (Nakamizo et al., 16 Oct 2025). Another challenge report describes a closely related manual-review scale with Perfect $=1.0$, Acceptable $=0.5$, Missing $=0.0$, and Incorrect $=-1.0$, and notes that only the first 75 BPE tokens are scored automatically before final human review (Chen et al., 27 Jul 2025).

This scoring regime changes system optimization. Because an incorrect answer is worse than no answer, competitive systems frequently trade answer coverage for hallucination suppression. The behavior is especially salient in multi-turn QA, where one reported competition setup terminates a conversation after two consecutive incorrect answers, with all remaining turns counted as Missing (Zhang et al., 14 Aug 2025). A plausible implication is that CRAG-MM is as much a benchmark of calibration and selective abstention as of retrieval and reasoning.

Latency constraints reinforce this design. Solution papers report 10 seconds per conversational turn and 30 seconds per dialogue in the multi-turn setting, which favors lightweight routing, conservative retrieval policies, and compact verification stages over exhaustive search or long-form deliberation (Jiang et al., 7 Aug 2025).

4. Baselines and empirical difficulty

CRAG-MM is difficult even for straightforward MM-RAG pipelines. The benchmark paper reports that straightforward RAG approaches achieve only 32% truthfulness on single-turn QA and 43% on multi-turn QA, while state-of-the-art industry solutions reach similar quality at 32% and 45%, respectively (Wang et al., 30 Oct 2025). This establishes that the challenge is not saturated by off-the-shelf retrieval plus generation.

Task-specific studies reinforce that point. QA-Dragon reports that, on Task 1, an LLM-only baseline achieved 15.79% accuracy and 18.98% knowledge overlap, CoT reached 16.25% and 20.07%, Direct RAG reached 14.40% and 22.67%, and QA-Dragon reached 21.31% and 41.09%. On Task 2, QA-Dragon reached 23.22% accuracy and 41.77% overlap versus 16.87% and 27.15% for Direct RAG; on Task 3, it reached 24.78% and 48.26% versus 20.95% and 33.74% (Jiang et al., 7 Aug 2025).

Ablations from the Dianping-Trust-Safety solution show how severe the hallucination problem is under the challenge metric. On a local single-turn evaluation set, the base vision model produced 0.197 accuracy, 0.085 missing, 0.718 hallucination, and $-0.520$ truthfulness. SFT improved accuracy to 0.262 and reduced hallucination to 0.501, but truthfulness remained negative at $-0.238$. Full SFT+RL+curriculum+RAG reached 0.282 accuracy, 0.587 missing, 0.131 hallucination, and 0.151 truthfulness, illustrating the benchmark’s characteristic trade-off between correctness and abstention (Zhang et al., 14 Aug 2025).

These results suggest that CRAG-MM penalizes naïve confidence more heavily than many existing multimodal benchmarks. Strong retrieval helps, but only when combined with query routing, refusal behavior, and answer verification.

5. Competition strategies and representative systems

High-ranking systems did not converge on a single architecture, but they did converge on a small set of tactics: query routing, conservative answer gating, selective retrieval, and post-hoc verification.

The y3h2 team’s 5th-place system is the clearest example of pure hallucination control. It kept meta-llama/Llama-3.2-11B-Vision-Instruct unchanged, did not use external retrieval in the final submission, and instead trained 65 logistic-regression detectors over hidden states and attention-head outputs to estimate whether an answer was likely correct. On single-turn public-test data, the baseline had Accuracy 0.207, Missing 0.058, Hallucination 0.735, and Trustfulness $+1$0; the combined hidden-state plus attention-head filter reduced hallucination to 0.045 and moved trustfulness to 0.036, albeit with Accuracy falling to 0.082. The final private leaderboard scores were 0.036 on Task 1, 0.034 on Task 2, and 0.104 on Task 3 (Nakamizo et al., 16 Oct 2025).

The CRUISE system, which placed 3rd in Task 1, adopted a verification-centric MM-RAG design: a lightweight query router, query-aware retrieval and summarization, dual-path generation with and without retrieved context, and Chain-of-Verification-style post-hoc filtering. In local Task-1 evaluation, a naïve vision-only baseline had 25.00% accuracy, 59.62% hallucination, and $+1$1% truthfulness score, while the full pipeline reached 14.42% accuracy, 2.88% hallucination, and 11.54% truthfulness score. The method is a canonical example of challenge-aware conservatism: it answers less often, but its answers are far less harmful under the scoring rule (Chen et al., 27 Jul 2025).

QA-Dragon represents the opposite end of the design space: a fully dynamic MM-RAG stack with a domain router, search router, tool router, image search agent, text search agent, coarse-to-fine multimodal reranking, and dual verification. Its main contribution is not abstention alone, but better retrieval orchestration and evidence fusion, yielding consistent gains over LLM-only, CoT, and Direct RAG baselines across all three tasks (Jiang et al., 7 Aug 2025).

The Dianping-Trust-Safety solution combines these themes with training-centric optimization. It uses SFT with GPT-4.1 distillation, then reinforcement learning with a three-stage curriculum over easy and hard samples to avoid the “always refuse” collapse that appears under direct RL. The system achieved 1st place in Task 1 and 3rd place in Task 3, and the paper states that in manual evaluation its Task-1 truthfulness score outperformed the second-place team by 52.38% (Zhang et al., 14 Aug 2025).

6. Significance, controversies, and likely research directions

CRAG-MM establishes a benchmark regime in which multimodal grounding, retrieval quality, and calibrated refusal are inseparable. Its reported baseline gap—32%/43% for straightforward RAG and 32%/45% for industry solutions on single-turn/multi-turn truthfulness—shows that conventional MM-RAG pipelines remain brittle in wearable settings (Wang et al., 30 Oct 2025). The benchmark’s large egocentric component, explicit image-quality stressors, and multi-turn termination rules make it a stronger test of deployment realism than standard VQA.

One recurring controversy is whether the benchmark rewards over-conservatism. Several successful systems achieve strong truthfulness by sharply increasing Missing rates, sometimes above 80% in ablation settings (Nakamizo et al., 16 Oct 2025, Chen et al., 27 Jul 2025). This does not invalidate the benchmark; rather, it exposes a genuine systems problem. In a regime where hallucinations are costly, the benchmark asks whether a model can remain useful without becoming recklessly speculative. This suggests that future progress will depend on better confidence estimation, not merely stronger backbones.

Subsequent evaluation research around visual RAG points in the same direction. MRAG-Suite argues that difficulty-based and ambiguity-aware filtering substantially reduces apparent performance and that claim-level multimodal diagnostics are needed to expose unsupported long-form explanations (Ji, 29 Sep 2025). MetaRAG, in turn, proposes reference-free factoid-level metamorphic testing for hallucination detection in RAG systems, with span-level localization of unsupported claims (Sok et al., 11 Sep 2025). A plausible extension of the Meta CRAG-MM Challenge is therefore a next-generation evaluation stack that supplements truthfulness with claim-level grounding, ambiguity sensitivity, and modality-specific evidence attribution.

In that sense, the Meta CRAG-MM Challenge is not only a competition. It is an empirical program for multimodal factuality under retrieval, dialogue, and wearable perception constraints: one that has already shown that better routing, better verification, and better calibration matter as much as better vision-language modeling (Wang et al., 30 Oct 2025).