RAG Challenge Leaderboards

• RAG Challenge Leaderboards are evaluation frameworks that measure retrieval-augmented generation systems based on accuracy, faithfulness, and hallucination across complex tasks.
• They utilize benchmarks like CRAG, CRAG-MM, and LiveRAG with defined subtasks and specific metrics such as the CRAG score to rigorously assess system performance.
• These leaderboards drive innovation by highlighting advances in dynamic routing, multi-adapter fine-tuning, and hybrid retrieval while uncovering challenges in grounding and abstention calibration.

Retrieval-Augmented Generation (RAG) Challenge Leaderboards provide quantitative and comparative evaluation frameworks for state-of-the-art RAG systems. These leaderboards, emerging in conjunction with benchmarks such as CRAG, CRAG-MM, and LiveRAG, enable systematic measurement of accuracy, faithfulness, hallucination, and robustness across diverse tasks—including multi-source, multi-modal, and multi-turn settings. The leaderboards foster competitive progress, encourage transparency of system architectures and evaluation metrics, and reveal open challenges in grounding, abstention calibration, and fine-grained error diagnosis.

1. Principal RAG Challenge Benchmarks and Formats

RAG challenge leaderboards hinge on well-defined competitive benchmarks designed to stress RAG systems under realistic constraints:

• CRAG and CRAG-MM Benchmarks: The original CRAG challenge, as in Meta KDD Cup 2024, presents three core subtasks—Web-Only QA, API-Only QA, and Web+API QA—requiring competitors to synthesize answers from restricted web passages and structured APIs, with diverse question domains, temporal dynamics, and entity popularity (Ouyang et al., 2024). The CRAG-MM extension incorporates multi-modal (image, text) and conversational (multi-turn) tasks, using egocentric images and wearables-inspired scenarios to evaluate RAG for real-world, multi-source settings (Wang et al., 30 Oct 2025).
• LiveRAG and Other Leaderboards: The SIGIR LiveRAG Challenge leverages the FineWeb-10BT dataset and a standard LLM (Falcon3-10B), assessing participant systems on both correctness (coverage and relevance) and faithfulness, under strict latency and parameterization constraints (Carmel et al., 7 Jul 2025, Bakagianni et al., 18 Jun 2025, Martinez et al., 20 Jun 2025).
• Open, Evolving Faithfulness Leaderboards: Complementary to competition-based leaderboards, continuous faithfulness/hallucination leaderboards track LLM generations on summarization and QA tasks, using automated and LLM-judge–based scoring pipelines (Tamber et al., 7 May 2025).

2. Core Evaluation Metrics and Scoring Methodology

Leaderboards employ metrics reflecting both factual accuracy and the reliability of grounding in retrieved knowledge:

• CRAG Score: $$\displaystyle \text{Score} = \frac{1}{N} \sum_{i=1}^N s_i,\quad s_i = \begin{cases} +1 & \text{correct} \ 0 & \text{missing/abstention} \ -1 & \text{incorrect/hallucinated} \end{cases}$$.
• Accuracy: Fraction of examples exactly answered correctly.
• Hallucination Rate: Fraction of answers containing unsupported or fabricated facts.
• Correctness and Faithfulness: LiveRAG and associated tasks introduce composite correctness $\in [-1, 2]$ and faithfulness $\in [-1, 1]$ metrics. Correctness is computed as the harmonic mean of (i) coverage—the proportion of vital reference facts present—and (ii) relevance or precision—the fraction of on-topic claims. Faithfulness measures the extent to which each atomic claim in the answer can be supported by a retrieved passage, using LLM-as-judge entailment scoring (Carmel et al., 7 Jul 2025).
• Truthfulness (CRAG-MM): For multi-modal single- or multi-turn QA, per-turn scoring 1 (correct), 0 ("don't know"), –1 (incorrect); early-stopping on consecutive errors for dialogue tasks (Wang et al., 30 Oct 2025).
• Continuous Hallucination Leaderboards (FaithJudge): Hallucination rate per model, computed via few-shot–prompted LLM-judge (e.g., o3-mini-high) binary annotation, evaluated on cumulative RAG tasks (summarization, QA, data-to-text). Percentages reflect aggregate hallucination fraction across tasks (Tamber et al., 7 May 2025).

3. Notable System Architectures and Methodological Advances

Top competitors and leaderboards consistently reveal methodological innovations that advance SOTA in RAG:

• Routing and Domain Adaptation: The APEX system for CRAG implements domain-specific and dynamic routers to control retrieval, augmentation, and generation, allowing for tailored processing pathways (e.g., API for finance/sports), dramatically lowering hallucinations, especially in dynamic or tail-entity queries (Ouyang et al., 2024).
• Multi-adapter LoRA Specialization: MARAGS utilizes multiple lightweight adapters on a common Llama 3 backbone, enabling subtask specialization for API call generation, web QA, and hybrid settings, with relabeling of irretrievable questions as abstentions for higher CRAG Score (DeHaven, 2024).
• Hybrid and Clustered Retrieval Pipelines: TopClustRAG (SIGIR 2025) employs fused BM25 and E5 dense retrieval, SVD-based dimensionality reduction, dynamic K-Means clustering for redundancy filtering, and cluster-specific prompts—all contributing to improved faithfulness and balanced correctness (Bakagianni et al., 18 Jun 2025).
• LLM-based Query Rewriting and Reranking: Leading LiveRAG submissions (RMIT-ADMS, Magikarp, UDInfo) use LLMs for question decomposition and expansion, hybrid BM25+dense retrieval, and cross-encoder or LLM-based passage reranking, yielding correctness above 1.20 and faithfulness above 0.62 (scale normalized) (Carmel et al., 7 Jul 2025).
• Fine-tuning and Knowledge Distillation: Winning solutions in CRAG-MM leverage multi-task fine-tuning of vision-instruct models, knowledge distillation from closed-source LLMs (e.g., GPT-4o), and preprocessing with object-centric cropping and LLM-based context reranking (Wang et al., 30 Oct 2025).

4. Leaderboard Results and Comparative Performance

Key leaderboard outcomes are summarized in the following tables and observations:

CRAG Benchmark (Meta KDD Cup 2024, selected rows from (Ouyang et al., 2024))

Team	Task	Score (%)	Place	Accuracy (%)	Hallucination (%)	Missing (%)
APEX–CRAG	1 (Web Only)	11.82	—	29.98	18.16	51.86
APEX–CRAG	2 (API Only)	31.22	2nd	46.75	15.54	37.71
APEX–CRAG	3 (Web+API)	31.66	2nd	48.21	16.56	35.23

LiveRAG Challenge (SIGIR 2025, Session 2, (Carmel et al., 7 Jul 2025))

Rank	Team Name	Correctness	Faithfulness
1	Magikarp	1.2316	0.6565
2	UDInfo	1.2006	0.6232
3	RAGtifier	1.1345	0.5524
4	HLTCOE	1.0701	0.3407

CRAG-MM KDD Cup 2025 (Private Test, (Wang et al., 30 Oct 2025))

System	Accuracy	Missing	Hallucination	Truthfulness	Multi-turn
Llama 3.2 11B RAG	35.3%	20.8%	43.9%	–8.6%	8.8%
Winning Team	29.3%	61.2%	9.6%	19.7%	26.6%
GPT-5	62.7%	6.8%	30.5%	32.2%	45.0%

FaithJudge Hallucination Leaderboard (selected, (Tamber et al., 7 May 2025))

Model	Hallucination Rate
gemini-2.5-pro-exp	7.63 %
gpt-4.5-preview	11.94 %
o3-mini-high	12.52 %

Across these challenges, all SOTA RAG pipelines surpassed no-retrieval or naive retrieval baselines by large margins in correctness, faithfulness, and truthfulness. Hallucination rates exceeding 30% persist in demanding settings such as CRAG-MM single-turn, even as winning teams suppressed hallucinations by up to 70% via abstention or robust reranking (Ouyang et al., 2024, Wang et al., 30 Oct 2025).

5. Current Limitations and Open Challenges

Despite system advances, leaderboards expose persistent bottlenecks:

• High Hallucination Rates and Fragile Robustness: Truthfulness remains limited—with hallucinations > 30% common in image-based and conversational benchmarks, and performance dropping precipitously for low-light images, occlusion, rare entities, or time-dynamic queries (Wang et al., 30 Oct 2025).
• Metric Design and Evaluation Biases: Automated or LLM-as-judge scoring (e.g., HHEM-2.1, FaithJudge) yields only moderate reliability (F1-macro ≈ 62–84%) on adversarial hallucination datasets; closed-labeling and single-judge pipelines introduce variance (Tamber et al., 7 May 2025). All energy is focused on precision/faithfulness, with less scrutiny on diversity, coverage, or multi-task capabilities.
• Abstention Calibration: Systematic abstention ("I don't know") reduces hallucinations but increases missing rates, lowering absolute coverage and occasionally reducing aggregate correctness or utility.
• Resource and Task Generalization: Many techniques remain narrow: strong domain routing, query rewriting, or subquestion decomposition fail to generalize to multi-modal or continuous updating scenarios.

6. Trends and Recommendations for Future Leaderboards

Best practice recommendations and observed trends for future leaderboard design include:

• Mandated, Transparent Metric Reporting: Truthfulness, hallucination, correctness, and early stop rates must be clearly published per system; open-source evaluation and error-slice breakdowns are advised (Wang et al., 30 Oct 2025).
• Rich Benchmark Slicing: Explicit slicing by input modality, question type, entity popularity, robustness (e.g., low-light, occlusion) is necessary to expose system limitations (Wang et al., 30 Oct 2025).
• Continuous and Human-in-the-Loop Evaluation: Evolving leaderboards should combine LLM-judge scoring with periodic human calibration, use few-shot–prompting from adversarial examples, and regularly update both tasks and scoring models (Tamber et al., 7 May 2025).
• Emphasis on Multi-turn, Multi-source, and Multi-modal Tasks: Benchmarks like CRAG-MM signal a shift toward real-world deployment scenarios (wearables, multi-hop dialogue) and require RAG systems to incorporate retrieval fusion, abstention calibration, and bandwidth/latency management.
• Open Data, Prompts, and Harnesses: Release of evaluation code, prompts, and labeled examples is recommended to improve replicability, facilitate method transfer, and encourage submissions from both open- and closed-source models (Tamber et al., 7 May 2025).

This overview draws exclusively from the cited papers, capturing the technical realities, strengths, and boundaries of current RAG Challenge Leaderboards and the evolving landscape of robust, grounded, and trustworthy retrieval-augmented systems.