---
title: RAG Challenge Leaderboards
url: https://www.emergentmind.com/topics/rag-challenge-leaderboards
type: topic
---

# RAG Challenge Leaderboards

Retrieval-Augmented Generation (RAG) Challenge Leaderboards provide quantitative and comparative evaluation frameworks for state-of-the-art RAG systems. These leaderboards, emerging in conjunction with benchmarks such as CRAG, CRAG-MM, and LiveRAG, enable systematic measurement of accuracy, faithfulness, hallucination, and robustness across diverse tasks—including multi-source, multi-modal, and multi-turn settings. The leaderboards foster competitive progress, encourage transparency of system architectures and evaluation metrics, and reveal open challenges in grounding, abstention calibration, and fine-grained error diagnosis.

## 1. Principal RAG Challenge Benchmarks and Formats

RAG challenge leaderboards hinge on well-defined competitive benchmarks designed to stress RAG systems under realistic constraints:

- **CRAG and CRAG-MM Benchmarks**: The original CRAG challenge, as in Meta KDD Cup 2024, presents three core subtasks—Web-Only QA, API-Only QA, and Web+API QA—requiring competitors to synthesize answers from restricted web passages and structured APIs, with diverse question domains, temporal dynamics, and entity popularity [2409.15337]. The CRAG-MM extension incorporates multi-modal (image, text) and conversational (multi-turn) tasks, using egocentric images and wearables-inspired scenarios to evaluate RAG for real-world, multi-source settings [2510.26160].

- **LiveRAG and Other Leaderboards**: The SIGIR LiveRAG Challenge leverages the FineWeb-10BT dataset and a standard LLM (Falcon3-10B), assessing participant systems on both correctness (coverage and relevance) and faithfulness, under strict latency and parameterization constraints [2507.04942; 2506.15246; 2506.17493].

- **Open, Evolving Faithfulness Leaderboards**: Complementary to competition-based leaderboards, continuous faithfulness/hallucination leaderboards track LLM generations on summarization and QA tasks, using automated and LLM-judge–based scoring pipelines [2505.04847].

## 2. Core Evaluation Metrics and Scoring Methodology

Leaderboards employ metrics reflecting both factual accuracy and the reliability of grounding in retrieved knowledge:

- **CRAG Score**: $\displaystyle \text{Score} = \frac{1}{N} \sum_{i=1}^N s_i,\quad s_i =  \begin{cases} +1 & \text{correct} \\ 0 & \text{missing/abstention} \\ -1 & \text{incorrect/hallucinated} \end{cases}$.
  
- **Accuracy**: Fraction of examples exactly answered correctly.

- **Hallucination Rate**: Fraction of answers containing unsupported or fabricated facts.

- **Correctness and Faithfulness**: LiveRAG and associated tasks introduce composite correctness $\in [-1, 2]$ and faithfulness $\in [-1, 1]$ metrics. Correctness is computed as the harmonic mean of (i) coverage—the proportion of vital reference facts present—and (ii) relevance or precision—the fraction of on-topic claims. Faithfulness measures the extent to which each atomic claim in the answer can be supported by a retrieved passage, using LLM-as-judge entailment scoring [2507.04942].

- **Truthfulness (CRAG-MM)**: For multi-modal single- or multi-turn QA, per-turn scoring 1 (correct), 0 ("don't know"), –1 (incorrect); early-stopping on consecutive errors for dialogue tasks [2510.26160].

- **Continuous Hallucination Leaderboards (FaithJudge)**: Hallucination rate per model, computed via few-shot–prompted LLM-judge (e.g., o3-mini-high) binary annotation, evaluated on cumulative RAG tasks (summarization, QA, data-to-text). Percentages reflect aggregate hallucination fraction across tasks [2505.04847].

## 3. Notable System Architectures and Methodological Advances

Top competitors and leaderboards consistently reveal methodological innovations that advance SOTA in RAG:

- **Routing and Domain Adaptation**: The APEX system for CRAG implements domain-specific and dynamic routers to control retrieval, augmentation, and generation, allowing for tailored processing pathways (e.g., API for finance/sports), dramatically lowering hallucinations, especially in dynamic or tail-entity queries [2409.15337].

- **Multi-adapter LoRA Specialization**: MARAGS utilizes multiple lightweight adapters on a common Llama 3 backbone, enabling subtask specialization for API call generation, web QA, and hybrid settings, with relabeling of irretrievable questions as abstentions for higher CRAG Score [2409.03171].

- **Hybrid and Clustered Retrieval Pipelines**: TopClustRAG (SIGIR 2025) employs fused BM25 and E5 dense retrieval, SVD-based dimensionality reduction, dynamic K-Means clustering for redundancy filtering, and cluster-specific prompts—all contributing to improved faithfulness and balanced correctness [2506.15246].

- **LLM-based Query Rewriting and Reranking**: Leading LiveRAG submissions (RMIT-ADMS, Magikarp, UDInfo) use LLMs for question decomposition and expansion, hybrid BM25+dense retrieval, and cross-encoder or LLM-based passage reranking, yielding correctness above 1.20 and faithfulness above 0.62 (scale normalized) [2507.04942].

- **Fine-tuning and Knowledge Distillation**: Winning solutions in CRAG-MM leverage multi-task fine-tuning of vision-instruct models, knowledge distillation from closed-source LLMs (e.g., GPT-4o), and preprocessing with object-centric cropping and LLM-based context reranking [2510.26160].

## 4. Leaderboard Results and Comparative Performance

Key leaderboard outcomes are summarized in the following tables and observations:

**CRAG Benchmark (Meta KDD Cup 2024, selected rows from [2409.15337])**

| Team        | Task              | Score (%) | Place         | Accuracy (%) | Hallucination (%) | Missing (%) |
|-------------|-------------------|-----------|---------------|--------------|-------------------|-------------|
| APEX–CRAG   | 1 (Web Only)      | 11.82     | —             | 29.98        | 18.16             | 51.86       |
| APEX–CRAG   | 2 (API Only)      | 31.22     | 2nd           | 46.75        | 15.54             | 37.71       |
| APEX–CRAG   | 3 (Web+API)       | 31.66     | 2nd           | 48.21        | 16.56             | 35.23       |

**LiveRAG Challenge (SIGIR 2025, Session 2, [2507.04942])**

| Rank | Team Name  | Correctness | Faithfulness |
|------|------------|-------------|--------------|
| 1    | Magikarp   | 1.2316      | 0.6565       |
| 2    | UDInfo     | 1.2006      | 0.6232       |
| 3    | RAGtifier  | 1.1345      | 0.5524       |
| 4    | HLTCOE     | 1.0701      | 0.3407       |

**CRAG-MM KDD Cup 2025 (Private Test, [2510.26160])**

| System                  | Accuracy | Missing | Hallucination | Truthfulness | Multi-turn |
|-------------------------|----------|---------|---------------|--------------|------------|
| Llama 3.2 11B RAG       | 35.3%    | 20.8%   | 43.9%         | –8.6%        | 8.8%       |
| Winning Team            | 29.3%    | 61.2%   | 9.6%          | 19.7%        | 26.6%      |
| GPT-5                   | 62.7%    | 6.8%    | 30.5%         | 32.2%        | 45.0%      |

**FaithJudge Hallucination Leaderboard (selected, [2505.04847])**

| Model             | Hallucination Rate |
|-------------------|-------------------|
| gemini-2.5-pro-exp| 7.63 %            |
| gpt-4.5-preview   | 11.94 %           |
| o3-mini-high      | 12.52 %           |

Across these challenges, all SOTA RAG pipelines surpassed no-retrieval or naive retrieval baselines by large margins in correctness, faithfulness, and truthfulness. Hallucination rates exceeding 30% persist in demanding settings such as CRAG-MM single-turn, even as winning teams suppressed hallucinations by up to 70% via abstention or robust reranking [2409.15337; 2510.26160].

## 5. Current Limitations and Open Challenges

Despite system advances, leaderboards expose persistent bottlenecks:

- **High Hallucination Rates and Fragile Robustness**: Truthfulness remains limited—with hallucinations > 30% common in image-based and conversational benchmarks, and performance dropping precipitously for low-light images, occlusion, rare entities, or time-dynamic queries [2510.26160].

- **Metric Design and Evaluation Biases**: Automated or LLM-as-judge scoring (e.g., HHEM-2.1, FaithJudge) yields only moderate reliability (F1-macro ≈ 62–84%) on adversarial hallucination datasets; closed-labeling and single-judge pipelines introduce variance [2505.04847]. All energy is focused on precision/faithfulness, with less scrutiny on diversity, coverage, or multi-task capabilities.

- **Abstention Calibration**: Systematic abstention ("I don't know") reduces hallucinations but increases missing rates, lowering absolute coverage and occasionally reducing aggregate correctness or utility.

- **Resource and Task Generalization**: Many techniques remain narrow: strong domain routing, query rewriting, or subquestion decomposition fail to generalize to multi-modal or continuous updating scenarios.

## 6. Trends and Recommendations for Future Leaderboards

Best practice recommendations and observed trends for future leaderboard design include:

- **Mandated, Transparent Metric Reporting**: Truthfulness, hallucination, correctness, and early stop rates must be clearly published per system; open-source evaluation and error-slice breakdowns are advised [2510.26160].

- **Rich Benchmark Slicing**: Explicit slicing by input modality, question type, entity popularity, robustness (e.g., low-light, occlusion) is necessary to expose system limitations [2510.26160].

- **Continuous and Human-in-the-Loop Evaluation**: Evolving leaderboards should combine LLM-judge scoring with periodic human calibration, use few-shot–prompting from adversarial examples, and regularly update both tasks and scoring models [2505.04847].

- **Emphasis on Multi-turn, Multi-source, and Multi-modal Tasks**: Benchmarks like CRAG-MM signal a shift toward real-world deployment scenarios (wearables, multi-hop dialogue) and require RAG systems to incorporate retrieval fusion, abstention calibration, and bandwidth/latency management.

- **Open Data, Prompts, and Harnesses**: Release of evaluation code, prompts, and labeled examples is recommended to improve replicability, facilitate method transfer, and encourage submissions from both open- and closed-source models [2505.04847].

*This overview draws exclusively from the cited papers, capturing the technical realities, strengths, and boundaries of current RAG Challenge Leaderboards and the evolving landscape of robust, grounded, and trustworthy retrieval-augmented systems.*

Source: https://www.emergentmind.com/topics/rag-challenge-leaderboards