---
title: CRAG Dataset for RAG QA Evaluation
url: https://www.emergentmind.com/topics/crag-dataset
type: topic
---

# CRAG Dataset for RAG QA Evaluation

The Comprehensive RAG Benchmark (CRAG) is a large-scale, multi-domain, and temporally diverse benchmark designed to evaluate the capabilities of Retrieval-Augmented Generation (RAG) systems in factually grounded question answering. Developed to address the limitations of earlier QA datasets, CRAG introduces dynamic, realistic retrieval scenarios by simulating both web search and structured knowledge graph (KG) retrieval, with a strong emphasis on penalizing hallucinations and rewarding abstention in cases of insufficient evidence. CRAG is the official benchmark for the KDD Cup 2024 Retrieval-Augmented Generation challenge and is publicly released, fostering reproducibility and ongoing advances in the RAG research community [2406.04744].

## 1. Motivation and Design Principles

CRAG was constructed to resolve major gaps in existing QA benchmarks. Prior datasets typically focus on static facts, offer limited variety, or lack authentic retrieval noise (e.g., synthetic contexts, absence of unstructured and structured retrieval simulation). CRAG provides a large collection of question–answer pairs that vary across domains, question types, entity popularity, and temporal dynamics. Each question is annotated for fine-grained evaluation, facilitating diagnosis of model performance on nuanced real-world scenarios including highly dynamic queries and low-popularity entities [2406.04744] [2409.15337].

Key design goals include:
- Evaluation of RAG pipelines under authentic, noisy retrieval conditions.
- Exposure of system behavior on queries spanning static to real-time facts.
- Performance assessment over diverse domains and reasoning types.
- Formal penalization of hallucinations, promoting "I don't know" abstention when evidence is lacking.

## 2. Dataset Structure and Annotations

### 2.1 Domains and Question Categories

CRAG encompasses 4,409 QA pairs split across five major domains: Finance, Sports, Music, Movie, and Open (encyclopedic). These domains were selected to balance heavy-traffic applications (e.g., financial data, live sports) and stable, encyclopedic queries such as birth dates and discographies [2406.04744] [2409.15337].

Questions are assigned to one of eight categories, capturing the core reasoning skills required in RAG:

| Type                    | Definition (Example)                                                      |
|-------------------------|---------------------------------------------------------------------------|
| Simple                  | Single-fact lookup (“When was Albert Einstein born?” → Mar 14, 1879)      |
| Simple w. Condition     | Single fact + constraint (“IBM stock price on 01/02/2024?” → $124.50)     |
| Set                     | Set retrieval (“Continents in Southern Hemisphere?” → {Antarctica, ...})   |
| Comparison              | Comparing (“Who debuted earlier, Taylor Swift or Billie Eilish?”)         |
| Aggregation             | Aggregate (“How many Oscar awards for Meryl Streep?” → 3)                 |
| Multi-hop               | Chain reasoning (“Who directed Zendaya’s 2019 movie?” → Jon Favreau)      |
| Post-processing Heavy   | Computed answers (“Days Marshall served on the Supreme Court?”)           |
| False Premise           | Invalid queries (“Taylor Swift’s rap album?” → “invalid question”)        |

### 2.2 Entity Popularity

CRAG stratifies entities in KG-derived questions into head (popular), torso (mid-popularity), and tail (long-tail) buckets, following a search-log derived popularity score. The KG question set is distributed equally: 661 head, 658 torso, 665 tail [2406.04744].

### 2.3 Temporal Dynamism

Questions are manually annotated with four dynamism levels:

| Level         | Description                                                          | Count (%)    |
|---------------|---------------------------------------------------------------------|--------------|
| Real-time     | Second-level update (“Apple’s stock price right now?”)               | 437 (10%)    |
| Fast-changing | Daily fluctuation (“Tonight’s Lakers game?”)                         | 564 (13%)    |
| Slow-changing | Monthly/yearly (“Who won 2023 Grammy?”)                              | 1,007 (23%)  |
| Static        | Invariant fact (“Capital of France?”)                                | 2,401 (54%)  |

Each question includes a `query_time` to support precise temporal evaluation [2406.04744].

## 3. Retrieval Simulation and Data Modalities

### 3.1 Web Search Mocking

A web search API, based on Brave Search, provides up to 50 full HTML pages per question. Each record includes metadata (page name, URL, snippet, last-modified) and raw HTML. This simulates the presence of retrieval noise, such as advertisements and partially relevant content. Estimated recall is 84% for web questions and 63% for KG questions [2406.04744].

### 3.2 Knowledge Graph Mock APIs

A synthetic KG of 2.6 million entities is paired with 38 mock domain-specific JSON endpoints (e.g., price history, movie credits, award history). API access is provided through JSON-defined request/response schemas, delivering attribute–value records. To simulate realistic retrieval, the KG comprises authentic and “hard negative” entities [2406.04744] [2409.15337].

## 4. Task Definitions and Evaluation Protocols

### 4.1 CRAG Tasks

CRAG is partitioned into three tasks of increasing complexity:

| Task   | Input Modality                | Objective                                                                     |
|--------|------------------------------|--------------------------------------------------------------------------------|
| Task 1 | 5 web pages                  | Retrieval summarization; synthesize answer from noisy, limited text corpus     |
| Task 2 | 5 web pages + mock KG APIs   | Integrate web retrieval and structured KG info for answer synthesis            |
| Task 3 | 50 web pages + mock KG APIs  | End-to-end; test large-scale retrieval, ranking, filtration, API integration   |

For every task, systems must answer or abstain (“I don’t know”). Task 3 emulates real-world user information-seeking, with extensive retrieval noise and limited memory/time [2406.04744] [2409.15337] [2409.03171].

### 4.2 Dataset Splits

Typical splits include ~1,323 examples each for validation and public test (total 2,706) and a hidden private test for challenge evaluation. Retrieval content per question is provided according to task requirements [2409.15337].

## 5. Evaluation Metrics and Human-in-the-Loop Scoring

Formal metrics for CRAG responses include:

- **Accuracy**: fraction of correct (perfect or acceptable) answers.
- **Hallucination rate**: fraction of answers that are incorrect or unsupportable.
- **Missing rate**: fraction of abstentions (“I don’t know”).
- **Scoreₐ**: auto-eval metric: 

  $$
  \mathrm{Score}_a = \frac{\#\text{Accurate} - \#\text{Incorrect}}{N}
  $$

- **Scoreₕ** (human-eval): answers labeled as Perfect (+1), Acceptable (+0.5), Missing (0), or Incorrect (–1), and mean taken [2406.04744].

Human evaluation is supplemented by LLM-based "judge" systems for scalability [2601.02522]. The protocol strongly penalizes unsupported or hallucinated answers, thereby promoting reliable uncertainty estimation by RAG models.

## 6. Empirical Results and System Analysis

CRAG exposes robust differences between RAG pipelines and traditional LLM-only QA. Key findings include:

- GPT-4 Turbo in a straightforward RAG setup attains only a modest 10-point accuracy improvement over LLM-only (44% vs. 34%), at the cost of increased hallucination. SOTA RAG industry systems yield a maximum 63% perfect answers, with hallucination rates of 17–25% persisting [2406.04744].
- Slice-level analyses indicate the greatest difficulties in Finance and Sports domains, real-time and fast-changing queries (<20% auto-score), tail entities, and complex categories such as set, aggregation, and false-premise questions.
- Enhanced systems demonstrate significant gains via domain and dynamism routing, specialized preprocessing (e.g., HTML parsing, chunk ranking), and practical abstention strategies [2409.15337][2409.03171][2408.05141].
- Energy and latency experiments confirm that CRAG supports measurement of performance trade-offs under realistic workloads; judicious choice of retrieval parameters reduces energy usage without severe accuracy loss [2601.02522].

## 7. Availability, Community Adoption, and Impact

The CRAG dataset, along with all supporting mock APIs and code, is publicly available under a CC BY-NC license at [https://github.com/facebookresearch/CRAG/](https://github.com/facebookresearch/CRAG/) and [competition portals](https://www.aicrowd.com/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024). It serves as the official benchmark for the Meta KDD Cup 2024, attracting thousands of participants and submissions [2406.04744]. The dataset is maintained and updated, with expansion into additional domains and question types anticipated.

CRAG has rapidly become the de facto benchmark for rigorous analysis of RAG QA systems, catalyzing research into robust retrieval, hallucination mitigation, structured/unstructured information integration, and temporal reasoning. Its formal scoring, coverage of real-world search conditions, and fine-grained annotations position it as a central resource for both system development and diagnostic research [2406.04744][2409.15337][2409.03171][2601.02522][2408.05141].

Source: https://www.emergentmind.com/topics/crag-dataset