TREC RAG 2024: Pioneering RAG Evaluation

• TREC RAG 2024 is the inaugural track that defined core tasks, data infrastructure, and evaluation methods for end-to-end retrieval-augmented generation systems with explicit citations.
• It introduced the MS MARCO V2.1 corpus and the Ragnarök framework to standardize retrieval, reranking, and generation into a unified, reproducible pipeline.
• The track’s multi-layered evaluation methodology combined retrieval relevance, nugget-based metrics, and support assessments to diagnose and improve system performance.

TREC RAG 2024 was the inaugural edition of the Retrieval-Augmented Generation track at the Text REtrieval Conference, organized around end-to-end systems that combine retrieval and generation to answer information needs with explicit citations to supporting evidence. It established a shared problem framing, a common corpus and software stack, and a multi-layered evaluation methodology spanning retrieval relevance, response completeness, and factual grounding. Later overview papers describe the 2025 track as the “second edition” that “builds on the foundation” of the inaugural 2024 track, which had already defined the core tasks, data infrastructure, and evaluation concepts for community-wide RAG benchmarking (Pradeep et al., 2024, Upadhyay et al., 10 Mar 2026).

1. Founding scope and problem framing

The track was introduced as an arena to “build, test, visualize, and systematically evaluate RAG-based search systems,” motivated by the shift from ranked lists of documents to attributed summaries generated from retrieved evidence (Pradeep et al., 2024). In this framing, a RAG system is not only a generator conditioned on retrieved context, but an end-to-end pipeline in which retrieval quality, reranking, answer synthesis, and citation behavior are all first-class objects of evaluation.

Later retrospectives characterize the 2024 information needs as shorter and more “keyword-style” than the “multi-sentence, long and complex narratives” adopted in 2025, while still being suitable for multi-sentence cited answers (Upadhyay et al., 10 Mar 2026). At the same time, a large support-assessment study conducted on 36 judged topics describes those 2024 topics as “non-factoid, multifaceted, often subjective, long-form queries” drawn from fresh Bing Search logs via the “Researchy Questions” methodology, indicating that the track already targeted materially richer questions than classical factoid QA (Thakur et al., 21 Apr 2025). Taken together, these descriptions suggest a 2024 task space centered on open-domain, evidence-grounded answer generation for comparatively concise but still decompositional information needs.

2. Corpus, segmentation, and infrastructure

A central contribution of the 2024 track was the construction of the MS MARCO V2.1 document and segment collections for RAG evaluation (Pradeep et al., 2024). MS MARCO V2.1 was derived from MS MARCO V2 through document deduplication followed by segmentation into overlapping passages suitable for retrieval and citation.

Resource	Construction	Size
MS MARCO V2.1 documents	Deduplicated from MS MARCO V2 with MinHash + LSH over 9-gram shingles	10,960,555
MS MARCO V2.1 segments	Sliding window over sentences, window size 10 and stride 5	113,520,750
Original V2 documents	Pre-deduplication source collection	11,959,635

The segmentation procedure produced overlapping segments of roughly 500–1000 characters, with metadata including start_char and end_char offsets in the source document (Pradeep et al., 2024). Later track documentation emphasized that, unlike earlier MS MARCO passage resources that contained short human-readable answer passages, the V2.1 segments were generated from full documents with a sliding window, yielding more “natural” content suitable for RAG (Upadhyay et al., 10 Mar 2026).

The track infrastructure was standardized in the “Ragnarök” framework (Pradeep et al., 2024). Ragnarök coupled Pyserini-based retrieval, rank_llm reranking, REST APIs, and a Gradio-based web interface, and exposed a unified JSON I/O format for retrieval-only, generation-only, and end-to-end systems. It also released development topics, notably TREC-RAGgy 2024 and TREC-Researchy 2024, to support baseline construction and debugging. TREC-RAGgy 2024 was filtered from TREC Deep Learning 2021–2023 topics, whereas TREC-Researchy 2024 was sampled from “Researchy Questions” to maximize diversity in an eight-dimensional intrinsic-attribute space (Pradeep et al., 2024).

3. Task structure, interfaces, and baseline system design

Later overview papers reconstruct the 2024 track as comprising three principal tasks—Retrieval (R), Augmented Generation (AG), and end-to-end Retrieval-Augmented Generation (RAG)—with the Relevance Judgment task added only in 2025 (Upadhyay et al., 10 Mar 2026).

Task	Input and constraint	Output and purpose
Retrieval (R)	Query over MS MARCO V2.1 segmented collection	Ranked list of top-$k$ segments
Augmented Generation (AG)	Organizer-provided retrieved segments	Answer with sentence-level citations
Retrieval-Augmented Generation (RAG)	Participant-supplied retrieval and generation pipeline	End-to-end cited answer

The standardized output format used a references array containing the top-20 segment IDs, an answer array of sentence objects with text and citations, and a response_length field (Pradeep et al., 2024). Citations were represented as zero-based indices into the references list, thereby separating answer structure from corpus identifiers while preserving traceability.

The Ragnarök baseline pipeline implemented a canonical 2024 architecture: BM25 first-stage retrieval over MS MARCO V2.1 segments, LLM-based reranking, and answer generation from the top reranked contexts (Pradeep et al., 2024). In the baseline configuration, BM25 with $k_1 = 0.9$ and $b = 0.4$ retrieved the top 100 segments; RankZephyr$_\rho$ reranked those candidates; and the top 20 segments were passed to either Cohere Command R+ or GPT-4o for answer generation. Command R+ produced span-level citations that Ragnarök mapped to sentence-level citations, whereas GPT-4o was prompted in a ChatQA-like format with IEEE-style bracketed references (Pradeep et al., 2024).

Subsequent reconstructions of 2024 system practice describe a more elaborate reference retrieval stack centered on Reciprocal Rank Fusion over SPLADE-v3 and Arctic-Embed-L, followed by RankLLM and RankQwen3-32B reranking (Upadhyay et al., 10 Mar 2026). Those descriptions are retrospective rather than part of the original Ragnarök release, but they indicate that the dominant 2024 design pattern had already crystallized as retriever $\rightarrow$ reranker $\rightarrow$ generator with explicit sentence-level attribution.

4. Evaluation architecture

The track’s most distinctive feature was its multi-layered evaluation methodology. Later documentation states that the 2025 framework—relevance assessment, nugget-based response evaluation, support evaluation, and agreement analysis—“builds on last year’s evaluation strategy at the TREC 2024 RAG Track” (Upadhyay et al., 10 Mar 2026).

Relevance assessment

Relevance assessment in 2024 used a 0–4 scale and standard IR metrics such as nDCG@20, nDCG@100, and Recall@100 (Upadhyay et al., 10 Mar 2026). In 2024, relevance was judged relative to a single query rather than to narrative subcomponents, and LLM-based assessors such as UMBRELA were compared with human qrels at the run level. Subsequent summaries report that LLM-based relevance assessments correlated well with human judgments at the run level, while per-topic agreement was weaker, a distinction that later motivated heavier use of automation with explicit caveats about fine-grained diagnosis (Upadhyay et al., 10 Mar 2026).

Nugget-based response evaluation

The response-evaluation layer adapted the TREC QA nugget methodology to long-form RAG answers (Pradeep et al., 21 Apr 2025). A nugget is an atomic fact that should appear in a good answer; nuggets are labeled as vital or okay. The track adopted strict vital recall as the primary response metric. If $V$ denotes the set of vital nuggets for a topic and $V_{\text{covered}} \subseteq V$ the subset fully supported by a system answer, then

$$\text{Strict Vital Recall} = \frac{|V_{\text{covered}}|}{|V|}.$$

“The Great Nugget Recall” introduced the AutoNuggetizer framework, which used GPT-4o to create nuggets automatically from relevant documents and to assign nuggets to system answers with labels support, partial_support, or not_support (Pradeep et al., 21 Apr 2025). In the context of the TREC 2024 RAG Track, the paper compared fully automatic nugget evaluation with semi-manual and fully manual alternatives, thereby turning the nugget layer itself into an object of methodological study.

Support evaluation

Support evaluation measured whether cited evidence actually supported the generated answer at the sentence–citation level (Thakur et al., 21 Apr 2025). For each answer sentence $a_i$ and cited passage $k_1 = 0.9$0, the judge assigns one of three labels:

• FS (Full Support): all of the sentence is supported;
• PS (Partial Support): only some parts are supported;
• NS (No Support): the cited passage supports no part of the sentence.

These labels map to weights $k_1 = 0.9$1, $k_1 = 0.9$2, and $k_1 = 0.9$3, respectively (Thakur et al., 21 Apr 2025). The track then computed weighted precision and weighted recall over judged sentence–passage pairs. Because sentences without citations contribute to the denominator of weighted recall but not to the numerator, the support layer simultaneously penalized unsupported citations and missing citations.

5. Empirical studies and methodological findings

The 2024 track rapidly became the basis for a series of large-scale empirical studies on RAG evaluation.

A support-assessment study examined 45 runs on 36 topics: 31 RAG runs and 14 AG runs, selected as the top two priority runs per group from 93 RAG runs submitted by 20 groups and 53 AG runs submitted by 11 groups (Thakur et al., 21 Apr 2025). It collected 10,907 human sentence–passage support labels across two conditions: manual-from-scratch and manual-with-post-editing of GPT-4o labels. Exact human–GPT-4o agreement was 56% in the manual-from-scratch condition and 72.1% in the post-editing condition, while run-level Kendall’s $k_1 = 0.9$4 for weighted precision and recall was above 0.79 in all cases (Thakur et al., 21 Apr 2025). The same study reported that, on sampled disagreement cases, an independent human judge correlated better with GPT-4o than with the original human judge, indicating that a substantial portion of “human versus LLM” disagreement was actually inter-human disagreement (Thakur et al., 21 Apr 2025).

The nugget-evaluation study reached a parallel conclusion at the run level (Pradeep et al., 21 Apr 2025). Fully automatic nugget evaluation, using automatic nugget creation and automatic assignment, showed run-level Kendall’s $k_1 = 0.9$5 of approximately 0.887 for strict vital recall and 0.901 for all-nugget strict recall against a semi-manual condition with post-edited nuggets and manual assignment; against a fully manual condition, run-level correlations were approximately 0.727 and 0.758, rising when topic-set differences were controlled (Pradeep et al., 21 Apr 2025). The same paper found that automating only nugget assignment while retaining human-created or post-edited nuggets improved per-topic agreement markedly, which implies that nugget assignment was more amenable to automation than nugget creation.

A third line of work investigated crowdsourced pairwise judgment on TREC RAG 2024 outputs (Gienapp et al., 22 Apr 2025). CrowdRAG-25 reused the 301 TREC RAG’24 topics, one of the top retrieval systems (webis-01), and the official top-20 passages per topic to build a corpus of 903 human-written and 903 GPT-4o-generated responses across bullet, essay, and news styles (Gienapp et al., 22 Apr 2025). For 65 topics, it added 47,320 pairwise human judgments and 10,556 pairwise LLM judgments over seven utility dimensions, including topical correctness, broad and deep coverage, and overall quality. That study concluded that human pairwise judgments were reliable and cost-effective, whereas zero-shot LLM-as-judge showed high internal consistency but poor agreement with human utility preferences (Gienapp et al., 22 Apr 2025).

6. Legacy, reinterpretation, and continuing influence

The 2025 RAG track explicitly presented itself as the “second edition” of a line inaugurated in 2024, preserving the MS MARCO V2.1 corpus, the Ragnarök framework, and the layered evaluation design while shifting toward “deep search” with long, multi-sentence narratives and sub-narrative decomposition (Upadhyay et al., 10 Mar 2026). In that sense, TREC RAG 2024 functioned as the prototype from which later TREC RAG tasks generalized.

Later analytical work on TREC-RAG also revealed that the 2024 benchmark exposed a nontrivial gap between retrieval and generation objectives. A 2026 study on 55 judged RAG24 queries reported a “utility gap” between retrieval relevance and generation fidelity: query variants that maximize ranking metrics such as nDCG often fail to produce the best generated answers, even though lightweight pre-retrieval predictors could still improve end-to-end nugget-based quality over the original query (Arabzadeh et al., 24 Apr 2026). Another 2026 study, using TREC RAG 2024 alongside NeuCLIR 2024, found strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses, especially when retrieval objectives aligned with generation objectives, but also noted that more complex iterative RAG pipelines can partially decouple generation quality from retrieval effectiveness (Samuel et al., 9 Mar 2026).

A persistent misconception about TREC RAG 2024 is that it reduced RAG evaluation to a single scalar. The track’s actual design does the opposite. Retrieval relevance, nugget coverage, and support evaluation diagnose different properties: a system may retrieve highly relevant segments yet omit vital nuggets; it may cover many nuggets while remaining weakly grounded; or it may cite passages precisely while failing to answer completely. This multi-axis design, together with the release of a reusable corpus, standard interfaces, and baseline systems, is why later overviews characterize TREC RAG 2024 as marking a transition from ad hoc demonstrations of RAG systems to systematic, community-wide evaluation of trustworthy, context-aware retrieval-augmented generation (Upadhyay et al., 10 Mar 2026).