TREC Deep Learning Track Overview
- TREC Deep Learning Track is a benchmark series that evaluates ad hoc retrieval using large, human-labeled datasets and rigorous blind testing.
- It advanced evaluation methods by integrating controlled pooling, dynamic judging, and synthetic query use to mitigate overfitting.
- Over five years, the track demonstrated that large-scale pretraining, hybrid pipelines, and prompt-based ranking consistently boost retrieval performance.
TREC Deep Learning Track was a TREC benchmark series for ad hoc retrieval in the “large data regime,” that is, retrieval settings in which ranking models can exploit large human-labeled training sets rather than only small judged collections. Across five years, it used MS MARCO-derived resources to support passage and document ranking, combined blind TREC-style evaluation with reusable test collection construction, and studied both full retrieval and reranking under increasingly large and difficult corpora (Craswell et al., 2020, Craswell et al., 2021, Craswell et al., 10 Jul 2025).
1. Origins and research objective
The track was introduced in 2019 to address a specific gap in information retrieval evaluation: much neural ranking research had relied on small datasets, proprietary datasets, or synthetic labels, which made it difficult to determine whether deep models truly outperformed strong traditional baselines in ad hoc retrieval. The organizers therefore framed the benchmark around three goals: provide large reusable datasets for ranking in a large training data regime, perform rigorous blind single-shot evaluation, and study both end-to-end retrieval and reranking-only scenarios (Craswell et al., 2020).
From the outset, the track was defined not merely as a leaderboard but as a TREC-style evaluation framework. Its central methodological premise was that blind submission before label creation reduces overfitting and multiple-testing effects that can arise on public leaderboards. The companion reuse paper made this explicit: train on the large supervised training set, use development data for checkpointing and model selection, and treat the TREC test collections as held-out final evaluations rather than iterative tuning surfaces (Craswell et al., 2021).
The phrase “large data regime” remained the track’s organizing concept throughout its lifespan. The 2020 overview restated the motivation as studying ad hoc ranking when large human-labeled training data are available, while the 2023 final overview described the benchmark’s scope as passage and document ranking, end-to-end retrieval and reranking, the utility of large pretrained and fine-tuned neural models, and later the validity of new test-set construction methods and prompt-based ranking (Craswell et al., 2021, Craswell et al., 10 Jul 2025).
2. Tasks, corpora, and training resources
The benchmark always centered on two tasks: document retrieval and passage retrieval. In the original MS MARCO v1-based setting used in 2019 and 2020, the document corpus contained 3,213,835 documents and the passage corpus contained 8,841,823 passages. The 2020 overview reported 367,013 document training queries with 384,597 document training qrels, and 502,939 passage training queries with 532,761 passage training qrels; it also reported development sets of 5,193 document queries and 6,980 passage queries (Craswell et al., 2021). The 2019 overview likewise emphasized that these were the first TREC ad hoc tasks with large human-labeled training sets derived from MS MARCO (Craswell et al., 2020).
The benchmark structure distinguished full retrieval from reranking. In 2019 and 2020, document reranking used top-100 candidates generated with Indri, Krovetz stemming, and stopword removal, while passage reranking used top-1000 candidates generated by BM25 with no stemming (Craswell et al., 2020, Craswell et al., 2021). This design supported controlled comparison of rerankers while preserving a full-retrieval setting for systems that implemented their own first-stage retrieval.
A major shift occurred in 2021 with adoption of MS MARCO v2. The organizers reported that the refresh produced 11.9 million documents and 138 million passages, corresponding to “a nearly four times increase in the document collection size and nearly $16$ times increase in the size of the passage collection.” The v2 passage corpus was derived by a query-independent passage extraction procedure averaging 11.6 passages per document, and the refresh added passage-document mappings that had not been safely usable in v1 (Craswell et al., 10 Jul 2025, Craswell et al., 10 Jul 2025).
The subtask definitions evolved with that refresh. In 2021, 2022, and 2023, both document and passage reranking used top-100 candidates supplied by Pyserini, while full-ranking runs retrieved from the entire collection (Craswell et al., 10 Jul 2025, Craswell et al., 10 Jul 2025, Craswell et al., 10 Jul 2025). In 2022 and 2023, passage ranking was treated as the primary task and document ranking as secondary, with document labels inferred from passage judgments rather than judged independently (Craswell et al., 10 Jul 2025, Craswell et al., 10 Jul 2025).
3. Evaluation design and test collection construction
A defining property of the track was blind, single-shot evaluation with denser NIST judgments than the sparse MS MARCO training labels. In 2019, 200 test queries were released for both tasks, but NIST selected 43 evaluated queries for document retrieval and 43 for passage retrieval. In 2020, the process again began with 200 test queries, but only 45 document queries and 54 passage queries were judged because of budget constraints (Craswell et al., 2020, Craswell et al., 2021).
The original collection construction combined shallow pooling with dynamic judging. The 2019 overview described top-10 pools across submitted runs, augmentation with MS MARCO judgments, and subsequent judging with HiCAL, a classifier-driven dynamic judging procedure. The resulting collections were then analyzed for reusability, and the stability of system rankings under simulated alternative qrels was reported as very high, with Kendall’s generally around $0.96$ to $0.99+$ in both tasks (Craswell et al., 2020). The reuse paper later generalized this argument and concluded that the collections were sufficiently stable for reuse, while also warning that repeated iteration on released test judgments could still induce overfitting (Craswell et al., 2021).
Relevance judgments used four-point ordinal scales, but the semantics differed by task. For documents, the levels were 3 = perfectly relevant, 2 = highly relevant, 1 = relevant, and 0 = irrelevant; for binary metrics, levels 3, 2, and 1 counted as relevant. For passages, the levels were 3 = perfectly relevant and contains the exact answer, 2 = highly relevant and has an answer, 1 = related but does not answer the query, and 0 = irrelevant; for binary metrics, only 3 and 2 counted as relevant (Craswell et al., 2021, Craswell et al., 10 Jul 2025, Craswell et al., 10 Jul 2025).
The v2 transition strained this methodology. In 2021, the much larger corpora produced many more relevant items per query, raising concerns about judgment completeness and the quality of qrels transferred from v1 into v2, especially for passages (Craswell et al., 10 Jul 2025). In response, the 2022 redesign concentrated human judging on passages only, propagated passage labels to documents by taking the maximum judged passage label in a document, introduced near-duplicate passage handling, and used Continuous Active Learning. The final 2022 evaluation set contained 76 topics, all below the organizers’ relevance-density threshold of 0.4, which they interpreted as supporting reusability (Craswell et al., 10 Jul 2025).
The 2023 edition extended this redesign and added synthetic queries. The initial pool had 700 queries: 200 held-out human MS MARCO queries, 250 T5-generated synthetic queries, and 250 GPT-4-generated synthetic queries. After assessor filtering, 82 queries remained for evaluation: 51 human, 13 T5-generated, and 18 GPT-4-generated. Human relevance assessments were applied to all query types, not only the human queries (Craswell et al., 10 Jul 2025).
4. Methodological trajectory and leaderboard-level findings
Across the five years, the track documented a clear progression in dominant retrieval paradigms, from early BERT-style rerankers and hybrid lexical-neural pipelines to prompt-based LLM systems.
| Year | Design or methodological turning point | Headline result |
|---|---|---|
| 2019 | First DL Track with large human-labeled training sets for ad hoc ranking | Best nnlm beat best trad by 29.4% on document NDCG@10 and 37.4% on passage NDCG@10 |
| 2020 | BERT-style pretraining consolidated; ORCAS added for document ranking | Best nnlm and nn document runs beat best trad by 23% and 11%; best passage gaps were 42% and 17% |
| 2021 | MS MARCO v2 refresh; single-stage retrieval explicitly studied | Best nnlm beat best trad by 15% on documents and 36% on passages |
| 2022 | Passage-centered judging on v2; some top systems were not dense | Best nnlm beat best trad by 76% on documents and 125% on passages |
| 2023 | Prompt-based systems added as a new class | Best passage run naverloo-rgpt4 (b) reached ndcg@10 = 0.6994; best document run D_naverloo-frgpt4 reached ndcg@10 = 0.6893 |
The 2019 results established the initial empirical claim that deep learning runs significantly outperformed traditional IR runs under blind TREC evaluation. The strongest systems were dominated by pretrained LLMs, especially BERT-based approaches, and this advantage was larger for passage ranking than for document ranking (Craswell et al., 2020). The 2020 overview sharpened the point: “rankers with BERT-style pretraining outperform other rankers in the large data regime,” with especially large gains in passage retrieval (Craswell et al., 2021).
The relation between full retrieval and reranking changed more slowly. In 2019, the best full-retrieval systems had only a small NDCG@10 advantage over the best rerankers: 0.9% for documents and 3.6% for passages (Craswell et al., 2020). In 2020, the best full-ranking document run exceeded the best reranker by about 5%, while in passage retrieval the best full-ranking run was about 0.3% lower in NDCG@10 than the best reranker (Craswell et al., 2021). In 2021, full retrieval again led, by 4% for documents and 6% for passages, but the organizers still judged single-stage systems to be competitive rather than dominant (Craswell et al., 10 Jul 2025). The largest gap appeared in 2022, when the best full-ranking run exceeded the best reranking run by 36% in passage ranking and 125% in document ranking (Craswell et al., 10 Jul 2025).
The 2023 final overview marked a different transition. For the first four years, the strongest paradigm had been the organizers’ nnlm category, meaning systems using pretrained neural models somewhere in the pipeline. In 2023, prompt-based systems outperformed nnlm systems. The organizers were careful to note that these prompt-based runs almost certainly did not apply expensive LLM inference to all 138 million passages; rather, they likely used conventional retrieval for candidate generation and applied prompting in later stages. Still, from the track’s perspective, prompt-based ranking created a new top performance tier (Craswell et al., 10 Jul 2025).
5. Representative system families and technical patterns
The track’s participant papers illustrate the methodological breadth hidden behind the yearly trad, nn, nnlm, and prompt labels.
Brown University’s TREC 2019 passage run is a canonical early example of neural reranking augmented by learned reformulation. Brown explicitly entered the passage reranking task rather than full first-stage retrieval. Its system used a two-phase pipeline: a Transformer query2query expansion model trained in OpenNMT on qrels-derived related-query pairs, followed by a BERT Large cross-encoder that scored each of the provided 1000 candidate passages. At inference time, the system appended the top 3 generated query variants to the original query and then sorted the 1000 candidates by predicted relevance. In the official 2019 results it ranked 3rd overall in passage retrieval and 2nd among reranking submissions (Zerveas et al., 2020).
Microsoft’s “Duet at TREC 2019 Deep Learning Track” provides a complementary example from the nn family. For document retrieval, it introduced Duet with Multiple Fields, which separately matched the query against URL, title, and body, used field-specific unshared parameters, summarized each query-field interaction as a vector, and aggregated those vectors into a final document score. A second run used DuetMF as one feature in a neural learning-to-rank model together with SDM, PRF, BM25, DESM, query length, and domain quality, while the passage run used an ensemble of eight Duet models (Mitra et al., 2019).
The 2020 Conformer-Kernel study shows how the track also functioned as a test bed for scalable neural first-stage retrieval. That paper benchmarked Conformer-Kernel models with three additions: explicit lexical matching in the “Duet principle,” query term independence for scalable full retrieval, and ORCAS click data as an additional document field. The strongest run, ndrm3-orc-full, was the best-performing nn run on both NDCG@10 and NCG@100 in the TREC 2020 document task, and the paper argued that QTI made it possible to precompute term-document scores and use an inverted index for fast retrieval (Mitra et al., 2020).
By 2021, strong systems had become visibly multi-stage and hybrid. PASH combined sparse retrieval based on docTTTTTquery-enhanced BM25 with ColBERT dense retrieval in the recall stage, then applied point-wise and pair-wise neural ranking using BERT-Large, ALBERT-XXLarge, ELECTRA-Large, XLNet-Large, and a newly introduced generative T5 reranker. The paper described the addition of pair-wise loss and T5 as the main extensions over its 2020 system, and the 2021 overview identified PASH runs such as pash_doc_f1 and pash_f1 as leaderboard leaders (Qiao et al., 2022, Craswell et al., 10 Jul 2025).
Alibaba’s 2022 submission exemplified the later full-ranking architecture pattern. Its retrieval stage combined BM25, Doc2query, and SPLADE on the sparse side with ROM-based dense retrieval; its ranking stage used a diverse full-interaction reranker ensemble based on BERT, RoBERTa, ERNIE, ELECTRA, and DeBERTa; and its final HLATR stage fused retrieval and reranking signals. The system achieved 1st place in passage ranking and 4th place in document ranking on the official TREC 2022 evaluation (Xu et al., 2023).
Taken together, these systems suggest a stable architectural pattern. Successful runs frequently combined strong lexical retrieval, neural semantic matching, and staged reranking rather than choosing a single paradigm. A plausible implication is that the track rewarded systems that treated recall, lexical coverage, and fine-grained relevance estimation as distinct bottlenecks rather than as interchangeable components.
6. Reusability, interpretive issues, and legacy
A persistent theme of the track was that it should be reused, but reused carefully. The reuse paper recommended a disciplined protocol: fit on train, make decisions on dev, report each TREC year separately, avoid checkpoint or architecture selection on the test sets, and run multiple random seeds to study approaches rather than single lucky trials. Its case study showed that even meaningless checkpoint differences could appear statistically significant if checkpoints were cherry-picked on the test collections, reporting an example with (Craswell et al., 2021).
Another recurring issue was the relation between official TREC evaluation and the public MS MARCO leaderboard. The track never treated them as equivalent. In 2020, agreement between sparse MS MARCO evaluation and official NIST evaluation was only moderate overall: for all document runs, Kendall’s between NDCG@10 and RR(MS) was 0.46, while for passage runs it was 0.69 (Craswell et al., 2021). In 2021, this agreement declined further, to 0.43 for document ranking and 0.51 for passage ranking, and the report identified additional complications such as “oldness” bias in the v2 document corpus and lower judgment completeness in the much larger v2 collections (Craswell et al., 10 Jul 2025). The common misconception that TREC DL was simply a denser mirror of MS MARCO leaderboard evaluation is therefore inaccurate.
The later years turned evaluation methodology itself into a first-class research problem. The 2022 redesign concentrated judging on passages, used deduplication and active learning, and produced a 76-topic passage-centered collection that the organizers considered more complete and reusable than the 2021 set (Craswell et al., 10 Jul 2025). The 2023 final track then tested whether synthetic queries could help construct reliable test collections. Its headline result was that evaluation on synthetic queries gave similar results to human queries, with system-order agreement of , but the paper also stressed that human effort was needed to select usable synthetic queries and that more analysis was still necessary (Craswell et al., 10 Jul 2025).
The track’s scientific legacy is correspondingly twofold. Methodologically, it left behind reusable collections and a detailed body of evidence on blind evaluation, pooling, active learning, deduplication, dense-versus-sparse disagreement, and synthetic-query validation. Empirically, it documented a progression in dominant ranking paradigms: pretrained neural LLMs displaced earlier neural and traditional baselines, multistage hybrid retrieval remained central even as single-stage systems improved, and by the final year prompt-based LLM ranking had surpassed the previously dominant nnlm class (Craswell et al., 2020, Craswell et al., 10 Jul 2025).
Because 2023 was the fifth and final year of the track, the series closed at a transition point. The final overview explicitly suggested that future prompt-based ranking work could continue in other tracks. This suggests a historically coherent interpretation: TREC Deep Learning Track began as a benchmark for determining whether large supervised data would make deep ranking methods decisively competitive, and ended after establishing that large-scale pretraining, hybrid retrieval-reranking pipelines, and then prompt-based LLM methods had become the central paradigms for ad hoc retrieval evaluation under rigorous TREC methodology (Craswell et al., 2021, Craswell et al., 10 Jul 2025).
