---
title: TREC Deep Learning Track Overview
url: https://www.emergentmind.com/topics/trec-deep-learning-track
type: topic
---

# TREC Deep Learning Track Overview

TREC Deep Learning Track was a TREC benchmark series for ad hoc retrieval in the “large data regime,” that is, retrieval settings in which ranking models can exploit large human-labeled training sets rather than only small judged collections. Across five years, it used MS MARCO-derived resources to support passage and document ranking, combined blind TREC-style evaluation with reusable test collection construction, and studied both full retrieval and reranking under increasingly large and difficult corpora [2003.07820][2104.09399][2507.08890].

## 1. Origins and research objective

The track was introduced in 2019 to address a specific gap in information retrieval evaluation: much neural ranking research had relied on small datasets, proprietary datasets, or synthetic labels, which made it difficult to determine whether deep models truly outperformed strong traditional baselines in ad hoc retrieval. The organizers therefore framed the benchmark around three goals: provide large reusable datasets for ranking in a large training data regime, perform rigorous blind single-shot evaluation, and study both end-to-end retrieval and reranking-only scenarios [2003.07820].

From the outset, the track was defined not merely as a leaderboard but as a TREC-style evaluation framework. Its central methodological premise was that blind submission before label creation reduces overfitting and multiple-testing effects that can arise on public leaderboards. The companion reuse paper made this explicit: train on the large supervised training set, use development data for checkpointing and model selection, and treat the TREC test collections as held-out final evaluations rather than iterative tuning surfaces [2104.09399].

The phrase “large data regime” remained the track’s organizing concept throughout its lifespan. The 2020 overview restated the motivation as studying ad hoc ranking when large human-labeled training data are available, while the 2023 final overview described the benchmark’s scope as passage and document ranking, end-to-end retrieval and reranking, the utility of large pretrained and fine-tuned neural models, and later the validity of new test-set construction methods and prompt-based ranking [2102.07662][2507.08890].

## 2. Tasks, corpora, and training resources

The benchmark always centered on two tasks: document retrieval and passage retrieval. In the original MS MARCO v1-based setting used in 2019 and 2020, the document corpus contained 3,213,835 documents and the passage corpus contained 8,841,823 passages. The 2020 overview reported 367,013 document training queries with 384,597 document training qrels, and 502,939 passage training queries with 532,761 passage training qrels; it also reported development sets of 5,193 document queries and 6,980 passage queries [2102.07662]. The 2019 overview likewise emphasized that these were the first TREC ad hoc tasks with large human-labeled training sets derived from MS MARCO [2003.07820].

The benchmark structure distinguished full retrieval from reranking. In 2019 and 2020, document reranking used top-100 candidates generated with Indri, Krovetz stemming, and stopword removal, while passage reranking used top-1000 candidates generated by BM25 with no stemming [2003.07820][2104.09399]. This design supported controlled comparison of rerankers while preserving a full-retrieval setting for systems that implemented their own first-stage retrieval.

A major shift occurred in 2021 with adoption of MS MARCO v2. The organizers reported that the refresh produced 11.9 million documents and 138 million passages, corresponding to “a nearly four times increase in the document collection size and nearly \(16\) times increase in the size of the passage collection.” The v2 passage corpus was derived by a query-independent passage extraction procedure averaging 11.6 passages per document, and the refresh added passage-document mappings that had not been safely usable in v1 [2507.08191][2507.10865].

The subtask definitions evolved with that refresh. In 2021, 2022, and 2023, both document and passage reranking used top-100 candidates supplied by Pyserini, while full-ranking runs retrieved from the entire collection [2507.08191][2507.10865][2507.08890]. In 2022 and 2023, passage ranking was treated as the primary task and document ranking as secondary, with document labels inferred from passage judgments rather than judged independently [2507.10865][2507.08890].

## 3. Evaluation design and test collection construction

A defining property of the track was blind, single-shot evaluation with denser NIST judgments than the sparse MS MARCO training labels. In 2019, 200 test queries were released for both tasks, but NIST selected 43 evaluated queries for document retrieval and 43 for passage retrieval. In 2020, the process again began with 200 test queries, but only 45 document queries and 54 passage queries were judged because of budget constraints [2003.07820][2102.07662].

The original collection construction combined shallow pooling with dynamic judging. The 2019 overview described top-10 pools across submitted runs, augmentation with MS MARCO judgments, and subsequent judging with HiCAL, a classifier-driven dynamic judging procedure. The resulting collections were then analyzed for reusability, and the stability of system rankings under simulated alternative qrels was reported as very high, with Kendall’s \(\tau\) generally around \(0.96\) to \(0.99+\) in both tasks [2003.07820]. The reuse paper later generalized this argument and concluded that the collections were sufficiently stable for reuse, while also warning that repeated iteration on released test judgments could still induce overfitting [2104.09399].

Relevance judgments used four-point ordinal scales, but the semantics differed by task. For documents, the levels were 3 = perfectly relevant, 2 = highly relevant, 1 = relevant, and 0 = irrelevant; for binary metrics, levels 3, 2, and 1 counted as relevant. For passages, the levels were 3 = perfectly relevant and contains the exact answer, 2 = highly relevant and has an answer, 1 = related but does not answer the query, and 0 = irrelevant; for binary metrics, only 3 and 2 counted as relevant [2102.07662][2507.08191][2507.10865].

The v2 transition strained this methodology. In 2021, the much larger corpora produced many more relevant items per query, raising concerns about judgment completeness and the quality of qrels transferred from v1 into v2, especially for passages [2507.08191]. In response, the 2022 redesign concentrated human judging on passages only, propagated passage labels to documents by taking the maximum judged passage label in a document, introduced near-duplicate passage handling, and used Continuous Active Learning. The final 2022 evaluation set contained 76 topics, all below the organizers’ relevance-density threshold of 0.4, which they interpreted as supporting reusability [2507.10865].

The 2023 edition extended this redesign and added synthetic queries. The initial pool had 700 queries: 200 held-out human MS MARCO queries, 250 T5-generated synthetic queries, and 250 GPT-4-generated synthetic queries. After assessor filtering, 82 queries remained for evaluation: 51 human, 13 T5-generated, and 18 GPT-4-generated. Human relevance assessments were applied to all query types, not only the human queries [2507.08890].

## 4. Methodological trajectory and leaderboard-level findings

Across the five years, the track documented a clear progression in dominant retrieval paradigms, from early BERT-style rerankers and hybrid lexical-neural pipelines to prompt-based LLM systems.

| Year | Design or methodological turning point | Headline result |
|---|---|---|
| 2019 | First DL Track with large human-labeled training sets for ad hoc ranking | Best `nnlm` beat best `trad` by 29.4% on document NDCG@10 and 37.4% on passage NDCG@10 |
| 2020 | BERT-style pretraining consolidated; ORCAS added for document ranking | Best `nnlm` and `nn` document runs beat best `trad` by 23% and 11%; best passage gaps were 42% and 17% |
| 2021 | MS MARCO v2 refresh; single-stage retrieval explicitly studied | Best `nnlm` beat best `trad` by 15% on documents and 36% on passages |
| 2022 | Passage-centered judging on v2; some top systems were not dense | Best `nnlm` beat best `trad` by 76% on documents and 125% on passages |
| 2023 | Prompt-based systems added as a new class | Best passage run `naverloo-rgpt4 (b)` reached `ndcg@10 = 0.6994`; best document run `D_naverloo-frgpt4` reached `ndcg@10 = 0.6893` |

The 2019 results established the initial empirical claim that deep learning runs significantly outperformed traditional IR runs under blind TREC evaluation. The strongest systems were dominated by pretrained language models, especially BERT-based approaches, and this advantage was larger for passage ranking than for document ranking [2003.07820]. The 2020 overview sharpened the point: “rankers with BERT-style pretraining outperform other rankers in the large data regime,” with especially large gains in passage retrieval [2102.07662].

The relation between full retrieval and reranking changed more slowly. In 2019, the best full-retrieval systems had only a small NDCG@10 advantage over the best rerankers: 0.9% for documents and 3.6% for passages [2003.07820]. In 2020, the best full-ranking document run exceeded the best reranker by about 5%, while in passage retrieval the best full-ranking run was about 0.3% lower in NDCG@10 than the best reranker [2102.07662]. In 2021, full retrieval again led, by 4% for documents and 6% for passages, but the organizers still judged single-stage systems to be competitive rather than dominant [2507.08191]. The largest gap appeared in 2022, when the best full-ranking run exceeded the best reranking run by 36% in passage ranking and 125% in document ranking [2507.10865].

The 2023 final overview marked a different transition. For the first four years, the strongest paradigm had been the organizers’ `nnlm` category, meaning systems using pretrained neural models somewhere in the pipeline. In 2023, prompt-based systems outperformed `nnlm` systems. The organizers were careful to note that these prompt-based runs almost certainly did not apply expensive LLM inference to all 138 million passages; rather, they likely used conventional retrieval for candidate generation and applied prompting in later stages. Still, from the track’s perspective, prompt-based ranking created a new top performance tier [2507.08890].

## 5. Representative system families and technical patterns

The track’s participant papers illustrate the methodological breadth hidden behind the yearly `trad`, `nn`, `nnlm`, and `prompt` labels.

Brown University’s TREC 2019 passage run is a canonical early example of neural reranking augmented by learned reformulation. Brown explicitly entered the passage reranking task rather than full first-stage retrieval. Its system used a two-phase pipeline: a Transformer query2query expansion model trained in OpenNMT on qrels-derived related-query pairs, followed by a BERT Large cross-encoder that scored each of the provided 1000 candidate passages. At inference time, the system appended the top 3 generated query variants to the original query and then sorted the 1000 candidates by predicted relevance. In the official 2019 results it ranked 3rd overall in passage retrieval and 2nd among reranking submissions [2009.04016].

Microsoft’s “Duet at TREC 2019 Deep Learning Track” provides a complementary example from the `nn` family. For document retrieval, it introduced Duet with Multiple Fields, which separately matched the query against URL, title, and body, used field-specific unshared parameters, summarized each query-field interaction as a vector, and aggregated those vectors into a final document score. A second run used DuetMF as one feature in a neural learning-to-rank model together with SDM, PRF, BM25, DESM, query length, and domain quality, while the passage run used an ensemble of eight Duet models [1912.04471].

The 2020 Conformer-Kernel study shows how the track also functioned as a test bed for scalable neural first-stage retrieval. That paper benchmarked Conformer-Kernel models with three additions: explicit lexical matching in the “Duet principle,” query term independence for scalable full retrieval, and ORCAS click data as an additional document field. The strongest run, `ndrm3-orc-full`, was the best-performing `nn` run on both NDCG@10 and NCG@100 in the TREC 2020 document task, and the paper argued that QTI made it possible to precompute term-document scores and use an inverted index for fast retrieval [2011.07368].

By 2021, strong systems had become visibly multi-stage and hybrid. PASH combined sparse retrieval based on docTTTTTquery-enhanced BM25 with ColBERT dense retrieval in the recall stage, then applied point-wise and pair-wise neural ranking using BERT-Large, ALBERT-XXLarge, ELECTRA-Large, XLNet-Large, and a newly introduced generative T5 reranker. The paper described the addition of pair-wise loss and T5 as the main extensions over its 2020 system, and the 2021 overview identified PASH runs such as `pash_doc_f1` and `pash_f1` as leaderboard leaders [2205.11245][2507.08191].

Alibaba’s 2022 submission exemplified the later full-ranking architecture pattern. Its retrieval stage combined BM25, Doc2query, and SPLADE on the sparse side with ROM-based dense retrieval; its ranking stage used a diverse full-interaction reranker ensemble based on BERT, RoBERTa, ERNIE, ELECTRA, and DeBERTa; and its final HLATR stage fused retrieval and reranking signals. The system achieved 1st place in passage ranking and 4th place in document ranking on the official TREC 2022 evaluation [2308.12039].

Taken together, these systems suggest a stable architectural pattern. Successful runs frequently combined strong lexical retrieval, neural semantic matching, and staged reranking rather than choosing a single paradigm. A plausible implication is that the track rewarded systems that treated recall, lexical coverage, and fine-grained relevance estimation as distinct bottlenecks rather than as interchangeable components.

## 6. Reusability, interpretive issues, and legacy

A persistent theme of the track was that it should be reused, but reused carefully. The reuse paper recommended a disciplined protocol: fit on train, make decisions on dev, report each TREC year separately, avoid checkpoint or architecture selection on the test sets, and run multiple random seeds to study approaches rather than single lucky trials. Its case study showed that even meaningless checkpoint differences could appear statistically significant if checkpoints were cherry-picked on the test collections, reporting an example with \(p = 0.00837\) [2104.09399].

Another recurring issue was the relation between official TREC evaluation and the public MS MARCO leaderboard. The track never treated them as equivalent. In 2020, agreement between sparse MS MARCO evaluation and official NIST evaluation was only moderate overall: for all document runs, Kendall’s \(\tau\) between NDCG@10 and RR(MS) was 0.46, while for passage runs it was 0.69 [2102.07662]. In 2021, this agreement declined further, to 0.43 for document ranking and 0.51 for passage ranking, and the report identified additional complications such as “oldness” bias in the v2 document corpus and lower judgment completeness in the much larger v2 collections [2507.08191]. The common misconception that TREC DL was simply a denser mirror of MS MARCO leaderboard evaluation is therefore inaccurate.

The later years turned evaluation methodology itself into a first-class research problem. The 2022 redesign concentrated judging on passages, used deduplication and active learning, and produced a 76-topic passage-centered collection that the organizers considered more complete and reusable than the 2021 set [2507.10865]. The 2023 final track then tested whether synthetic queries could help construct reliable test collections. Its headline result was that evaluation on synthetic queries gave similar results to human queries, with system-order agreement of \(\tau = 0.8487\), but the paper also stressed that human effort was needed to select usable synthetic queries and that more analysis was still necessary [2507.08890].

The track’s scientific legacy is correspondingly twofold. Methodologically, it left behind reusable collections and a detailed body of evidence on blind evaluation, pooling, active learning, deduplication, dense-versus-sparse disagreement, and synthetic-query validation. Empirically, it documented a progression in dominant ranking paradigms: pretrained neural language models displaced earlier neural and traditional baselines, multistage hybrid retrieval remained central even as single-stage systems improved, and by the final year prompt-based LLM ranking had surpassed the previously dominant `nnlm` class [2003.07820][2507.08890].

Because 2023 was the fifth and final year of the track, the series closed at a transition point. The final overview explicitly suggested that future prompt-based ranking work could continue in other tracks. This suggests a historically coherent interpretation: TREC Deep Learning Track began as a benchmark for determining whether large supervised data would make deep ranking methods decisively competitive, and ended after establishing that large-scale pretraining, hybrid retrieval-reranking pipelines, and then prompt-based LLM methods had become the central paradigms for ad hoc retrieval evaluation under rigorous TREC methodology [2102.07662][2507.08890].

Source: https://www.emergentmind.com/topics/trec-deep-learning-track