TREC NeuCLIR Track Overview

• TREC NeuCLIR Track is an annual benchmark designed to assess neural cross-language information retrieval using multilingual news and scientific collections.
• It employs varied methodologies—including document/query translation, dense and sparse retrieval, and hybrid reranking—to drive advances in multilingual IR.
• The track standardizes evaluation with rigorous metrics and evolving tasks, promoting reproducibility and domain adaptation research in cross-language retrieval.

The TREC NeuCLIR Track (Neural Cross-Language Information Retrieval) is an annual shared task and benchmark evaluation designed to advance research in cross-language and multilingual information retrieval with a special focus on neural methods. Established in 2022 within the Text REtrieval Conference series, NeuCLIR provides large, reusable test collections and rigorously judged benchmarks in which systems must retrieve relevant non-English documents—primarily news or scientific texts in Chinese, Persian, and Russian—given queries or report requests formulated in English. NeuCLIR’s infrastructure, data resources, and evaluation protocols have catalyzed significant progress in cross-lingual retrieval, neural ranking, and retrieval-augmented language generation, culminating in a succession of increasingly complex tasks, including report generation and technical-document retrieval, over its three-year span (Lawrie et al., 2023, Lawrie et al., 11 Apr 2024, Lawrie et al., 17 Sep 2025).

1. Objectives and Track Evolution

NeuCLIR's principal objective is to enable robust, reproducible, and scalable evaluation of neural pipelines for cross-language IR. The track addresses longstanding gaps in CLIR—where prior TREC evaluations emphasized traditional lexical translation plus BM25—and leverages recent advances in large multilingual LLMs, dense and sparse neural retrieval, and end-to-end neural rerankers. NeuCLIR encompasses multiple task types:

• Ad Hoc CLIR: English queries, return a ranked list from a non-English target corpus.
• MLIR (Multilingual IR): Retrieve from the union of Chinese, Persian, and Russian news, requiring cross-language score normalization and rank fusion.
• Technical CLIR: Retrieval over technical/scientific Chinese abstracts, emphasizing domain adaptation and vocabulary coverage.
• Report Generation (2024): Given an English “report request,” synthesize factually supported English summaries from non-English source texts, with citation/evidence requirements (Lawrie et al., 17 Sep 2025, Yang et al., 30 Sep 2025).

The track’s longitudinal design—with persistent corpora and evolving tasks—facilitated method comparison, system reuse, and longitudinal paper of technique effectiveness across years (Lawrie et al., 11 Apr 2024, Lawrie et al., 17 Sep 2025).

2. Test Collections, Topics, and Relevance Judgments

NeuCLIR’s document collections are derived from Common Crawl newswire (Chinese, Persian, and Russian, ~2–5 million articles per language) and, uniquely, a Chinese scientific abstracts corpus (CSL, ~400k abstracts). Test sets are augmented with massive translation resources: human and machine-translated queries, document translations, and multilingual MS MARCO (“NeuMARCO”) (Lawrie et al., 11 Apr 2024, Lawrie et al., 17 Sep 2025, Yang et al., 30 Sep 2025).

Topics (queries) are drafted in English, refined, and—where needed for monolingual or “upper-bound” runs—translated into target languages. Report Generation tasks employ “report requests” (background/problem statement) jointly with a “nugget” annotation protocol for fine-grained evidence assessment (Lawrie et al., 17 Sep 2025).

Relevance judgments are performed using deep pooling from submitted runs and annotated with a graded scale (typically 0 = not relevant, 1 = somewhat valuable, 3 = very valuable). Nugget-based evaluations in report generation demand per-fact, per-citation annotation (Lawrie et al., 17 Sep 2025). Reusability of pools is validated by leave-one-run/team-out stability analyses, yielding Kendall’s τ ≥ 0.97 for nDCG-based system ordering (Lawrie et al., 17 Sep 2025, Lawrie et al., 11 Apr 2024).

3. Conceptual Frameworks and Retrieval Pipelines

NeuCLIR’s baseline and participant systems are grounded in the multi-stage IR architecture familiar from monolingual neural IR: a high-recall first-stage retriever (sparse, dense, or hybrid), optionally pseudo-relevance feedback, followed by a cross-encoder or LLM-based reranker, and, in some cases, post-hoc run fusion (Lin et al., 2023, Yang et al., 11 Apr 2024, Yang et al., 30 Sep 2025).

First-Stage Retrieval Paradigms

• Document Translation (DT): Translate the non-English collection into English, then leverage mature English IR models. Accelerates method deployment but incurs a static translation cost and may miss language-specific nuances.
• Query Translation (QT): Translate the English query to the target language. Fast and efficient, but short-query MT quality is variable.
• Language-Independent (Dense/Multilingual Retrieval): Map both English queries and non-English documents into a shared embedding space (e.g., via mBERT, XLM-R, ColBERT-X, PLAID), retrieving via maximum inner product or late-interaction (Lin et al., 2023, Yang et al., 11 Apr 2024, Yang et al., 30 Sep 2025).

Hybrid retrieval—combining dense (bi-encoder) scores with sparse (e.g., SPLADE) or BM25 scores—consistently exhibits state-of-the-art effectiveness, outperforming pure approaches (Lawrie et al., 2023, Lawrie et al., 11 Apr 2024, Lawrie et al., 17 Sep 2025).

Reranking and Fusion

Rerankers are typically cross-encoder architectures (e.g., mT5, GPT-4), fine-tuned on MS MARCO (and translated versions) or distilled as teacher-student pipelines (e.g., Translate Distill, Multilingual Translate Distill) (Yang et al., 11 Apr 2024, Yang et al., 30 Sep 2025). Reciprocal rank fusion, CombSUM, and cross-system ensembles are deployed, particularly in MLIR, to balance language exposure and maximize recall/nDCG (Yang et al., 30 Sep 2025, Lawrie et al., 17 Sep 2025).

Report Generation

2024 introduced a pilot Report Generation task. Systems must both retrieve and select relevant non-English texts and generate coherent English reports, supporting each statement with explicit document citations. Extractive pipelines (LLM-based fact clustering) yield higher factual coverage (“nugget recall”), while abstractive pipelines optimize fluency but currently lag on evidence coverage (Yang et al., 30 Sep 2025, Lawrie et al., 17 Sep 2025).

4. Methods: Models, Training Regimens, and Implementation

Key neural CLIR models include:

• SPLADE (Learned Sparse): Pre-trained from scratch on each non-English corpus using MLM+FLOPS objectives and fine-tuned on translated MS MARCO; implemented for efficient inverted-indexing within Lucene (Lin et al., 2023, Lassance et al., 2023).
• ColBERT / PLAID (Late Interaction Dense): Variants include monolingual, Translate-Train (MS MARCO English queries + translated documents), Multilingual TT (mixed-language batches), and Translate Distill (student trained to match mT5 teacher scores) (Yang et al., 11 Apr 2024, Yang et al., 30 Sep 2025).
• Cross-Encoder Rerankers: mT5 XXL and GPT-4 (and variants) are deployed in pointwise binary classification, scoring top-k retrieved candidates, typically with softmax over “true”/“false” for relevance (Jeronymo et al., 2023, Yang et al., 30 Sep 2025).
• Fusion and Score Normalization: Reciprocal Rank Fusion, CombSUM, and rank-based heuristics address cross-system and cross-language scoring, particularly vital in MLIR and hybrid submissions (Yang et al., 30 Sep 2025, Lawrie et al., 17 Sep 2025).

Training recipes prioritize transfer learning from large-scale English data (MS MARCO), large-batch cross-lingual distillation, and domain adaptation via synthetic query generation (LLM-based “Generate-Distill”) (Yang et al., 30 Sep 2025). All major baselines and many participant solutions are released in open-source toolkits (Anserini, Pyserini, PLAID) (Lin et al., 2023, Yang et al., 11 Apr 2024).

5. Evaluation Metrics and Protocols

NeuCLIR employs multiple IR metrics, reporting them by topic and averaged over evaluation sets:

• nDCG@k:

$$\mathrm{nDCG}@k = \frac{1}{|Q|} \sum_{q \in Q} \frac{\sum_{i=1}^k \frac{2^{\mathrm{rel}_i} - 1}{\log_2(i+1)}}{\mathrm{IDCG}_k(q)}$$

• Mean Average Precision (MAP):

$$\mathrm{MAP} = \frac{1}{|Q|}\sum_{q \in Q}\mathrm{AP}(q)$$

• Recall@k: Fraction of relevant documents retrieved in the top k.
• Rank-Biased Precision (RBP):

$$\mathrm{RBP}(p) = (1 - p) \sum_{i=1}^\infty p^{i-1} \mathrm{rel}_i$$

• α-nDCG: Used in MLIR to reward both retrieval performance and diversity.
• Report Generation (ARGUE Framework): Combines citation precision, nugget recall, nugget support, and sentence support into a single composite score assessing both accuracy and factual coverage of generated summaries (Lawrie et al., 17 Sep 2025, Yang et al., 30 Sep 2025).

Consistent metric definitions and annotation protocol across years facilitate longitudinal benchmarking.

6. Experimental Findings and Best Practices

Retrieval Performance

• Strongest news CLIR runs in 2024 reach nDCG@20 ≈ 0.698 (Persian), 0.664 (Chinese), and 0.593 (Russian), driven by GPT-4 and mT5 reranking over hybrid (dense + sparse) or fused retrieval (Lawrie et al., 17 Sep 2025).
• Monolingual baselines (BM25 or neural retrieval on human-translated queries) are now often surpassed by neural CLIR runs, reversing the 2022 trend (Lawrie et al., 11 Apr 2024, Lawrie et al., 17 Sep 2025).
• Technical document CLIR (CSL abstracts) remains challenging: best nDCG@20 ≈ 0.496 (Lawrie et al., 17 Sep 2025), highlighting domain adaptation as an open research frontier.

Report Generation

• Best systems achieve ARGUE scores: Chinese 0.726, Persian 0.872, Russian 0.808, with high citation precision (0.85–0.90) but only 30–42% nugget recall, revealing persistent factual coverage bottlenecks (Lawrie et al., 17 Sep 2025, Yang et al., 30 Sep 2025).
• Extractive report generation pipelines (fact extraction, clustering, citation) show higher coverage, while abstractive (summarization + meta-combiner) pipelines favor fluency and readability (Yang et al., 30 Sep 2025).

Model Design and Training

• Translate Distill and Multilingual Translate Distill significantly enhance dense retrieval, while Generate-Distill approaches improve adaptation to domain and language-resource sparsity (Yang et al., 30 Sep 2025).
• Larger passage windows (e.g., 450 tokens, no overlap) yield small but consistent gains in nDCG (Yang et al., 30 Sep 2025).
• Score fusion and cross-language normalization remain essential in MLIR and highly imbalanced topical distributions.

Pooling and Evaluation

• Pool diversity is sufficient for robust judgment reuse; system ranking stability is very high under run/team ablation (Lawrie et al., 11 Apr 2024, Lawrie et al., 17 Sep 2025).

7. Impact and Outlook

NeuCLIR has established neural CLIR as a state-of-the-art baseline, nearly closing the gap with monolingual IR in high-resource language pairs and exposing critical challenges in technical vocabulary, citation-based summarization, and multilingual fairness.

Key research directions emerging from NeuCLIR include:

• Developing robust, translation-free multilingual embedding models to reduce MT dependence (Lawrie et al., 17 Sep 2025).
• Enhancing cross-document and cross-language context modeling, particularly for generation/summarization (Yang et al., 30 Sep 2025).
• Advancing domain-adapted retrieval and LLM-based reranking for specialized corpora.
• Improving evaluation metrics, especially for factual consistency and evidence coverage in report generation.
• Transition efforts, such as the RAGTIME track (TREC 2025), will shift the focus toward four-way multilingual retrieval, de novo report requests, and broader automation in evaluation (Lawrie et al., 17 Sep 2025).

By releasing large-scale, reusable test collections, standardized annotations, and open-source pipelines, NeuCLIR has created a sustained foundation for research in neural cross-language IR, report generation, and retrieval-augmented LLMs, with broad influence across the academic IR and NLP community (Lawrie et al., 2023, Lawrie et al., 11 Apr 2024, Lawrie et al., 17 Sep 2025, Yang et al., 30 Sep 2025).