TREC NeuCLIR: Neural Cross-Language IR
- The paper introduces NeuCLIR as a neural evolution of classical CLIR, leveraging multilingual embeddings and neural rerankers to bridge lexical gaps across languages.
- It evolved from a basic ad hoc retrieval track in 2022 to a multi-task program by 2024, encompassing news, technical documents, and report generation with refined evaluation measures.
- NeuCLIR’s derivative benchmarks like NeuCLIRBench and NeuCLIRTech emphasize reproducibility and extended research in both general-news and technical domains.
TREC Neural Cross-Language Information Retrieval (NeuCLIR) is a TREC evaluation initiative and, across 2022–2024, a three-year track designed to study neural approaches to cross-language information access. It extends the older CLIR tradition into a setting centered on multilingual transformers, dense and learned-sparse retrieval, neural reranking, and, by 2024, report generation, using English topics over Chinese, Persian, and Russian news collections and, later, Chinese technical abstracts (Lawrie et al., 17 Sep 2025, Galuščáková et al., 2021).
1. Conceptual position within cross-language information retrieval
NeuCLIR is best understood as a modern, neural incarnation of a long line of cross-language information retrieval work. Classical CLIR begins from two assumptions that hold in monolingual ranked retrieval but fail across languages: the searcher cannot choose words that literally appear in the target-language documents, and the searcher may not be able to recognize relevance from raw target-language text. CLIR therefore must both reconcile the lexical mismatch between query and document languages and support usable cross-language access (Galuščáková et al., 2021).
Within that broader tradition, the main CLIR paradigms remain query translation, document translation, translation of both sides into a pivot or shared semantic space, and minimal-translation methods that rely on anchor phenomena and feedback. NeuCLIR is most closely associated with the shared-space paradigm, in which multilingual encoders place queries and documents into comparable representations, but the track also explicitly evaluates translation-based and hybrid systems. The result is not a break with earlier CLIR practice so much as a third-wave continuation: the fundamental problem remains unchanged, while translation probabilities, structured queries, and lexical matching are increasingly complemented by multilingual embeddings, late interaction, and neural rerankers (Galuščáková et al., 2021).
A useful technical continuity runs from probabilistic structured queries to neural late interaction. In classical CLIR, expected query-language term counts can be obtained from document-language term counts by multiplying through a translation probability matrix. In neural systems, the explicit matrix is replaced by learned multilingual representations and similarity functions. A canonical late-interaction form, used in ColBERT-style models that became central in NeuCLIR systems, is
which can be read as a dense analogue of translation-aware token matching (Galuščáková et al., 2021).
2. Track evolution from 2022 to 2024
NeuCLIR evolved from a first-year ad hoc CLIR benchmark into a multi-task program covering multilingual ranking, technical-domain retrieval, and report generation.
| Edition | Core tasks | Distinctive features |
|---|---|---|
| 2022 | Ad hoc CLIR, reranking CLIR, monolingual retrieval | First year; Chinese, Persian, and Russian news; 172 total runs from twelve teams |
| 2023 | News CLIR, monolingual news, reranking CLIR, MLIR, technical documents CLIR | Second year; four collections; 220 runs across all tasks from six teams |
| 2024 | News CLIR, news MLIR, technical documents CLIR, report generation | Final iteration; 274 runs for eight tasks from five teams |
The 2022 edition defined the basic NeuCLIR pattern: ad hoc ranked retrieval of Chinese, Persian, or Russian newswire documents using English topics, together with a reranking task built on NIST-provided BM25-over-machine-translated-document candidate sets, and monolingual retrieval for comparison and pool enrichment. It was the first year of the track and received 172 total runs submitted by twelve teams (Lawrie et al., 2023).
The 2023 edition retained the news CLIR tasks, added multilingual IR over the union of the three news collections, and introduced a pilot technical-documents CLIR task over Chinese scientific abstracts. It reported 220 runs across all tasks from six participating teams and framed the track as a modern CLIR benchmark explicitly designed to study neural methods at scale and across domains (Lawrie et al., 2024).
The 2024 edition completed the three-year arc. The track included test collections containing Chinese, Persian, and Russian news stories and Chinese academic abstracts, with four task types: CLIR from news, MLIR from news, report generation from news, and CLIR from technical documents. A total of 274 runs were submitted for eight tasks (Lawrie et al., 17 Sep 2025).
3. Collections, languages, and topic construction
The core news collections, often referred to as NeuCLIR-1, were drawn from the Common Crawl news dataset covering August 2016 to July 2021. Final statistics reported for the 2022 collection were 3,179,209 Chinese documents, 2,232,016 Persian documents, and 4,627,543 Russian documents. Documents were distributed in JSONL with fields such as id, time, title, text, and url, and filtering removed excessively long and very short texts (Lawrie et al., 2023).
Topic construction followed TREC-style procedures but was adapted for cross-language viability. In 2022, assessors produced English topic descriptions and topics were judged on whether they had relevant material in each language. A distinctive design choice was that topics developed by an annotator for one language were assessed by a different annotator when evaluating that topic on a different language. This encouraged multilingual viability rather than overfitting topics to a single national news ecosystem. After review for ambiguity and redundancy, 114 topics were released to participants (Lawrie et al., 2023).
The 2023 news tasks extended topic creation with both NIST topics and coordinator-created topics. Final retained counts were 62 topics for Chinese CLIR, 60 for Persian CLIR, 62 for Russian CLIR, and 65 for MLIR. Judgment statistics were reported per topic family and language, with average prevalence ranging from 3.27% to 9.86% depending on language and topic source (Lawrie et al., 2024).
The technical collection introduced in 2023 and later consolidated in NeuCLIRTech came from the Chinese Scientific Literature dataset. The 2023 pilot used 396,209 journal and thesis abstracts from 1,980 Chinese academic journals spanning 67 disciplines and the years 2010–2020 (Lawrie et al., 2024). NeuCLIRTech later described the same resource as 396,209 journal abstracts from 1,980 Chinese academic journals, with fields including doc_id, title, abstract, keywords, category, and discipline, and machine-translated English versions generated with Google Translate (Lawrie et al., 5 Feb 2026).
4. Tasks, judgments, and evaluation methodology
NeuCLIR’s task family is broader than ad hoc CLIR alone. In 2022 it comprised ad hoc CLIR, reranking CLIR, and monolingual retrieval. By 2023 and 2024 it also included MLIR over the three news languages, technical-documents CLIR over Chinese scientific abstracts, and report generation from news (Lawrie et al., 2023, Lawrie et al., 17 Sep 2025).
Across the retrieval tasks, the dominant evaluation measures were standard ranked-retrieval metrics. The 2022 overview specified nDCG@20, MAP, RBP, Recall@100, and Recall@1000. The graded form of discounted cumulative gain was defined as
with
Average precision and MAP were computed in the standard way using binary relevance derived from judged labels (Lawrie et al., 2023).
Judgments used a four-point relevance scale in which documents could be labeled Very Valuable, Somewhat Valuable, Not that Valuable, or Not Relevant. For official qrels in the 2022 track, these were mapped to a graded scale of 3 points for Very Valuable, 1 point for Somewhat Valuable, and 0 points for both Not that Valuable and Not Relevant. The merging of the two lowest labels was motivated by assessor agreement analysis, which showed that most disagreements were between those categories (Lawrie et al., 2023).
MLIR added a calibration problem absent from single-language CLIR: systems had to produce one ranked list across Chinese, Persian, and Russian documents. The 2023 overview therefore reported not only standard effectiveness measures but also a language-exposure fairness analysis that compared document exposure in the top ranks with the language distribution of relevant material. This made explicit a core MLIR question: whether a system’s score calibration across languages was aligned with multilingual relevance, rather than merely with per-language retrieval strength (Lawrie et al., 2024).
Report generation introduced a different evaluation layer. HLTCOE’s 2024 system description reported ARGUE score together with citation precision, nugget recall, nugget support, and sentence support, making report generation a retrieval-grounded synthesis task rather than a pure ranking task (Yang et al., 30 Sep 2025).
5. Retrieval architectures and methodological paradigms
NeuCLIR systems span the full CLIR design space. Translation-centered pipelines remain fundamental: query translation and document translation are both directly compatible with strong monolingual infrastructures, and this compatibility is one reason they remain competitive in NeuCLIR-style systems (Lin et al., 2023, Yao et al., 2020).
A concrete example of the query-translation branch is the integration of neural machine translation into CLIR. One study replaced phrase-based SMT with Transformer-based neural query translation, mined bilingual query pairs from clickthrough logs for domain adaptation, and introduced an asynchronous SMT+NMT architecture to hide neural latency behind caching. The broader implication for NeuCLIR is that translation quality, in-domain adaptation, and latency are coupled systems problems rather than isolated model choices (Yao et al., 2020).
On the ranking side, the 2022 NeuCLIR baseline study organized reproducible systems around document translation, query translation, and language-independent representations. It implemented BM25, SPLADE, xDPR, Rocchio feedback, reciprocal-rank fusion, monoT5, and mT5 in Anserini and Pyserini, showing how multi-stage monolingual architectures could serve as a scaffold for CLIR (Lin et al., 2023).
Dense and late-interaction architectures became especially prominent. In HLTCOE’s 2023 and 2024 submissions, PLAID served ColBERT-X models over passage-segmented documents, using MaxP aggregation from passage scores to document scores. Training regimes included Translate-Train, Translate-Distill, and Multilingual Translate-Train or Multilingual Translate-Distill, all aimed at making multilingual dense retrieval viable under NeuCLIR constraints (Yang et al., 2024, Yang et al., 30 Sep 2025).
A key training pattern for MLIR is mixed-language batching. In neural approaches to multilingual information retrieval, XLM-R fine-tuned with mixed-language batches from neural translations of MS MARCO passages achieved about 98% of the best MAP obtained by document translation plus neural ranking, while reducing indexing time by 84%; the 2% difference was not statistically significant. This is directly relevant to NeuCLIR’s MLIR task, where one ranked list must cover multiple document languages (Lawrie et al., 2022).
Reranking has progressively moved toward large multilingual sequence models and, later, general-purpose LLMs. mT5-XXL rerankers were already highly effective in NeuCLIR 2022, even when fine-tuned only on same-language query-document pairs and then applied to cross-lingual pairs (Jeronymo et al., 2023). By 2024, systems such as HLTCOE’s combined PLAID retrieval, mT5 cross-encoder reranking, and GPT-4o list-wise reranking via heapsort over selected passages (Yang et al., 30 Sep 2025).
6. Empirical findings and recurrent lessons
The 2022 track established several baseline facts. Hybrid models had the best average nDCG@20 across languages, roughly 0.42–0.52; learned-sparse systems performed very well in Persian and Russian; dense retrieval was highly variable; and sparse BM25-based runs remained strong non-neural baselines. Perhaps the most striking result was that, for Chinese and Persian, the best CLIR systems slightly outperformed the best monolingual systems in nDCG@20, while for Russian the best CLIR system reached roughly the same nDCG@20 as the best monolingual system (Lawrie et al., 2023).
The 2023 news tasks showed that top CLIR runs had moved into the high-0.5 nDCG@20 range. The best reported Chinese CLIR run reached 0.585, the best Persian run 0.603, and the best Russian run 0.598. MLIR remained harder: the best run reported nDCG around 0.523 together with strong α-nDCG, indicating that top systems could build single cross-language rankings but at some cost relative to per-language CLIR. Technical documents proved markedly more difficult: the best technical CLIR run reached nDCG@20 of 0.496, and the overview explicitly noted that no submission achieved an nDCG@20 above 0.5 (Lawrie et al., 2024).
These results sharpened an important track-level distinction. News CLIR had become a setting in which neural hybrid systems and LLM reranking could approach monolingual-level effectiveness, whereas technical CLIR remained bottlenecked by terminology, domain mismatch, and translation quality. This suggests that “neural CLIR” is not a single problem class: the same architecture family behaves very differently across general-news and technical-scientific corpora (Lawrie et al., 2024).
Detailed system studies from 2024 reinforced the centrality of multi-stage reranking. In HLTCOE’s 2024 paper, a CLIR news run that combined rank fusion, mT5 reranking, and GPT-4o reranking reported nDCG@20 values of 0.690 for Persian, 0.585 for Russian, and 0.664 for Chinese, with Recall@1000 between 0.974 and 0.993. For MLIR, the best reported HLTCOE run achieved nDCG@20 of 0.460, and for technical documents the best reported run reached 0.455 (Yang et al., 30 Sep 2025). Those are team-specific results rather than official track summaries, but they are technically revealing: strong first-stage retrieval remained necessary, yet large rerankers increasingly determined the top of the ranking. This suggests that, by the final year of the track, NeuCLIR effectiveness was often limited less by candidate generation than by second-stage cross-language relevance modeling.
7. Derivative benchmarks, extensions, and continuing significance
NeuCLIR did not end with the annual shared task. It generated a family of derivative resources that turned yearly evaluations into reusable benchmark infrastructure.
NeuCLIRBench consolidated the TREC NeuCLIR track topics of 2022, 2023, and 2024 into a single evaluation collection for monolingual, cross-language, and multilingual retrieval. It contains 250,128 graded relevance judgments across approximately 150 queries for the monolingual and cross-language tasks and 100 queries for multilingual retrieval, and it includes a fusion baseline of strong neural retrieval systems so that reranking research need not begin from BM25 candidates alone (Lawrie et al., 18 Nov 2025).
NeuCLIRTech performed a similar consolidation for the technical-documents task. It combines the TREC NeuCLIR 2023 and 2024 technical topics into a benchmark with 110 queries and 35,962 document judgments, supporting monolingual Chinese retrieval and English-to-Chinese CLIR over technical abstracts. It is explicitly positioned as a modern benchmark for neural CLIR in a domain where machine translation is weak and technical vocabulary is central (Lawrie et al., 5 Feb 2026).
mFollowIR repurposed NeuCLIR narratives into a multilingual benchmark for instruction-following retrieval. It used the narratives and their controlled edits to test whether retrieval models follow subtle changes in relevance conditions, and it reported strong cross-lingual performance with English-based retrievers trained using instructions but a notable drop in the multilingual setting. This suggests that NeuCLIR’s topic and narrative design is rich enough not only for ranked retrieval evaluation but also for studying instruction-sensitive retrieval behavior across languages (Weller et al., 31 Jan 2025).
Taken together, these derivatives mark a shift from NeuCLIR as a yearly shared task to NeuCLIR as a benchmark ecosystem. The track began as a neural CLIR evaluation over English queries and non-English documents; it matured into a platform for studying multilingual ranking calibration, technical-domain CLIR, LLM reranking, and retrieval-grounded generation. That trajectory places NeuCLIR at the center of contemporary cross-language IR research rather than at its periphery (Lawrie et al., 17 Sep 2025).
