TREC NeuCLIR 2024: Neural Cross-Language IR

• The paper introduced a comprehensive cross-language IR benchmark that evaluated neural dense retrieval, LLM reranking, and a novel report-generation pilot across multiple languages.
• It detailed a robust methodology using multilingual news and technical document collections with machine and human translations to compare neural and sparse retrieval techniques.
• The track’s innovative report-generation pilot combined retrieval-augmented generation with citation-based evaluation, setting a new standard for grounded, multilingual information access.

Searching arXiv for papers on TREC NeuCLIR 2024 and closely related system/evaluation papers. TREC NeuCLIR 2024 was the third and final year of TREC’s Neural Cross-Language Information Retrieval effort, organized to study the effect of neural approaches on cross-language information access. The track created test collections containing Chinese, Persian, and Russian news stories and Chinese academic abstracts, and in 2024 it included four task types: Cross-Language Information Retrieval (CLIR) from news, Multilingual Information Retrieval (MLIR) from news, Report Generation from news, and CLIR from technical documents (Lawrie et al., 17 Sep 2025). A total of 274 runs were submitted by five participating teams and as baselines by the track coordinators for eight tasks across these four task types (Lawrie et al., 17 Sep 2025). Relative to earlier NeuCLIR editions, 2024 is notable for introducing a report-generation pilot that connected cross-language retrieval to retrieval-augmented generation with citation checking, while preserving the established emphasis on large-scale multilingual retrieval evaluation (Lawrie et al., 17 Sep 2025).

1. Track scope and task structure

TREC NeuCLIR 2024 comprised eight concrete tasks grouped into four task types: three News CLIR tasks, one News MLIR task, three Report Generation tasks, and one Technical Documents CLIR task (Lawrie et al., 17 Sep 2025). In the News CLIR tasks, English queries were used to search one non-English news collection at a time: Chinese, Persian, or Russian (Lawrie et al., 17 Sep 2025). In the MLIR task, English queries were used over the union of Chinese, Persian, and Russian news collections, and systems were required to return a single unified ranked list per topic (Lawrie et al., 17 Sep 2025). In the Report Generation tasks, the report request language and required report language were both English, while cited evidence had to come from one designated news collection language at a time: Chinese, Persian, or Russian (Lawrie et al., 17 Sep 2025). In the Technical Documents CLIR task, English queries were used to retrieve Chinese academic abstracts from the CSL collection (Lawrie et al., 17 Sep 2025).

The principal goal of the track was to study the effect of neural approaches on cross-language information access, but the 2024 design also exposed several specific research problems: comparison of neural and sparse methods, comparison of translation-based and direct CLIR approaches, multilingual score calibration in MLIR, and grounded report synthesis with citation support in a cross-language setting (Lawrie et al., 17 Sep 2025). This suggests that NeuCLIR 2024 functioned not only as a benchmark for ranked retrieval, but also as a bridge between classical CLIR evaluation and long-form, citation-backed generation.

2. Collections, languages, and query formulations

NeuCLIR 2024 used four languages in different roles. English served as the query language for all CLIR, MLIR, and Technical Documents tasks, and also as the language of report requests and generated reports (Lawrie et al., 17 Sep 2025). Chinese, Persian, and Russian were the target languages for the news collections, while Chinese was additionally the target language for the technical documents collection (Lawrie et al., 17 Sep 2025).

For the news tasks, the track reused the NeuCLIR-1 news document set, unchanged since 2022, with approximate collection sizes before the language-identification issue of about 2 million Persian documents, about 3 million Chinese documents, and about 5 million Russian documents, all drawn from Common Crawl News from August 2016 through July 2021 (Lawrie et al., 17 Sep 2025). The 2024 overview reports a language identification bug in which 1,157 of 16,951 CommonCrawl files were mis-identified and excluded, removing 765,299 Chinese, 317,392 Persian, and 3,410,884 Russian documents pre-filtering; Russian was down-sampled anyway, so the effective impact is smaller there, while for Chinese and Persian roughly half of those missing documents would otherwise have been included (Lawrie et al., 17 Sep 2025).

For the technical task, the collection was the CSL dataset, consisting of 396,209 Chinese journal abstracts from 1,980 journals across 67 disciplines, with Engineering, Science, Agriculture, and Medicine dominating (Lawrie et al., 17 Sep 2025). The 2026 NeuCLIRTech resource later described this same Chinese technical subtask as a stand-alone evaluation collection derived directly from the TREC NeuCLIR 2023 and 2024 technical documents task, which indicates the technical branch of NeuCLIR had become sufficiently stable for post-track reuse (Lawrie et al., 5 Feb 2026).

To lower barriers and support CLIR, coordinators provided machine-translated queries from English into Chinese, Persian, and Russian via Google Translate, machine-translated English versions of all news documents produced with Sockeye v2 Transformer models, and translations of the CSL abstracts and Chinese translations of the English topic fields for technical documents (Lawrie et al., 17 Sep 2025). They also released NeuMARCO, consisting of English MS MARCO queries and passages translated into the three NeuCLIR document languages (Lawrie et al., 17 Sep 2025). Human translations of some news topics into Chinese, Persian, and Russian were used for monolingual baselines and pool enrichment (Lawrie et al., 17 Sep 2025).

3. Topic development and relevance assessment

Topic development differed by task. For News CLIR and MLIR in 2024, topic development was done entirely by NIST assessors. Paired assessors with different language skills met virtually, brainstormed topics that revolve around events, people, and places and are significant enough to have coverage in more than one language, explored the collection using monolingual neural search in each language, and for each candidate topic issued a single monolingual search in each of the two languages and counted relevant documents in the top 25 results (Lawrie et al., 17 Sep 2025). If the count was less than 1 or greater than 20 in either language, they revised the topic to adjust specificity (Lawrie et al., 17 Sep 2025). This yielded 92 draft topics; after additional pruning, the final retained sets were 56 topics for Chinese CLIR, 68 for Persian CLIR, 64 for Russian CLIR, and 52 for MLIR (Lawrie et al., 17 Sep 2025).

For News CLIR and MLIR relevance judging, by-language pools were built using top-ranked documents from CLIR runs, top-ranked documents from MLIR runs, and any documents cited by Report Generation systems for corresponding report requests (Lawrie et al., 17 Sep 2025). For each team’s top nine runs, the top 100 documents were pooled, while for all other runs the top 50 documents were pooled (Lawrie et al., 17 Sep 2025). Relevance was judged on a four-point scale, then mapped to a three-grade scale for qrels: $3 \rightarrow 3$, $2 \rightarrow 1$, $1 \rightarrow 0$, and $0 \rightarrow 0$ (Lawrie et al., 17 Sep 2025). Topics were dropped from CLIR in a given language and from MLIR if more than 40% of judged documents in that language were somewhat or very valuable, and dropped from CLIR in a language if fewer than 2 documents were somewhat or very valuable, while still being kept in MLIR under the conditions specified in the overview (Lawrie et al., 17 Sep 2025).

For the Technical Documents task, assessors were graduate students and a postdoc in fields such as Biology, Computer Science, Earth Science, Economics, Engineering, Mathematics, and Physics, all with Chinese language skills and familiarity with research reading (Lawrie et al., 17 Sep 2025). They created English title, description, and narrative fields, translated into Chinese, and judged documents under instructions framed around writing the background or related work section of a scientific paper on the topic (Lawrie et al., 17 Sep 2025). Pools were created from the top 35 documents of each system run plus seed documents identified during topic development (Lawrie et al., 17 Sep 2025). After judging, 71 technical topics were retained in the official 2024 track overview (Lawrie et al., 17 Sep 2025).

4. Retrieval and reranking methods in submitted systems

NeuCLIR 2024 submissions were dominated by multi-stage neural architectures with dense retrieval, reranking, and fusion. The overview states that neural dense retrieval and reranking were universal among participants, while classical-only systems appeared only as coordinator baselines (Lawrie et al., 17 Sep 2025). Large generative LLMs as rerankers were especially prominent, and the top systems in CLIR, MLIR, and technical tasks all used generative models in the final reranking stage (Lawrie et al., 17 Sep 2025).

One important system description is HLTCOE’s TREC 2024 submission, which applied PLAID, an mT5 reranker, GPT-4 reranker, score fusion, and document translation to the TREC 2024 NeuCLIR track, and submitted runs to all NeuCLIR tasks (Yang et al., 30 Sep 2025). For PLAID, the team included Translate Distill (TD), Generate Distill (GD), and multi-lingual translate-distill (MTD) (Yang et al., 30 Sep 2025). TD uses scores from the mT5 model over English MS MARCO query-document pairs to learn how to score query-document pairs where the documents are translated to match the CLIR setting (Yang et al., 30 Sep 2025). GD follows TD but uses passages from the collection and queries generated by an LLM for training examples (Yang et al., 30 Sep 2025). MTD uses MS MARCO translated into multiple languages, allowing experiments on how to batch the data during training (Yang et al., 30 Sep 2025).

For CLIR news, HLTCOE reported that the best overall runs were kitchen_rankfuse.mt5rerank.gpt4rerank and closely related variants, showing that GPT-4 reranking on top of fused and mT5-ranked candidate sets was strongest overall (Yang et al., 30 Sep 2025). For MLIR, the best runs similarly used score fusion and GPT-4 reranking, with plaid_distill_clir.mt5rerank.scorefuse.gpt4rerank and kitchen_rankfuse.mt5rerank.scorefuse.gpt4rerank at the top of their results (Yang et al., 30 Sep 2025). For technical documents, plaid_distill_engzho.mt5rerank.gpt4rerank reached the best nDCG@20 among their runs, and the paper concludes that the technical documents task represents a huge domain shift, which leads to different algorithm rankings, and that after reranking, combining TD and GD is the most effective of the dense retrieval approaches (Yang et al., 30 Sep 2025).

This system-level evidence is consistent with broader NeuCLIR results from earlier years. In 2022, simple yet effective baselines showed that document translation with SPLADE was consistently the strongest single first-stage approach, while reranking narrowed the performance gap between first-stage retrievers and often made weaker initial retrieval pipelines competitive end-to-end (Lin et al., 2023). NeuralMind-UNICAMP’s 2022 study also reported that mT5-XXL, fine-tuned only on monolingual query-document pairs, proved viable for CLIR tasks where query-document pairs are in different languages, even in the presence of suboptimal first-stage retrieval performance (Jeronymo et al., 2023). These earlier findings remained methodologically relevant to NeuCLIR 2024 because the 2024 track continued to evaluate CLIR, MLIR, and reranking at scale over the same multilingual news collections (Lawrie et al., 17 Sep 2025).

5. Report generation pilot and ARGUE-style evaluation

The principal novelty of NeuCLIR 2024 was the Report Generation pilot. The 2024 overview states that NeuCLIR introduced a new Report Generation pilot and that report generation would continue in the planned TREC 2025 RAGTIME track (Lawrie et al., 17 Sep 2025). Each report generation task used an English report request and required an English report whose substantive statements were supported by citations to documents in one designated news collection language (Lawrie et al., 17 Sep 2025).

The report request format included a request ID, collection ID, background text, problem statement text, and a character limit (Lawrie et al., 17 Sep 2025). Report requests were derived from MLIR topics, with assessors expanding the topic description and narrative into background and problem statement while keeping the core information need (Lawrie et al., 17 Sep 2025). The 2024 overview reports 59 report requests developed, assigned to all three languages for 177 requests in the test data, with 21 Chinese, 20 Persian, and 21 Russian requests actually assessed in 2024 (Lawrie et al., 17 Sep 2025).

Assessment followed a nugget-based framework derived from ARGUE. For each report request, assessors wrote an example report, defined nugget questions, recorded known answers in English aligned to documents, later refined the questions, flagged each nugget as “ok” or “vital,” and extended answer lists by scanning relevant documents from CLIR qrels and cited documents in participant reports (Lawrie et al., 17 Sep 2025). Citation judging was performed sentence by sentence, with up to two citations per sentence, and assessors labeled whether each citation was fully supported, partially supported, or not supported (Lawrie et al., 17 Sep 2025). Sentence–nugget mapping was then performed without showing citations to avoid bias (Lawrie et al., 17 Sep 2025).

The 2024 overview identifies the principal report-generation evaluation dimensions as Citation Precision, Nugget Recall, Nugget Support, Sentence Support, and a composite ARGUE score (Lawrie et al., 17 Sep 2025). The later Auto-ARGUE paper clarified how the report generation pilot functioned as a case-study evaluation benchmark for cross-lingual report generation: systems generated English reports from one of three non-English collections, assessors judged sentence support and nugget recall on reports for the same 21 topics for each run, each topic had 10–20 nuggets, and assessors identified attesting documents for each answer, thus providing the set of relevant documents (Walden et al., 30 Sep 2025). Auto-ARGUE used the 51 runs from the RG pilot task of TREC 2024 NeuCLIR, with 17 runs on each of the Chinese, Russian, and Farsi collections (Walden et al., 30 Sep 2025). Its analysis reported good agreement between automatic LLM-based system rankings and human rankings on both sentence precision and nugget recall, with particularly strong results on sentence precision (Walden et al., 30 Sep 2025).

This suggests that the NeuCLIR 2024 report-generation pilot was designed not merely as a generative task, but as an explicitly citation-grounded and nugget-structured benchmark, closely aligned with evaluation concepts that could later be automated.

6. Results, system behavior, and research implications

The 2024 overview reports that top CLIR runs in all three languages were neural pipelines with LLM reranking, with HLTCOE and h2oloo runs among the strongest performers, often containing gpt4 or mt5rerank in their identifiers (Lawrie et al., 17 Sep 2025). For Chinese, HLTCOE’s kitchen_rankfuse.mt5rerank.gpt4rerank reached nDCG@20 of about 0.66; for Persian, h2oloo fusion and reranking runs reached about 0.69–0.70; for Russian, HLTCOE’s plaid_distill_engrus.mt5rerank.gpt4rerank reached about 0.59 (Lawrie et al., 17 Sep 2025). Sparse coordinator baselines such as BM25 with RM3 were substantially below neural systems (Lawrie et al., 17 Sep 2025).

For MLIR, best runs were fusion-heavy pipelines, with h2oloo systems reaching nDCG@20 of about 0.54–0.55 (Lawrie et al., 17 Sep 2025). The overview emphasizes that MLIR is harder than language-specific CLIR because of cross-language calibration and uneven coverage of relevant documents across languages (Lawrie et al., 17 Sep 2025). It also reports a positive but imperfect relationship between effectiveness and fairness, with Kendall’s $\tau = 0.61$ between MLIR effectiveness and fairness, indicating that the best systems by nDCG@20 are not necessarily the fairest across languages (Lawrie et al., 17 Sep 2025). This suggests that MLIR in NeuCLIR 2024 exposed a distinct systems problem beyond simple per-language retrieval quality.

For technical documents, top neural CLIR systems achieved nDCG@20 around 0.49–0.50, while classical baselines were around 0.25–0.31 (Lawrie et al., 17 Sep 2025). The overview remarks that the gap of about 0.3 nDCG@20 between neural and sparse methods is larger than in news, reflecting the challenge of technical vocabulary and the benefit of domain-adapted dense models and LLM reranking (Lawrie et al., 17 Sep 2025). The later NeuCLIRTech paper reported a best single-retriever first-stage of 0.472 nDCG@20 in CLIR with Qwen3-Embedding 8B and best reranked performance of 0.533 nDCG@20 with Rank-K over fusion candidates, offering a post-track indication of how the technical subtask continued to support stronger reranking research (Lawrie et al., 5 Feb 2026). A plausible implication is that the technical documents branch of NeuCLIR became a particularly useful setting for isolating the effects of multilingual retrieval, translation, and reranking under domain shift.

For report generation, the 2024 overview reports best ARGUE scores of about 0.73 for Chinese, 0.87 for Persian, and 0.81 for Russian, all from HLTCOE extractive GPT-4o-based systems (Lawrie et al., 17 Sep 2025). The HLTCOE 2024 system paper states that the extractive approach performed better than the abstractive approach in terms of ARGUE score and nugget recall, and that this is not surprising because the extractive approach was designed to favor the inclusion of more facts over producing a fluent report (Yang et al., 30 Sep 2025). The same paper reports that GPT3.5 struggled especially in terms of citation precision (Yang et al., 30 Sep 2025). This aligns with a later nugget-augmented generation study, which evaluated Crucible on the TREC NeuCLIR 2024 Report Generation Pilot and found that nugget-first pipelines decisively outperformed GINGER and generic agentic RAG baselines on nugget recall, density, and citation support, while BulletPoints retained the highest nugget recall among the compared systems (Dietz et al., 19 Jan 2026). That paper also observed that Crucible achieved citation support around 0.9–0.96 on NeuCLIR 2024, raising the possibility that either the citation-support problem was close to solved under Auto-ARGUE’s metric or that more precise evaluation methods were needed (Dietz et al., 19 Jan 2026).

7. Legacy and transition beyond 2024

NeuCLIR 2024 completed a three-year effort that provided enduring CLIR and MLIR testbeds across multiple languages and domains, established realistic baselines and best practices for neural cross-language retrieval, and seeded the next generation of TREC tracks that move from ranking documents to producing grounded, multilingual reports (Lawrie et al., 17 Sep 2025). The 2024 overview explicitly states that NeuCLIR ends in 2024 and that future work moves to RAGTIME 2025, which will introduce a new multilingual news collection with balanced size, include Arabic and English in addition to Chinese and Russian, allow systems to use all four languages as evidence, emphasize automation of evaluation, and harmonize output formats with other TREC RAG tracks (Lawrie et al., 17 Sep 2025).

In retrospect, NeuCLIR 2024 can be situated at the intersection of three strands of research. First, it preserved the TREC tradition of large-scale, reusable retrieval evaluation with leave-one-run-out and leave-one-team-out analyses demonstrating strong reusability of the collections (Lawrie et al., 17 Sep 2025). Second, it consolidated the role of neural dense retrieval, learned-sparse retrieval, document and query translation, fusion, and LLM reranking in multilingual IR (Yang et al., 30 Sep 2025, Lin et al., 2023). Third, it introduced a report-generation task whose nugget- and citation-based evaluation made NeuCLIR an early benchmark for multilingual, retrieval-augmented generation with explicit grounding (Lawrie et al., 17 Sep 2025, Walden et al., 30 Sep 2025).

A common misconception is to treat NeuCLIR 2024 solely as a ranked retrieval benchmark. The official overview and subsequent methodological papers indicate a broader significance: the track simultaneously evaluated ranked retrieval over multilingual news and technical domains, multilingual single-list ranking, and long-form grounded report generation, all within a common cross-language information access framework (Lawrie et al., 17 Sep 2025, Walden et al., 30 Sep 2025). Another misconception is that the report-generation pilot displaced retrieval. In practice, the pilot made retrieval even more central, because report systems were judged not just on textual fluency but on nugget coverage and citation support grounded in the underlying collections (Lawrie et al., 17 Sep 2025, Walden et al., 30 Sep 2025).

TREC NeuCLIR 2024 therefore occupies a distinctive place in the evolution of multilingual IR evaluation: it was the final NeuCLIR edition, but also the point at which cross-language retrieval evaluation and citation-grounded generation evaluation were integrated into a single experimental ecosystem (Lawrie et al., 17 Sep 2025).