---
title: TREC NeuCLIR 2024 Collection
url: https://www.emergentmind.com/topics/trec-neuclir-2024-collection
type: topic
---

# TREC NeuCLIR 2024 Collection

The TREC NeuCLIR 2024 Collection is the preeminent benchmark suite for evaluating neural cross-language information retrieval (CLIR), multilingual information retrieval (MLIR), and report generation systems over large-scale, multi-source document corpora. Building on the HC4 test collections and expanded for the TREC Neural CLIR track, this suite provides rigorously constructed datasets, in-depth graded relevance judgments, and comprehensive task protocols supporting both ad hoc search and retrieval-augmented generation. It encompasses native and machine-translated news archives in Chinese, Persian, and Russian, as well as a technical abstracts corpus, with thousands of queries and over a quarter million deep relevance judgments. The collection is foundational for methodologically robust CLIR and MLIR experimentation and now underpins RAG architectures leveraging explicit nugget-oriented factual evaluation.

## 1. Collection Composition and Origins

The TREC NeuCLIR 2024 Collection is anchored in news and technical-domain corpora:

- **News Corpora (“NeuCLIR-1”)**: 
  - Chinese: ~3.18M documents (Common Crawl News, 2016–2021)
  - Persian: ~2.23M documents
  - Russian: ~4.63M documents (down-sampled for corpus balance)
  - Each corpus is natively authored and paired with machine translations into English using a high-quality Sockeye 2 Transformer model (FLORES-200 BLEU: 32.7–39.4; NTREX-128 BLEU: 29.3–30.5).

- **Technical Abstracts**:
  - 396K Chinese academic abstracts (CSL dataset, 67 disciplines).

- **Judged Topics**:
  - 56 Chinese, 68 Persian, 64 Russian topics for news CLIR.
  - 52 MLIR topics (requiring ≥2 relevant docs across languages).
  - 71 technical CLIR topics post pool-based pruning.

- **Topic Formulation**:
  - Each topic has a “title + description + narrative” structure; runtime retrieval uses “title + description.”
  - Topics are manually authored in English and professionally translated.
  - Topic creation is tightly coupled to corpus presence—iterative pilot searches ensure each query covers a viable scope and content availability [2509.14355, 2511.14758].

## 2. Relevance Assessment and Annotation Methodologies

Relevance judgments underpin the discriminatory power and depth of the collection:

- **Judgment Scales**:
  - News: Four-point (0–3), mapped to a three-point scale for evaluation (3→3, 2→1, 1→0, 0→0).
  - Technical: Three-valued (“very”, “somewhat”, “not” valuable).

- **Pooling and Topic Pruning**:
  - Pools constructed via the top 100 docs per team for each topic (news), top 35 (technical).
  - Deep pooling ensures nearly complete coverage for nDCG@20 and strong statistical power: ~1,500–1,700 judgments per topic.
  - Topics with <3 or >20% relevant docs were excluded to stabilize evaluation.

- **Annotation Strategies**:
  - HiCAL (High-Recall Interactive CALibration): An active learning iterative classifier selects high-probability relevant documents for manual judgment, ceasing with 20 consecutive non-relevant assessments or preset topic relevance criteria [2201.09992].
  - All relevance assignments are done by bilingual assessors on the native-language text. 

## 3. Tasks, Retrieval Scenarios, and Evaluation Protocols

The collection supports a broad array of retrieval and generative tasks:

| Task                       | Source Language | Target Document(s)                | Retrieval Output             |
|----------------------------|----------------|-----------------------------------|------------------------------|
| News CLIR                  | English        | Chinese, Persian, or Russian      | Ranked list (1,000 docs)     |
| News MLIR                  | English        | Chinese∪Persian∪Russian           | Single unified list (1,000)  |
| Technical CLIR             | English        | Chinese abstracts                 | Ranked list (1,000 docs)     |
| Report Generation (Pilot)  | English        | News docs (target language)       | Citation-grounded report     |
| Monolingual Retrieval      | C/P/R/E        | Native language                   | (Baseline, for pool depth)   |

- **Protocols**:
  - “Constrained” condition: no use of external bitext beyond defined corpora
  - “Open” condition: external MT/dictionary resources permitted
  - Submissions: up to six per language pair, supporting system diversity

- **Retrieval Scenarios**:
  - Monolingual, cross-language (English→foreign), and MLIR are all explicitly supported.
  - Queries written in English or translated manually to maximize real-world comparability.
  - Multilingual ad-hoc ranking presents unique calibration challenges due to score normalization across languages [2509.14355, 2511.14758].

## 4. Baseline Systems and Neural Paradigms

TREC NeuCLIR 2024 catalyzed the transition from sparse, lexical CLIR baselines to dense, neural and generative-retrieval pipelines:

- **Sparse Baselines**:
  - BM25 (Patapsco) in QGT (query translation), DT (document translation), and HT (human translation) configurations, all with/without RM3 expansion.
  - Probabilistic Structured Queries (HMM-based PSQ).

- **Dense and Hybrid Models**:
  - PLAID-X: Twin-encoder, multi-vector dense retrieval, XLM-RoBERTa Large distilled from mT5.
  - MILCO: Learned-sparse retrieval, XLM-RoBERTa 0.6B-based, crosslingual expansion.
  - Qwen3-Embeddings: Single-vector dense.
  - Fusion: Reciprocal Rank Fusion (RRF) of PLAID-X, MILCO, and Qwen3 yields nDCG@20 ≈ 0.59 (cross-language), ≈0.47 (multilingual) [2511.14758].

- **Neural Rerankers**:
  - Pointwise and listwise Large Language Model (LLM) rerankers (Qwen3, mT5, Rank1, Rank-K) over dense fusion outputs.

- **Dominant Pipelines**:
  - Two-stage retrieval: dense (Orig+DT) first-stage, then LLM-based crosslingual reranking (MonoT5, GPT-4, Claude 2), with retrieval scoring $score(d,q) = \lambda\, s_{dense}(d,q) + (1-\lambda)\,s_{sparse}(d,q)$.
  - MLIR uses score fusion and generative reranking to temper cross-language disparity.

## 5. Evaluation Metrics and Statistical Power

Robust evaluation is enabled by deep pooling, granular labeling, and standardized metrics:

- **Primary Metrics**:
  - nDCG@k: $DCG@k = \sum_{i=1}^k (2^{\text{gain}_i} - 1) / \log_2(i + 1)$
  - MAP@k, AP, Recall@k, Precision@k
  - Rank-Biased Precision (RBP), Mean Reciprocal Rank (MRR) (secondary)

- **Report Generation**:
  - Auto-Argue toolkit for nugget-based evaluation: combines citation precision, nugget recall, nugget support, and sentence support [2601.13222].
  - Explicit metrics: 
    - Nugget recall $R_n = |\,\mathrm{Covered}(G)\,|\,/\,|G|$,
    - Nugget density $D_n = |\,\mathrm{Covered}(G)|\,/\,|C|$,
    - Citation grounding $G_c = |\,\mathrm{Supported}(H)|\,/\,|H|$.

- **Pooling and Topic Stability**:
  - All primary tasks achieve “deep” pooling (≥0.98 “judged@20”).
  - Leave-one-run-out and leave-one-team-out analyses show Kendall’s τ ≥ 0.94, ensuring benchmark stability [2509.14355, 2511.14758].

## 6. Report Generation: Nugget-Based RAG and Empirical Insights

The 2024 pilot contains an explicit report-writing benchmark with granular gold-standard “nugget” annotations:

- **Example System (“Crucible”)**:
  - Initial retrieval via PLAID-X and alternatives (Qwen3, MILCO).
  - Q&A nugget bank generated per topic with LLMs; paraphrase deduplication using pairwise LLM calls.
  - SVC-based nugget ranking, sentence extraction, and per-nugget sentence assembly with explicit citation.
  - Optional LLM-judge verification mitigates citation drift.

- **Reported Performance** (*Rₙ: nugget recall, Dₙ: density, G_c: citation grounding*):

| System               | Rₙ   | Dₙ   | G_c   |
|----------------------|------|------|-------|
| Crucible             |0.429 |0.448 |0.902  |
| Crucible + Verify    |0.438 |0.457 |0.961  |
| GINGER (GPT-4o)      |0.177 |0.131 |0.571  |
| BulletPoints         |0.508 |0.340 |0.835  |

- **Key Implications**:
  - Nugget-centric architectures ensure systematic coverage and robust citation traceability.
  - Explicit per-fact sentence extraction and verification outperform cluster-based or purely latent methods for grounded generation.
  - Nugget coverage and citation precision remain below 0.5 and 0.9, respectively, in the best runs, with recall posing an open challenge [2601.13222].

## 7. Impact, Accessibility, and Continued Benchmarking

The TREC NeuCLIR 2024 Collection, as embodied in NeuCLIRBench, sets the standard for controlled research in CLIR and MLIR:

- **Statistical Discrimination**: 
  - 250,128 labelings over ca. 150 topics for mono/cross-language and 100 topics for MLIR provide strong significance power.
- **Breaks Lexical Baseline Dependence**:
  - Dense fusion and neural reranking now define the performance frontier, supporting method development outside the traditional BM25 paradigm.
- **Public Access and Ongoing Use**:
  - Datasets and code are openly available at neuclir.github.io and Hugging Face [2511.14758].
  - Collections inform emerging tracks, such as RAGTIME 2025, dedicated to multi-lingual RAG and summarization evaluation.
- **A plausible implication is** that due to deep pools and multilingual topic diversity, NeuCLIR 2024 is now the reference for both system development and model-based evaluation, especially for fact-grounded RAG in low-resource and cross-domain scenarios.

## References

- HC4: A New Suite of Test Collections for Ad Hoc CLIR [2201.09992]
- Overview of the TREC 2024 NeuCLIR Track [2509.14355]
- NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval [2511.14758]
- Incorporating Q&A Nuggets into Retrieval-Augmented Generation [2601.13222]

Source: https://www.emergentmind.com/topics/trec-neuclir-2024-collection