Semi-Synthetic Test Collection

Updated 28 May 2026

Semi-synthetic test collections are hybrid evaluation resources that combine human-generated data with algorithmically produced labels or queries.
They offer a balanced alternative to fully manual and fully synthetic collections by mitigating cost, scalability, and bias trade-offs.
Construction pipelines involve corpus preparation, query formation, candidate pooling, and human-in-the-loop validation for robust reproducible benchmarking.

A semi-synthetic test collection is a hybrid evaluation resource for information retrieval (IR), machine learning, or interpretability research in which one axis of the canonical test-collection triplet—queries, corpus, relevance judgments—is constructed from real-world, human-generated data, while another axis is created or labeled algorithmically, often using LLMs or automated extraction pipelines. This approach occupies an intermediate position between fully manual collections (high reliability, high cost) and fully synthetic collections (maximum scale, but with higher risk of distribution shift or confounding bias). Semi-synthetic designs enable targeted analysis of bias and scalability trade-offs, facilitate reproducible benchmarking in data-sparse settings, and are now operationalized in diverse research subfields including ad hoc IR, entity-relationship search, retrieval-augmented generation, specialized legal corpora, and mechanistic neural network interpretability.

1. Semi-Synthetic vs. Manual and Synthetic Collections

Traditional fully manual test collections (e.g., those created under the Cranfield paradigm) involve human annotators writing queries and exhaustive relevance judgments. These are considered gold standards due to rigorous human curation but are slow and expensive to build at scale (Rahmani et al., 12 Jun 2025). Fully synthetic collections automate both axes—queries and labels—using LLMs (e.g., GPT-4), enabling rapid creation but increasing susceptibility to LLM-inherited distributional or reporting biases.

The semi-synthetic approach fixes one axis as "real" (human-authored queries or judgments) and the other as "synthetic" (LLM-generated queries or labels), yielding two principal forms:

Human queries + synthetic labels: e.g., real search topics judged by LLMs (Rahmani et al., 12 Jun 2025, Hatem et al., 1 Apr 2026).
Synthetic queries + human labels: e.g., LLM-generated information needs labeled by expert raters (Rahmani et al., 12 Jun 2025).

This design isolates query-induced vs. label-induced biases. Weaknesses relative to full synthesis include ongoing human annotation requirements and possible propagation of legacy data artifacts.

2. Construction Pipelines and Methodologies

A prototypical semi-synthetic construction pipeline consists of:

a) Corpus Preparation

Either an existing corpus (e.g., ClueWeb09-B, CORD-19, Algerian legal documents) or a constructed collection, sometimes augmented with automatically extracted elements such as tables, captions, or text chunks (Engelmann et al., 10 Apr 2025, Saleiro et al., 2017, Hatem et al., 1 Apr 2026).

b) Query Set Formation

Query topics can be wholly human-authored, LLM-synthesized, or selected from real search logs. Tools like web interfaces or controlled prompts ensure desired length, complexity, and coverage (Hatem et al., 1 Apr 2026, Rahmani et al., 12 Jun 2025).

c) Candidate Pooling

Documents or passages most likely relevant to the queries are retrieved using multi-system pooling (e.g., BM25, dense bi-encoders) and fused (e.g., by Reciprocal Rank Fusion) to increase recall (Hatem et al., 1 Apr 2026, Engelmann et al., 10 Apr 2025).

d) Relevance Labeling

Synthetic judgments are produced by LLMs in zero-shot or prompted mode. For instance, GPT-4 labeling with four-level granular judgments (0–3) as per TREC Deep Learning guidelines; zero-shot prompting is often sufficient (Rahmani et al., 12 Jun 2025, Engelmann et al., 10 Apr 2025). Alternatively, LLM-generated or automatically extracted queries can be labeled by humans (Saleiro et al., 2017).

e) Human-in-the-Loop Verification

A subset of judgments can be randomly sampled for human validation, with calibration steps (e.g., Bland–Altman analysis, inter-annotator κ agreement), and label blending or correction applied if systematic discrepancies are found (Engelmann et al., 10 Apr 2025).

3. Statistical Properties and Evaluation Metrics

Performance is evaluated using classical IR metrics:

Precision@k and Recall@k: Measures based on binary indicator of relevance among top-k results.
Mean Average Precision (MAP): Mean across queries of the average precision at relevant document ranks.
Normalized Discounted Cumulative Gain (nDCG@k): Uses graded relevance labels to discount lower-ranked hits, normalized by ideal ranking.

Labeling models are assessed for agreement using inter-annotator measures (Cohen's κ), label distribution divergence (KL), and bias estimation. Linear mixed-effects models quantify systematic bias induced by synthetic labeling, controlling for system architecture, query properties, and labeler (human vs. LLM) (Rahmani et al., 12 Jun 2025).

For generation tasks (RAG), context relevance, faithfulness, and answer relevance are scored using LLMs or embedding similarities (Engelmann et al., 10 Apr 2025).

4. Empirical Results and Bias Analysis

Empirical findings across multiple studies show:

Synthetic labelers (e.g., GPT-4) yield systematically higher relevance scores compared to human judges. In one benchmark, GPT-4 delivered MAP and nDCG increases of +0.069 and +0.207, respectively; thus, absolute system effectiveness can be inflated (Rahmani et al., 12 Jun 2025).
Despite absolute shifts, system-to-system correlations (Kendall's τ 0.6–0.8; Spearman's ρ up to 0.92) remain high, supporting relative ranking stability (Rahmani et al., 12 Jun 2025, Hatem et al., 1 Apr 2026).
Query formulation bias: LLMs tend to synthesize longer queries, with stylistic differences from real-user needs; this can lead IR systems to overfit to synthetic verbosity (Rahmani et al., 12 Jun 2025).
Label distribution bias: LLMs often underreport irrelevance, boosting low-level recall but compressing the range of truly relevant vs. nonrelevant items (Rahmani et al., 12 Jun 2025).
System–model alignment bias: IR architectures based on similar LLMs as the synthetic judge tend to score higher, introducing a feedback risk in comparative evaluations (Rahmani et al., 12 Jun 2025).

For specialized domains, semi-synthetic pipelines such as STCALIR demonstrate near-human agreement (Hit@10 ≈ 0.785; system-level τ ≈ 0.89) with up to 99% reduction in annotation workload (Hatem et al., 1 Apr 2026). REANIMATOR shows that table-based semi-synthetic labeling can make document collections usable for previously unsupported downstream tasks, with agreement between top LLMs and humans (Cohen's κ ≈ 0.35–0.39) exceeding human–human agreement in some settings (Engelmann et al., 10 Apr 2025).

5. Practical Applications and Representative Benchmarks

The semi-synthetic paradigm is implemented across several domains:

Collection/Framework	Domain/Task	Human Axis	Synthetic Axis
BiasSyntheticData (Rahmani et al., 12 Jun 2025)	Passage ranking/IR	Queries or labels	LLM-generated queries or labels
STCALIR (Hatem et al., 1 Apr 2026)	Legal IR, low-resource	Queries	Models for label pooling
REANIMATOR (Engelmann et al., 10 Apr 2025)	Table, passage, RAG	Topics, corpus	LLM-assigned relevance labels
RELink (Saleiro et al., 2017)	Entity-relationship search	Queries (manual phrasing)	Judgments from Wikipedia tables
InterpBench (Gupta et al., 2024)	Interpretability of NNs	Hand-written circuits/tasks	Strictly controlled model weights

These frameworks demonstrate cost savings, scalability, and benchmarking capabilities in settings where fully manual resources are unavailable or unscalable.

6. Recommendations and Best Practices

Key guidelines for constructing and using semi-synthetic test collections include (Rahmani et al., 12 Jun 2025, Hatem et al., 1 Apr 2026, Engelmann et al., 10 Apr 2025):

Bias Isolation: Use semi-synthetic designs to disentangle query, label, and model biases.
Human Calibration: Regularly sample for human validation, monitor inter-annotator agreement, and employ statistical corrections (e.g., label blending).
Pooling Diversity: Use multiple retrieval models for candidate pooling and fusion to reduce candidate miss and increase label set diversity.
Metric Reporting: Accompany absolute performance with rank correlations to human-judged orderings.
Open-source Infrastructure: Release indices, code, scripts, and model checkpoints for complete reproducibility.
Iterative Oversight: Update prompts, sampling, and modeling strategies in response to empirical drift or emerging annotation errors.

Such practices enable large-scale, cost-efficient, and robust test collection construction, while providing transparency and control over the specific biases introduced by synthetic components.

7. Limitations and Ongoing Research

The semi-synthetic approach, while highly effective for mitigating annotation cost and targeting specific evaluation needs, is subject to several limitations:

Synthetic labelers can encode biases and reporting artifacts not present in the target population, threatening absolute validity.
Query and judgment distributions may not fully represent future or adversarial information needs, especially in dynamic or evolving domains.
Multi-modal or structured data extraction (e.g., table parsing) may introduce extraction errors that propagate to downstream metrics (Engelmann et al., 10 Apr 2025).
Anchoring benchmark coverage to legacy corpora and topic sets constrains generalizability.

Ongoing research proposes scaling models (e.g., larger SIIT transformers), refining pooling/ranking pipelines, and deploying semi-synthetic benchmarks as "unit tests" for modular interpretability evaluation (Gupta et al., 2024). Routine calibration, human-in-the-loop augmentation, and careful documentation of sampling/randomization practices are recommended to ensure integrity as, by design, semi-synthetic collections will play an increasing role in IR, ML evaluation, and deep learning interpretability benchmarking.