---
title: Semi-Synthetic Test Collection
url: https://www.emergentmind.com/topics/semi-synthetic-test-collection
type: topic
---

# Semi-Synthetic Test Collection

A semi-synthetic test collection is a hybrid evaluation resource for information retrieval (IR), machine learning, or interpretability research in which one axis of the canonical test-collection triplet—queries, corpus, relevance judgments—is constructed from real-world, human-generated data, while another axis is created or labeled algorithmically, often using large language models (LLMs) or automated extraction pipelines. This approach occupies an intermediate position between fully manual collections (high reliability, high cost) and fully synthetic collections (maximum scale, but with higher risk of distribution shift or confounding bias). Semi-synthetic designs enable targeted analysis of bias and scalability trade-offs, facilitate reproducible benchmarking in data-sparse settings, and are now operationalized in diverse research subfields including ad hoc IR, entity-relationship search, retrieval-augmented generation, specialized legal corpora, and mechanistic neural network interpretability.

## 1. Semi-Synthetic vs. Manual and Synthetic Collections

Traditional fully manual test collections (e.g., those created under the Cranfield paradigm) involve human annotators writing queries and exhaustive relevance judgments. These are considered gold standards due to rigorous human curation but are slow and expensive to build at scale [2506.10301]. Fully synthetic collections automate both axes—queries and labels—using LLMs (e.g., GPT-4), enabling rapid creation but increasing susceptibility to LLM-inherited distributional or reporting biases.

The semi-synthetic approach fixes one axis as "real" (human-authored queries or judgments) and the other as "synthetic" (LLM-generated queries or labels), yielding two principal forms:
- Human queries + synthetic labels: e.g., real search topics judged by LLMs [2506.10301, 2604.00731].
- Synthetic queries + human labels: e.g., LLM-generated information needs labeled by expert raters [2506.10301].

This design isolates query-induced vs. label-induced biases. Weaknesses relative to full synthesis include ongoing human annotation requirements and possible propagation of legacy data artifacts.

## 2. Construction Pipelines and Methodologies

A prototypical semi-synthetic construction pipeline consists of:

### a) Corpus Preparation
Either an existing corpus (e.g., ClueWeb09-B, CORD-19, Algerian legal documents) or a constructed collection, sometimes augmented with automatically extracted elements such as tables, captions, or text chunks [2504.07584, 1706.03960, 2604.00731].

### b) Query Set Formation
Query topics can be wholly human-authored, LLM-synthesized, or selected from real search logs. Tools like web interfaces or controlled prompts ensure desired length, complexity, and coverage [2604.00731, 2506.10301].

### c) Candidate Pooling
Documents or passages most likely relevant to the queries are retrieved using multi-system pooling (e.g., BM25, dense bi-encoders) and fused (e.g., by Reciprocal Rank Fusion) to increase recall [2604.00731, 2504.07584].

### d) Relevance Labeling
Synthetic judgments are produced by LLMs in zero-shot or prompted mode. For instance, GPT-4 labeling with four-level granular judgments (0–3) as per TREC Deep Learning guidelines; zero-shot prompting is often sufficient [2506.10301, 2504.07584]. Alternatively, LLM-generated or automatically extracted queries can be labeled by humans [1706.03960].

### e) Human-in-the-Loop Verification
A subset of judgments can be randomly sampled for human validation, with calibration steps (e.g., Bland–Altman analysis, inter-annotator κ agreement), and label blending or correction applied if systematic discrepancies are found [2504.07584].

## 3. Statistical Properties and Evaluation Metrics

Performance is evaluated using classical IR metrics:
- Precision@k and Recall@k: Measures based on binary indicator of relevance among top-k results.
- Mean Average Precision (MAP): Mean across queries of the average precision at relevant document ranks.
- Normalized Discounted Cumulative Gain (nDCG@k): Uses graded relevance labels to discount lower-ranked hits, normalized by ideal ranking.

Labeling models are assessed for agreement using inter-annotator measures (Cohen's κ), label distribution divergence (KL), and bias estimation. Linear mixed-effects models quantify systematic bias induced by synthetic labeling, controlling for system architecture, query properties, and labeler (human vs. LLM) [2506.10301].

For generation tasks (RAG), context relevance, faithfulness, and answer relevance are scored using LLMs or embedding similarities [2504.07584].

## 4. Empirical Results and Bias Analysis

Empirical findings across multiple studies show:

- Synthetic labelers (e.g., GPT-4) yield systematically higher relevance scores compared to human judges. In one benchmark, GPT-4 delivered MAP and nDCG increases of +0.069 and +0.207, respectively; thus, absolute system effectiveness can be inflated [2506.10301].
- Despite absolute shifts, system-to-system correlations (Kendall's τ 0.6–0.8; Spearman's ρ up to 0.92) remain high, supporting relative ranking stability [2506.10301, 2604.00731].
- Query formulation bias: LLMs tend to synthesize longer queries, with stylistic differences from real-user needs; this can lead IR systems to overfit to synthetic verbosity [2506.10301].
- Label distribution bias: LLMs often underreport irrelevance, boosting low-level recall but compressing the range of truly relevant vs. nonrelevant items [2506.10301].
- System–model alignment bias: IR architectures based on similar LLMs as the synthetic judge tend to score higher, introducing a feedback risk in comparative evaluations [2506.10301].

For specialized domains, semi-synthetic pipelines such as STCALIR demonstrate near-human agreement (Hit@10 ≈ 0.785; system-level τ ≈ 0.89) with up to 99% reduction in annotation workload [2604.00731]. REANIMATOR shows that table-based semi-synthetic labeling can make document collections usable for previously unsupported downstream tasks, with agreement between top LLMs and humans (Cohen's κ ≈ 0.35–0.39) exceeding human–human agreement in some settings [2504.07584].

## 5. Practical Applications and Representative Benchmarks

The semi-synthetic paradigm is implemented across several domains:

| Collection/Framework   | Domain/Task               | Human Axis                    | Synthetic Axis                   |
|-----------------------|---------------------------|-------------------------------|----------------------------------|
| BiasSyntheticData [2506.10301] | Passage ranking/IR          | Queries or labels             | LLM-generated queries or labels  |
| STCALIR [2604.00731]           | Legal IR, low-resource      | Queries                       | Models for label pooling         |
| REANIMATOR [2504.07584]        | Table, passage, RAG         | Topics, corpus                | LLM-assigned relevance labels    |
| RELink [1706.03960]            | Entity-relationship search  | Queries (manual phrasing)     | Judgments from Wikipedia tables  |
| InterpBench [2407.14494]       | Interpretability of NNs     | Hand-written circuits/tasks   | Strictly controlled model weights|

These frameworks demonstrate cost savings, scalability, and benchmarking capabilities in settings where fully manual resources are unavailable or unscalable.

## 6. Recommendations and Best Practices

Key guidelines for constructing and using semi-synthetic test collections include [2506.10301, 2604.00731, 2504.07584]:
- **Bias Isolation**: Use semi-synthetic designs to disentangle query, label, and model biases.
- **Human Calibration**: Regularly sample for human validation, monitor inter-annotator agreement, and employ statistical corrections (e.g., label blending).
- **Pooling Diversity**: Use multiple retrieval models for candidate pooling and fusion to reduce candidate miss and increase label set diversity.
- **Metric Reporting**: Accompany absolute performance with rank correlations to human-judged orderings.
- **Open-source Infrastructure**: Release indices, code, scripts, and model checkpoints for complete reproducibility.
- **Iterative Oversight**: Update prompts, sampling, and modeling strategies in response to empirical drift or emerging annotation errors.

Such practices enable large-scale, cost-efficient, and robust test collection construction, while providing transparency and control over the specific biases introduced by synthetic components.

## 7. Limitations and Ongoing Research

The semi-synthetic approach, while highly effective for mitigating annotation cost and targeting specific evaluation needs, is subject to several limitations:

- Synthetic labelers can encode biases and reporting artifacts not present in the target population, threatening absolute validity.
- Query and judgment distributions may not fully represent future or adversarial information needs, especially in dynamic or evolving domains.
- Multi-modal or structured data extraction (e.g., table parsing) may introduce extraction errors that propagate to downstream metrics [2504.07584].
- Anchoring benchmark coverage to legacy corpora and topic sets constrains generalizability.

Ongoing research proposes scaling models (e.g., larger SIIT transformers), refining pooling/ranking pipelines, and deploying semi-synthetic benchmarks as "unit tests" for modular interpretability evaluation [2407.14494]. Routine calibration, human-in-the-loop augmentation, and careful documentation of sampling/randomization practices are recommended to ensure integrity as, by design, semi-synthetic collections will play an increasing role in IR, ML evaluation, and deep learning interpretability benchmarking.

Source: https://www.emergentmind.com/topics/semi-synthetic-test-collection