Ground-Truth Contamination in ML and Science

Updated 4 April 2026

Ground-truth contamination is the leakage of test labels into training or evaluation setups, which undermines empirical performance measures.
It spans various modes including pre-training, search-time, and synthetic contamination in both machine learning and experimental science.
Mitigation strategies such as statistical watermarking, rigorous auditing, and provenance tracking are essential to maintain benchmark integrity.

Ground-truth contamination refers to the direct or indirect leakage of test labels or held-out answers into settings (training, evaluation, retrieval, or even ground-truth construction pipelines) in a way that undermines the validity of performance metrics and benchmark comparisons. It encompasses not only classical pre-training contamination but also the proliferation of synthetic, self-referential labels and at-inference contamination in tool-augmented LLMs. The phenomenon, while prominent in LLMs and ML, extends to experimental science, measurement assays, and large-scale detector infrastructure. Despite its centrality to empirical claims, ground-truth contamination increasingly presents subtle, multi-stage, and epistemically complex challenges across research domains.

1. Definitions and Taxonomy

Ground-truth contamination, as codified in recent literature, divides into several operational modes with distinct causal mechanisms:

Pre-training ground-truth contamination: A strong form of leakage where the full triplet (input, prompt, answer)—not just the input text—appears during model pre-training. If evaluation example $(x_i, p_i, y_i)$ is added as sequence $s_i=[x_i\|p_i\|y_i]$ to the pre-training corpus $\mathcal{T}$ , the model can memorize exact input–answer relations rather than generalizing (Jiang et al., 2024).
Search-time contamination (STC): In search-augmented LLM agents, retrieval steps may directly surface external sources containing both the test question and its answer. If $\exists s \in S_i$ such that $s$ contains both $q_i$ and $y_i$ , evaluation is contaminated at inference time (Han et al., 12 Aug 2025).
Self-referential/synthetic contamination: When "ground truth" labels are generated by other models—possibly recursively—the notion of an external reference deteriorates, producing a looping, mimetic construct where models are evaluated on data derived from previous models rather than reality (Offenhuber, 14 Sep 2025).
Surface and bulk contamination in experimental science: In ultra-clean experiments (e.g., JUNO, Majorana), ground-truth contamination denotes trace levels of U/Th or other radioisotopes deposited on critical surfaces, requiring stringent sampling, detection, and process controls to validate integrity (Zhao et al., 8 Jul 2025, Christofferson et al., 2017, Vacri et al., 2020).
Benchmark contamination: Entry of public test sets into pre-training or instruction-tuning data, detectable empirically via order-sensitive statistical tests (e.g., Black Box test), biases reported model accuracy (Ahuja et al., 2024).

The term extends beyond simple dataset overlap and captures the epistemic consequences when models, evaluation protocols, or even "truth" itself become entangled with artificial, derivative, or mis-tracked labels.

2. Formal Frameworks and Detection Methodologies

Ground-truth contamination is rigorously formalized using explicit overlap or information-flow constructs:

$n$ -gram overlap: Define $\mathcal{N}_n(s)$ as all length- $n$ contiguous n-grams in sequence $s_i=[x_i\|p_i\|y_i]$ 0. Contamination occurs if $s_i=[x_i\|p_i\|y_i]$ 1 for test $s_i=[x_i\|p_i\|y_i]$ 2 and training corpus $s_i=[x_i\|p_i\|y_i]$ 3. The percentage metric is $s_i=[x_i\|p_i\|y_i]$ 4 (Jiang et al., 2024).
STC Indicator: For inference-time retrieval agents, contamination is measured as $s_i=[x_i\|p_i\|y_i]$ 5, and the contamination rate $s_i=[x_i\|p_i\|y_i]$ 6 over a benchmark (Han et al., 12 Aug 2025).
Memorization-based signal (LNE): Contamination is often associated with unusually low output entropy; Length-Normalized Entropy scores under greedy decoding are leveraged to flag high-confidence (potentially memorized) output sequences (Hou et al., 18 Sep 2025).
Statistical watermarking detection: By paraphrasing and cryptographically watermarking benchmarks, then quantifying model bias toward the watermark in token predictions, one can derive a valid $s_i=[x_i\|p_i\|y_i]$ 7-value for contamination post hoc, with binomial or beta-complete statistical tests (Sander et al., 24 Feb 2025).
Black Box testing: For multilingual and other benchmarks, statistical tests compare a model's performance on canonical vs. permuted test-orderings; order bias beyond chance is interpreted as contamination (Ahuja et al., 2024).
Physical/corrosive assay in low-background science: Direct sampling of surfaces (e.g., using PFA vials, leaching protocols, or ICP-MS quantification), comparing pre/post-cleaning or etching, establishes quantitative deposition rates and verifies protocol cleanliness (Zhao et al., 8 Jul 2025, Christofferson et al., 2017, Vacri et al., 2020).

Methodological rigor often requires distinction between benign text overlap and true answer leak, multilingual or cross-modal analogs, and calibration of thresholds for meaningful contamination.

3. Empirical Manifestations: LLMs, Scientific Instrumentation, and Beyond

The impact and signatures of ground-truth contamination vary by context:

LLM Benchmarks:
- Single-pass ground-truth leakage in GPT-2 can raise accuracy and ROUGE/UniEval scores by several percent, matching or exceeding the gains from larger models or more data (Jiang et al., 2024).
- In evaluation via internet-augmented tools, STC rates for major benchmarks (HLE, SimpleQA, GPQA) cluster around 1–4%, with contaminated items seeing 10–20 percentage-point accuracy boosts. Blocking contaminated sources (e.g., HuggingFace) yields 15pp drops specifically on the contaminated subset, directly attributing gains to answer retrieval, not reasoning (Han et al., 12 Aug 2025).
- Large-scale audits show that nearly all open models exhibit benchmark contamination on standard multilingual datasets; order-sensitivity is sufficient for rapid screening (Ahuja et al., 2024).
- Watermarked rerendering of questions makes contamination reliably detectable for even modest ( $s_i=[x_i\|p_i\|y_i]$ 80.04%) contamination: utility is preserved, and statistically minuscule $s_i=[x_i\|p_i\|y_i]$ 9-values are attainable upon training on watermarked versions (Sander et al., 24 Feb 2025).
Synthetic/Self-referential Truth Repositories:
- ML domains increasingly use synthetic data, teleologically adjusted labels, or "synthetic-in-the-loop" labeling, contaminating the reference with self-amplifying model artifacts. Paradoxically, such contamination sometimes improves robustness but also unmoors evaluation from reality (Offenhuber, 14 Sep 2025).
Image Restoration:
- The ground truth in deblurring/denoising is systematically contaminated by sensor limitations; frequency-domain corrections and "enhanced supervision" methods explicitly target the contaminated baseline to recover omitted detail and constrain hallucination (Ryou et al., 3 Dec 2025).
Low-background Physics:
- U/Th contamination is tracked at the picogram level via direct deposition assays. Stringent post-fabrication cleaning can recover bulk purity, and careful exposure time, air quality, and handling protocols—coupled with post-exposure quantitative assays—form the basis of ground-truth validation (Zhao et al., 8 Jul 2025, Christofferson et al., 2017, Vacri et al., 2020).

These findings confirm that even low contamination rates have outsize effects at the technical frontier—altering leaderboard rankings, invalidating benchmark claims, or exceeding physics or engineering thresholds.

4. Mechanisms and Dynamics of Contamination

Contamination is not confined to accidental overlap but is governed by complex, often endogenous, system dynamics:

Data publication and "benchmark leakage": Evaluation datasets released on public repositories (e.g., HuggingFace) are rapidly ingested by web-scale crawlers, propagating into future model pre-training by default (Han et al., 12 Aug 2025).
Self-referential contamination: Closed feedback loops, where models label data for future model generations, result in self-propagating bias, model drift, and loss of external calibration (Offenhuber, 14 Sep 2025).
Synthetic augmentation: Injection of noise, outliers, or invented samples (e.g., for privacy, diversity) can conceptually "contaminate" the truth pool, trading off external validity for generalization or robustness (Offenhuber, 14 Sep 2025).
Process/Materials Cross-contamination: In cleanroom environments or low-radioactivity experiments, process mistakes, improper acid etching, or insufficient surface handling control introduce measurable surface or bulk contamination recoverable only via iterative purification and protocol redesign (Christofferson et al., 2017).

A plausible implication is that, across domains, contamination risk is exacerbated by increased ease of dataset redistribution, model-centric workflows, and growing entanglement of public and private corpora.

5. Quantitative Impact and Benchmark Sensitivity

The empirical literature quantifies the effects of ground-truth contamination both at population and single-model levels:

Context / Experiment	Contamination Rate	Observed Effect	Reference
Search-augmented LLM agents (HLE)	$\mathcal{T}$ 03% (r_STC)	10–20pp accuracy boost; 15pp drop when HF blocked	(Han et al., 12 Aug 2025)
LLM pre-training (SQuAD/CNN)	$\mathcal{T}$ 1 copies	U-shaped curve: improvement, then overfitting	(Jiang et al., 2024)
Watermarked benchmarks	$\mathcal{T}$ 20.04\% data	$\mathcal{T}$ 3-value $\mathcal{T}$ 4 for +5% accuracy	(Sander et al., 24 Feb 2025)
Multilingual LLM evaluation	Not directly quantified	$\mathcal{T}$ 5 model–benchmark pairs contaminated	(Ahuja et al., 2024)
JUNO (U/Th air deposition)	$\mathcal{T}$ 60.02– $\mathcal{T}$ 71,600 pg d $\mathcal{T}$ 8 m $\mathcal{T}$ 9	<0.1–10 ng total exposure; well below physics req.	(Zhao et al., 8 Jul 2025)
Majorana (U/Th in Cu parts)	Pre/post etch: $\exists s \in S_i$ 00.01–0.5 μBq/kg	Surface or machining contamination reversible	(Christofferson et al., 2017)

Small but systematic contamination rates (1–4%) can be sufficient to drive material leaderboard shifts or invalidate cross-model comparisons, especially as the scale and re-use of benchmarks or ground-truth datasets intensifies.

6. Mitigation, Auditing, and Best Practices

Research emphasizes a spectrum of interventions:

Benchmark and Evaluation Design: Prefer dynamic or web-variant benchmarks, avoid static datasets, and restrict public test set releases or obfuscate benchmarks until post-evaluation (Han et al., 12 Aug 2025, Ahuja et al., 2024).
Source Filtering and Retrieval Guardrails: Multi-stage filtering (domain blacklists, date cutoffs, "Swiss cheese" exclusion) to preclude known dataset hosts or pre-release leaks (Han et al., 12 Aug 2025).
Systematic Auditing: Use automated substring/URL matching, entropy-based memorization detection (LNE), watermarking with post-hoc statistical checks, human/LLM trajectory review, and full publication of query and reasoning logs (Han et al., 12 Aug 2025, Sander et al., 24 Feb 2025, Hou et al., 18 Sep 2025).
Provenance and Versioning: Maintain detailed label lineage, versioning of datasets, and explicit tracking of synthetic versus externally-derived ground truth (Offenhuber, 14 Sep 2025).
Quantitative Assay: In low-background detector environments, implement direct surface/bulk sampling, process blanks, and calibration/validation cycles for all critical components (Zhao et al., 8 Jul 2025, Vacri et al., 2020).
Transparency: Release evaluation configurations, contamination rates, filtering steps, pre/post-mitigation scores; for closed models, provide sufficient design abstracts for audit (Han et al., 12 Aug 2025).

A robust strategy involves the integration of pre-training analysis, retrieval/intermediate result inspection, synthetic label tracking, and statistical defensibility for every step connecting model to reported performance.

7. Conceptual and Epistemological Implications

Ground-truth contamination is not merely a technical failure; it challenges the epistemological status of empirical results:

Circularity and Drift: As synthetic or model-labeled datasets become evaluation standards, performance becomes circular—models validated on their own artifacts (Offenhuber, 14 Sep 2025).
Transparency and Accountability: Without external reference, disagreements lack adjudication; audits and provenance become necessary but not always sufficient (Offenhuber, 14 Sep 2025).
Ethical Stakes: Privacy and synthetic data offer utility but also opacity, potentially obscuring whose interests or vulnerabilities are represented in "ground truth" (Offenhuber, 14 Sep 2025).
Permanence and Longevity: Once contaminated, benchmarks quickly become obsolete, with repeated leaks rendering even small evaluation datasets unfit for long-term frontier assessment (Han et al., 12 Aug 2025).

These features suggest that, absent a multi-pronged defense combining curation, control, auditing, and conceptual clarity, the research community faces ongoing risk that the meaning and validity of "ground truth" may further erode across both AI and the experimental sciences.