Benchmark Contamination in Model Evaluation

• Benchmark contamination is the leakage of test or similar data into model training, leading to artificially high performance results.
• It encompasses various forms of leakage such as direct train-test overlap, exposure to annotation guidelines, and semantic information leakage.
• Detection methods range from corpus auditing and overlap metrics to statistical testing and watermarking, guiding mitigation and decontamination strategies.

Benchmark contamination is the exposure of benchmark test data, or close variants of it, to a model during pre-training, fine-tuning, or deployment-time learning, so that evaluation no longer measures generalization on unseen data. The phenomenon includes direct train–test overlap, ingestion of raw source texts, exposure to annotation guidelines or labels, semantic or informational leakage, and operational definitions in which contamination is treated as artificially inflated, non-generalizing benchmark performance rather than literal membership alone. Across NLP, multilingual evaluation, NL2SQL, coding, and reasoning-model leaderboards, the reported consequence is score inflation, distorted model ranking, and compromised scientific conclusions (Sainz et al., 2023, Xu et al., 2024, Dekoninck et al., 2024).

1. Definition and conceptual scope

A central formulation defines contamination as the case in which a LLM is trained on the test split of a benchmark and is then evaluated on that same benchmark. In that formulation, the worst case is test-split contamination, which invalidates any performance claim; train/dev-split contamination is less severe, but still invalidates zero-shot or few-shot claims. The same account distinguishes guideline contamination, where the model has seen annotation instructions and label definitions; raw-text contamination, where it has ingested the unannotated source text used to build the benchmark; and annotation contamination, where it has seen benchmark labels (Sainz et al., 2023).

A broader exposure taxonomy distinguishes semantic, information, data, and label levels. At the semantic level, the model sees identical or highly similar content; at the information level, it sees metadata such as label distributions or dataset timestamps; at the data level, it sees test inputs without labels; and at the label level, it sees full input–label pairs. A related formulation for evaluation-time mitigation distinguishes exact, semantic, and domain contamination according to whether the leakage set is a subset of the test set, semantically equivalent to it, or drawn from the same distribution (Xu et al., 2024, Chai et al., 27 Jan 2026).

Another distinction concerns the training objective. PaCoST defines Type I contamination as next-token prediction over the concatenation $(x,y)$, minimizing $-\log \mathcal{P}(x,y)$, and Type II contamination as next-token prediction conditioned on the instruction only, minimizing $-\log \mathcal{P}(y \mid x)$. This makes explicit that leakage may concern either the full benchmark item or only the answer given the prompt (Zhang et al., 2024).

A performance-centered definition departs from membership alone. ConStat defines contamination as a model’s benchmark performance being artificially inflated and non-generalizing relative to a primary benchmark and a reference benchmark of the same task family. This suggests that contamination is not reducible to exact duplicate detection: a model may behave as contaminated when its benchmark advantage fails to transfer to rephrased samples, synthetic samples from the same distribution, or another benchmark for the same task (Dekoninck et al., 2024).

2. Formalizations and quantitative criteria

For open models with transparent corpora, one proposed measure is benchmark–data overlap. Given a pre-training corpus $D$ and a benchmark $B$,

$\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$

Its interpretation is the fraction of benchmark tokens or examples that can be found verbatim in the model’s training data (Sainz et al., 2023).

For closed models, an alternative is the extractability ratio. If $d$ is an evaluation dataset and $m$ is a model, an example $s \in d$ is extractable if some prompt context causes $m$ to reproduce $-\log \mathcal{P}(x,y)$0 exactly. The degree of contamination is then

$-\log \mathcal{P}(x,y)$1

This operationalizes contamination as the fraction of evaluation examples memorized verbatim (Sainz et al., 2023).

A simpler risk formulation defines

$-\log \mathcal{P}(x,y)$2

with expected contamination rate

$-\log \mathcal{P}(x,y)$3

where $-\log \mathcal{P}(x,y)$4. In practice, this formulation emphasizes that even when the training corpus is unknown, exposure probability can be treated conservatively (Jacovi et al., 2023).

Performance-based formulations attempt to estimate inflation directly. ConStat constructs a hardness-correction function $-\log \mathcal{P}(x,y)$5 from uncontaminated reference models and defines

$-\log \mathcal{P}(x,y)$6

A model is contaminated for $-\log \mathcal{P}(x,y)$7 if its score on the primary benchmark exceeds the reference-based expectation; bootstrap lower confidence bounds then produce a $-\log \mathcal{P}(x,y)$8-value for testing contamination (Dekoninck et al., 2024).

Dataset-level contamination can also be parameterized as a latent fraction. KDS writes an evaluation set as $-\log \mathcal{P}(x,y)$9 with

$-\log \mathcal{P}(y \mid x)$0

and measures the divergence between pre-fine-tuning and post-fine-tuning kernel similarity matrices of sample embeddings. In controlled experiments, the resulting Kernel Divergence Score showed near-perfect monotonicity with $-\log \mathcal{P}(y \mid x)$1 and high consistency across random splits (Choi et al., 2 Feb 2025).

A further aggregation-oriented framework, DCR, scores four levels—semantic, informational, data, and label contamination—via prompt-based checks, combines them with a Mamdani fuzzy inference system, and adjusts raw accuracy by

$-\log \mathcal{P}(y \mid x)$2

The intended use is contamination-aware reporting rather than only binary detection (Xu et al., 15 Jul 2025).

3. Detection methodologies

For open models, contamination detection often begins with direct corpus auditing. Proposed methods include string matching on pre-training corpora, as in Dodge et al. (2021), and large-scale auditing tools such as ROOTS search and Data Portraits. A more elaborate retrieval pipeline was built on Pyserini over The Pile and C4 using BM25 indexing, top-$-\log \mathcal{P}(y \mid x)$3 retrieval with $-\log \mathcal{P}(y \mid x)$4, overlapping 13-gram chunking, and chunk–example scoring via cosine similarity, Jaccard index, n-Gram $-\log \mathcal{P}(y \mid x)$5, BM25, or GPTScore. For each example, the maximum chunk score becomes its contamination indicator, and the benchmark-level score is the average of these maxima (Sainz et al., 2023, Deng et al., 2023).

For proprietary or otherwise opaque models, memorization-based probing is common. TS-Guessing masks either a key token in a question or a wrong answer option in a multiple-choice item and tests whether the model reconstructs the hidden content. The overall guessing success rate is

$-\log \mathcal{P}(y \mid x)$6

This procedure is intended to expose benchmark familiarity without requiring access to the training corpus (Deng et al., 2023).

PaCoST replaces overlap search with paired confidence testing. For each original instruction $-\log \mathcal{P}(y \mid x)$7, it generates a rephrased counterpart $-\log \mathcal{P}(y \mid x)$8, estimates a confidence difference $-\log \mathcal{P}(y \mid x)$9, and performs a one-sided paired-samples $D$0-test of $D$1 versus $D$2. If $D$3, the benchmark is declared contaminated. The method is designed to be training-data-access free, contamination-type free, training-data-length free, stable, and threshold free (Zhang et al., 2024).

Another black-box method is the shard-level permutation test used for multilingual benchmarks. The evaluation set is partitioned into $D$4 shards, each shard is scored in canonical order and under $D$5 random permutations, and the shard $D$6-value is

$D$7

With $D$8, contamination is reported if any shard is significant (Ahuja et al., 2024).

More task-specific detectors probe robustness rather than direct recall. SPENCE generates 10 paraphrases of each NL2SQL query, ranks them by dependency-tree Tree Edit Distance, and measures execution-accuracy degradation and Kendall’s $D$9 as syntactic divergence increases. Cross-Context Verification solves the same coding benchmark problem in $B$0 independent sessions, computes solution diversity and gold proximity, and uses a contamination score derived from diversity, proximity, and variance; on 9 SWE-bench Verified problems it reported perfect separation between contaminated and genuine reasoning by Mann–Whitney $B$1, $B$2, and $B$3 (Safarzadeh et al., 20 Apr 2026, Song, 23 Mar 2026).

4. Empirical evidence across benchmarks and modalities

In QA-style benchmarks, retrieval-based overlap and slot-guessing produce substantial signals. Using top-10 retrieval and 13-gram ROUGE-L, average maximum overlaps were reported as 35% for MMLU in The Pile and 37% in C4, and 43% for TruthfulQA in The Pile and 34% in C4. Under question-multichoice TS-Guessing, ChatGPT and GPT-4 reached exact-match rates of 0.52 and 0.57 on MMLU test data; for TruthfulQA question-based guessing, exact match without hint was 0.16 for ChatGPT and 0.17 for GPT-4, rising to 0.25 and 0.29 with a URL hint. After fine-tuning ChatGPT on the MMLU test set, TS-Guessing exact match approached 1.00 (Deng et al., 2023).

Multilingual evaluation shows similar breadth. A black-box contamination report on 7 multilingual benchmarks and 7 models found statistically significant contamination in 45 of 49 benchmark–model pairs, with only 4 pairs not significant; every dataset was contaminated in at least five of the seven models. Instruction tuning did not systematically purge contamination, and almost all tested models showed signs of contamination with almost all tested benchmarks (Ahuja et al., 2024).

Time-based benchmark construction yields further evidence that pre-cutoff knowledge inflates performance. AntiLeak-Bench formalizes contamination via facts $B$4 and treats knowledge-cutoff time as a proxy. Its experiments reported that performance on pre-cutoff items is uniformly higher than on post-cutoff items, with $B$5 up to 10–20 points. Post-cutoff generation EM/F1 averages remained under 50% for most open-source LLMs, while GPT-4o reached approximately 77/88 (Wu et al., 2024).

Task-specific studies show domain variation rather than uniform severity. In NL2SQL, SPENCE reported a temporal gradient: Spider had $B$6 with 95% CI $B$7, SParC had $B$8, CoSQL had $B$9, and BIRD had $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$0 with confidence intervals spanning zero. The stylized execution-accuracy degradation on Spider reached about $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$1 to $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$2 for paraphrase ranks 8–10, whereas BIRD showed only $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$3 to $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$4 (Safarzadeh et al., 20 Apr 2026).

Performance-based detection also identifies contamination in popular open models. ConStat reported, for example, a sample-specific contamination effect of $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$5 on GSM8K for Mistral-7B v0.1 with $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$6, and $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$7 on ARC for Llama-3-70B with $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$8. It further reported contamination signals in the top-3 Open LLM Leaderboard models on GSM8K, with $\mathrm{Overlap}(D,B)=\frac{|D \cap B|}{|B|}.$9, $d$0, and $d$1 and sub-1% $d$2-values (Dekoninck et al., 2024).

Watermark-based detection demonstrates that contamination can be statistically detected when it produces measurable gains. In controlled pre-training of 1B models on 10B tokens, ARC-Easy contamination of 4, 8, and 16 benchmark injections yielded $d$3, $d$4, and $d$5 accuracy gains, with corresponding $d$6 values of about 3.0, 5.5, and greater than 12. The same study states that even a 5% performance boost on ARC-Easy yields $d$7, while watermarking preserved benchmark utility with less than 1% absolute accuracy shift on Llama-3 variants (Sander et al., 24 Feb 2025).

5. Prevention, mitigation, and decontamination

The most immediate recommendations concern benchmark stewardship. Proposed practices include releasing only source text while keeping test labels private or behind an evaluation server, building new benchmarks with non-public test splits or dynamic data collection, publishing full training corpora and search/export APIs for open models, disclosing any inclusion of standard benchmarks in fine-tuning or deployment logs for closed models, requiring contamination checks in papers, and maintaining registries such as the LM Contamination Index. Workshop-based coordination and collaborative audits are also explicitly recommended (Sainz et al., 2023).

Practical operational controls have also been proposed. One strategy is to publish public test data only as ciphertext under a public key and a “No Derivatives” license such as CC BY-ND 4.0; another is to require closed APIs to support "exclude_from_training": true flags with cryptographic receipts or signed tokens and append-only audit logs; a third is to release web-context metadata—URL, snapshot, timestamp, DOM or JSON context—for internet-derived benchmarks so that trivially leaky instances can be filtered. These controls are framed as practical for sincere actors rather than adversarially robust guarantees (Jacovi et al., 2023).

A stronger design objective is contamination resistance. One proposal defines a transformation $d$8 to be contamination-resistant with respect to a model $d$9 if it satisfies inference utility and unlearnability, so that training on $m$0 does not improve downstream generalization. The same proposal requires irreversibility, equivalence, and interoperability, and exploits Transformer training–inference asymmetry by releasing only cached key–value pairs, the penultimate hidden state, and the plaintext label, rather than plaintext benchmark inputs (Al-Lawati et al., 19 May 2026).

Automated construction of new, explicitly uncontaminated benchmarks is another line of work. AntiLeak-Bench identifies updated facts from post-cutoff Wikidata claims, retrieves the first Wikipedia revision whose timestamp is at least the update time, verifies mentions of the relevant entities, and constructs single-hop and multi-hop QA samples with and without distractors. It describes itself as fully automated, multilingual, and strictly contamination-free, and is motivated by the claim that merely collecting new data does not guarantee novelty (Wu et al., 2024).

Watermarking benchmarks before release offers a post hoc audit mechanism. Questions are rephrased token by token with a secret-key-conditioned greenlist bias, and later a binomial radioactivity test is applied to the suspect model’s predictions under the null that green-token hits are Bernoulli$m$1. This approach is intended to detect whether the model absorbed the benchmark during training while leaving benchmark utility substantially unchanged (Sander et al., 24 Feb 2025).

Post hoc decontamination includes both benchmark selection and inference-time intervention. JECS formulates multi-model decontamination as joint selection, computes conformal $m$2-values per model, aggregates them by $m$3, fits an envelope to correct the super-uniformity tax, and applies adaptive Storey-BH to produce a shared decontaminated benchmark with asymptotic global contamination rate control. DeconIEP instead preserves the benchmark and perturbs input embeddings at inference time, with bounded $m$4 perturbations guided by a less-contaminated reference model; it reports low residual contamination and minimal benign utility drop, often under 0.05, across MMLU, TruthfulQA, and HumanEval (Liu et al., 20 May 2026, Chai et al., 27 Jan 2026).

6. Limitations, controversies, and open problems

A recurring result is that mitigation strategies often trade off contamination resistance against benchmark fidelity. A controlled evaluation over 10 LLMs, 5 benchmarks, 20 mitigation strategies, and 2 contamination scenarios concluded that no existing strategy significantly improves resistance over the vanilla case across all benchmarks, and none effectively balances fidelity and contamination resistance. In the reported averages, Vanilla had Fidelity $m$5, Resistance $m$6 under mild contamination, and $m$7 under intensive contamination; ITD improved resistance to $m$8 and $m$9 but reduced fidelity to $s \in d$0, while MPA reduced fidelity further to $s \in d$1 (Sun et al., 20 Mar 2025).

Another controversy concerns detector fragility in modern reasoning models. In LRMs, contamination introduced during supervised fine-tuning can initially be identified—reference-free detectors reached AUROC around 73% on six benchmarks and LiRA about 89%—but a short run of GRPO training on clean data collapsed the member/non-member NLL gap and reduced AUROC across all detectors, often to near-random, approximately 50%. When chain-of-thought contamination was applied to advanced LRMs for seven epochs, pass@1 inflated by at least 10%, yet all ten evaluated detectors again collapsed to AUROC around 50% on six benchmarks. The paper attributes the concealment effect to PPO-style importance sampling and clipping (Wang et al., 30 Sep 2025).

This fragility has motivated broader audits of contamination sensitivity and score confidence. In a router–worker audit, noisy benchmark perturbations with parallel routers $s \in d$2 sometimes outperformed the clean-control condition; at $s \in d$3, 10 of 12 models exceeded baseline, and at $s \in d$4 the mean positive excess was 0.086 with net improve $s \in d$5. The interpretation offered is that similar benchmark scores may carry substantially different levels of confidence because perturbation can reactivate contamination-related memory (Song et al., 23 Mar 2026).

A further misconception is that contamination is only a binary property of a benchmark. Know2Guess instead treats contamination risk as metadata with tags $s \in d$6 and reports performance sliced by contamination risk, answerability zone, parser, and prompt variant. Under its official strict parser, Qwen2.5-3B-Instruct achieved overall reliability $s \in d$7 with 95% CI $s \in d$8–$s \in d$9, Zone D abstention $m$0, and refusal $m$1; prompt and parser robustness analyses preserved the main ranking and qualitative conclusions. This suggests that contamination interacts with abstention, refusal, calibration, and parser sensitivity rather than operating as a single scalar alone (Meng et al., 30 Apr 2026).

The field therefore converges on several open directions: contamination-resistant benchmark construction, robust black-box auditing for closed models, evaluation protocols that remain informative after RL and CoT training, shared contamination registries, and contamination-aware reporting that includes effect sizes or adjusted scores rather than raw accuracy alone. The common premise is that benchmark scores remain useful only when the contamination pathway, contamination severity, and uncertainty of the score are made explicit (Sainz et al., 2023, Al-Lawati et al., 19 May 2026, Dekoninck et al., 2024).