Partial-Prompt Contamination Audit

• Partial-prompt contamination audit is the practice of assessing if fragments of evaluation prompts or answers appear during training, thereby inflating benchmark scores.
• Auditing methods include string matching, semantic retrieval, perplexity analysis, and behavioral testing to identify both exact and partial overlaps.
• Empirical findings reveal that even slight contamination can boost performance significantly, underlining the importance of robust auditing and cleaner benchmark design.

Partial-prompt contamination audit is the practice of determining whether a model’s evaluation prompts, answers, or semantically equivalent fragments have entered its training or adaptation pipeline in ways that inflate measured capability. In the recent literature, contamination is not limited to verbatim duplication: it includes text contamination, ground-truth contamination, input-only contamination, input-and-label contamination, and “soft contamination” by semantic duplicates. The central concern is that benchmark scores intended to estimate out-of-distribution generalization may instead reflect partial training on the test, or on close semantic shadows of the test, thereby biasing claims about reasoning, knowledge, or safety (Jiang et al., 2024, Spiesberger et al., 12 Feb 2026, Li et al., 2023).

1. Scope and conceptual boundaries

The audit problem begins with a distinction between forms of overlap. “Text contamination” denotes cases in which only the input text of evaluation samples is present in pre-training, whereas “ground-truth contamination” denotes cases in which the input text plus prompts and/or correct answers or labels are present in the pre-training data. Within this framework, partial-prompt contamination includes situations where only part of a prompt or answer, or only part of an instruction, appears in the corpus (Jiang et al., 2024). A related but operationally distinct taxonomy treats contamination as either input-only or input-and-label: the former means only the question or input is present, while the latter means the question and correct answer co-occur, which is the more severe case because the model can directly memorize the mapping (Li et al., 2023).

Recent work broadens the concept further by introducing “soft contamination,” defined as semantic duplicates that are equivalent or near-equivalent in content but not close in string space. On this view, benchmark integrity is threatened not only by exact copies of a test item, but also by paraphrases, reordered variants, compressed restatements, and benchmark-adjacent examples that induce the same solution pattern. This shift matters because a benchmark can remain “clean” under substring or n-gram filters while still being semantically present in the training distribution (Spiesberger et al., 12 Feb 2026).

The target of an audit is therefore not merely corpus overlap. It is the extent to which benchmark performance is confounded by prior exposure to the prompt, parts of the prompt, the answer, or semantically nearby instances. This suggests that partial-prompt contamination audit is simultaneously a provenance problem, a retrieval problem, and a generalization problem: it asks whether the model has seen something close enough to the evaluation item that benchmark accuracy no longer cleanly measures out-of-distribution ability (Spiesberger et al., 12 Feb 2026, Jiang et al., 2024).

2. Formal overlap criteria and their limitations

Early contamination audits operationalized overlap with simple string-based rules. A document was considered contaminated if any $n$-gram in one of its sentences appeared in the evaluation dataset (Jiang et al., 2024):

$$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$

Two widely cited variants made this more specific. The PaLM definition classified a sample as contaminated if at least $70\%$ of its possible 8-grams appeared in the pre-training set:

$$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$

The Llama 2 definition instead computed a contamination percentage over tokens that belonged to any shared long $n$-gram with $n>10$:

$$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$

These rules are precise, reproducible, and computationally convenient, but the literature now treats them as incomplete proxies rather than sufficient tests of cleanliness. In particular, they privilege exact lexical overlap and contiguous reuse, making them insensitive to paraphrase, prompt fragment reuse, or partial inclusion of prompts and labels (Jiang et al., 2024).

Rule	Detection signal	Main limitation
n-gram overlap	Exact phrase matches	High false positives and false negatives
PaLM	$70\%$ overlap of 8-grams	Misses partial matches and paraphrases
Llama 2	Token coverage in long shared $n$-grams	Misses semantic overlap and many partial matches

The empirical critique is consistent across several studies. N-gram-based methods can produce high false positives because unrelated texts share common short phrases, and high false negatives because simple paraphrasing or partial inclusion escapes detection. Parameters such as $n$, overlap thresholds, and token-coverage buckets materially change what is flagged, sometimes to the point that almost nothing is marked as contaminated (Jiang et al., 2024). Open-web audit pipelines that use METEOR with a threshold of $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$0, an order penalty of $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$1, and a bounded context window reduce some brittleness relative to exact match, but they still remain fundamentally overlap-based rather than fully semantic (Li et al., 2023).

3. Audit methodologies beyond exact matching

A major methodological transition is from lexical matching to semantic retrieval. In a large-scale audit of the Olmo3 training corpus, researchers embedded $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$2 of the base training set and all instruction-tuning data, preserved corpus structure through stratified reservoir sampling, and used the top-ranked MTEB model llama-embed-nemotron-8b to compute cosine similarity against benchmark items. High-similarity pairs were then manually or LLM-annotated as exact, equivalent, subset, superset, or unrelated, and these annotations were used to train scalable duplication classifiers with validation accuracy of about $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$3 (Spiesberger et al., 12 Feb 2026). This workflow is explicitly motivated by the failure of n-gram filters to detect semantic duplicates.

A second family of methods attempts contamination estimation without access to the training corpus. Perplexity-based auditing compares a benchmark’s log-perplexity to matched memorized and clean baselines. If the benchmark lies close to the memorized baseline, the model is interpreted as assigning anomalously high probability to material it likely encountered during training. The method can also be applied to the input sequence alone, which makes it relevant to partial-prompt contamination even when labels are not available (Li, 2023).

A third family uses statistical membership inference. Recent evaluations compare three paradigms: LLM Dataset Inference, Post-Hoc Dataset Inference, and CoDeC. LLM Dataset Inference aggregates weak per-sample membership features and applies a one-sided $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$4-test with detection declared at $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$5; Post-Hoc Dataset Inference attempts to synthesize a held-out set to reduce dependence on a naturally IID validation set; CoDeC measures whether in-context examples from the same dataset reduce next-token confidence, summarizing the effect with a dataset score

$$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$6

These methods target training-data membership statistically rather than by direct corpus matching (Zarzecki et al., 2 Jun 2026).

A fourth line of work uses behavioral rather than textual evidence. Cross-Context Verification (CCV) addresses coding benchmarks by solving the same problem in multiple session-isolated trials and measuring solution diversity. The premise is that memorized solutions recur with low diversity, whereas genuine reasoning produces varied outputs even at temperature $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$7. CCV combines pairwise AST similarity, line-level BLEU, and character edit distance into a diversity measure, then forms a contamination score

$$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$8

with higher values indicating contamination. Its companion Hierarchical Cross-Context Architecture (HCCA) distributes analysis across independent roles to prevent confirmation bias through intentional information restriction (Song, 23 Mar 2026).

Benchmark construction itself has also been redesigned as an audit instrument. Know2Guess attaches contamination-risk metadata to each item and partitions evaluation into zones: high-popularity public items, lower-popularity public items, transformed public items, and synthetic unknowns. Zone C, consisting of transformed public questions produced by paraphrasing, compression, and reordering, is specifically intended to challenge simple string-match memorization and thereby audit partial or approximate prompt contamination (Meng et al., 30 Apr 2026).

4. Empirical prevalence and effects on benchmark scores

The strongest recent evidence indicates that soft and partial contamination are common rather than exceptional. In the Olmo3 audit, after n-gram-based decontamination, semantic duplicates were still found for $$\exists n,~ \exists\, \text{n-gram}~g \in s,~~ g \in \text{EvalData}.$$9 of CodeForces problems, with $70\%$0 having at least one semantic duplicate among the top-100 cosine-similarity matches; MBPP showed semantic duplicates for $70\%$1 of programming problems; ZebraLogic contained exact duplicates for $70\%$2 of problems, with additional semantic duplicates beyond that. The study argues that these estimates exceed past literature largely because the search was more exhaustive and the corpora much larger (Spiesberger et al., 12 Feb 2026).

Performance effects are correspondingly large. Fine-tuning on semantic duplicates of benchmark data can raise accuracy by roughly $70\%$3 to $70\%$4 percentage points, often matching the effect size of fine-tuning on exact duplicates. On MuSR, a baseline of about $70\%$5 rose to about $70\%$6–$70\%$7 after fine-tuning on either semantic or exact duplicates, with gains on both the duplicated half and the supposedly unseen half of the same benchmark. On MBPP, a baseline of $70\%$8 rose to $70\%$9 on seen items and $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$0 on unseen items after semantic-duplicate fine-tuning; exact-duplicate fine-tuning yielded $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$1 on seen items and $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$2 on unseen items. ZebraLogic was more case-specific: exact duplicates improved a $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$3 baseline to $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$4 on seen items and $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$5 on unseen items, whereas semantic duplicates had negligible or negative effect (Spiesberger et al., 12 Feb 2026).

The ecological version of the result is especially important for auditing. When MuSR fine-tuning used a mix that reflected the natural occurrence rate of benchmark-matching semantic duplicates, only $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$6 of the training data consisted of semantic duplicates, yet accuracy still increased by $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$7 on the seen subset and by $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$8 on the unseen subset. The same intervention produced no gain on TrueDetective, an out-of-distribution sibling benchmark. This suggests shallow, in-benchmark generalization rather than genuine out-of-distribution capability growth (Spiesberger et al., 12 Feb 2026).

Controlled pre-training studies reach compatible conclusions. Even a single exposure of text, prompt, or answer can markedly increase downstream accuracy, F1, or ROUGE, and the improvement is usually greatest for ground-truth contamination. Repetition produces a U-shaped curve: performance improves up to a point, but excessive repetition can reduce downstream performance, in some datasets even below the uncontaminated baseline. Removal experiments are equally revealing: filtering documents under n-gram or Llama 2 rules does not consistently reduce performance, implying that these rules fail to capture the effective contamination that matters behaviorally (Jiang et al., 2024).

Open-source web-scale audits show that contamination rates can range from $$\frac{|\{g \in \text{8-grams}(x) ~|~ g \in \text{Pretrain}\}|}{|\text{8-grams}(x)|} \geq 70\%.$$9 to $n$0 across popular multiple-choice benchmarks, and that performance inflation is heterogeneous. Accuracy gains of up to $n$1 on C-Eval, $n$2 on ARC, and $n$3 on HellaSwag were observed on contaminated splits, while MMLU showed only minimal increase. Larger models appeared able to gain more advantage than smaller models on contaminated test sets (Li et al., 2023). A plausible implication is that raw benchmark improvement cannot be interpreted uniformly across tasks or scales without contamination analysis.

5. Reliability gaps, false positives, and evidentiary standards

Partial-prompt contamination audit is itself error-prone. A large evaluation across $n$4 models and $n$5 detection challenges found that only $n$6 outcomes were correct. The identified failure modes were distribution shift and scale constraints. LLM Dataset Inference produced false positives when suspect and validation sets were not IID; Post-Hoc Dataset Inference became underpowered when benchmarks were too small to support effective synthetic generator training; CoDeC provided only coarse provenance signals that could separate trained-on corpora from evaluation-only corpora at a broad level but could not verify individual benchmark splits (Zarzecki et al., 2 Jun 2026). The resulting conclusion was that transparent data provenance cannot yet be replaced by statistical detection.

Behavioral auditing complicates the picture further by showing that some static contamination labels are themselves unreliable. On nine SWE-bench Verified problems, CCV reported perfect separation between contaminated and genuine reasoning in its sample, with Mann–Whitney $n$7, $n$8, and effect size $n$9. The same study found that $n>10$0 of prior contamination labels were false positives and argued that contamination was binary in its sample: models either recalled perfectly or not at all. It also reported that reasoning absence was a perfect discriminator on these trials, and that information restriction in HCCA, rather than mere multi-stage review, was the mechanism that reduced sycophantic confirmation (Song, 23 Mar 2026). This does not invalidate static text audits, but it does show that artifact-based or prompt-only labels can overstate contamination.

A related line of work argues that benchmark scores should be accompanied by contamination-sensitivity analysis rather than reported as if all scores had equal evidentiary status. In a router–worker framework, problems were deleted, rewritten, and perturbed before being passed downstream. For a genuinely clean benchmark, noisy conditions should not systematically outperform a clean-control baseline. Yet above-baseline gains were widespread and heterogeneous: at router count $n>10$1, $n>10$2 models violated the clean baseline, with mean positive excess $n>10$3; at $n>10$4, mean positive excess rose to $n>10$5, and improve transitions exceeded degrade transitions by $n>10$6 to $n>10$7 (Song et al., 23 Mar 2026). This suggests that semantically fragmented cues can be reassembled by models in ways that reactivate contamination-related memory even when full prompt overlap has been disrupted.

Taken together, these findings support a demanding evidentiary standard. A credible audit should specify its contamination definition, report thresholds and failure modes, distinguish within-benchmark transfer from cross-benchmark generalization, and treat statistical detectors as suggestive rather than definitive when provenance is unavailable (Zarzecki et al., 2 Jun 2026, Song et al., 23 Mar 2026).

6. Extensions, benchmark design, and broader audit surfaces

Recent benchmark design work makes contamination risk an explicit variable rather than a post-hoc caveat. Know2Guess defines each item as

$n>10$8

where $n>10$9 is a contamination-risk tag in $$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$0. The benchmark contains $$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$1 items across five domains, with $$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$2 low-risk, $$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$3 medium-risk, and $$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$4 high-risk items, and all synthetic unknowns in Zone D are low-risk by design. Its primary reliability metric jointly rewards correct answering on answer-expected items and abstention on abstention-expected items:

$$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$5

The authors describe Zone C performance on transformed public items as the “smoking gun” for partial-prompt contamination, because high performance there can indicate either strong generalization or semantic contamination that survives paraphrase and reordering (Meng et al., 30 Apr 2026).

Audits have also expanded beyond text-only LLM benchmarks. In medical vision-language benchmarks, image-side near-neighbour overlap, canonical-order exchangeability, cohort-relative Min-K%++ tail enrichment, and cross-model top-$$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$6 overlap were applied to SLAKE-En, PathVQA, VQA-RAD, and a public OmniMedVQA mirror. Image-side source overlap on SLAKE-En reached $$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$7 under SigLIP-B-16 and $$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$8 under SigLIP-SO400M, but manual adjudication interpreted the matches as same-modality, same-projection images from different patients rather than verified duplicates. On the text side, only Qwen2.5-VL on SLAKE-En showed a canonical-order exchangeability signal that survived ordering ablation and external baselines. By contrast, cohort-relative Min-K%++ and cross-model top-$$\text{Contam\%}(e) = \frac{\text{Number of contaminated tokens in }e}{\text{Total tokens in }e}.$$9 overlap collapsed when BLIP-2 was added as an external pre-domain baseline, showing that these detectors are unreliable as standalone membership-inference signals in small medical-VLM cohorts (Xu et al., 8 Jun 2026).

The audit surface extends into deployment infrastructure. CacheProbe examined gateway prompt caching with $70\%$0-token prompts sharing a $70\%$1 prefix and found cross-account cache sharing in OpenRouter default mode, whereas direct provider access preserved isolation and OpenRouter BYOK restored it. Detection used both Kolmogorov–Smirnov tests on time-to-first-token with a threshold of $70\%$2 and explicit metadata such as cached_tokens. This reframes partial-prompt contamination as an inference-time privacy and provenance problem: a shared prefix can leak the existence of another user’s prompt even when model training data are untouched (Fahey, 28 May 2026).

A further extension concerns latent fine-tuning contamination. CANARY compares hidden states from a base and a fine-tuned model over an unlabeled prompt set, projects the difference through a Sparse Autoencoder, and scores prompts by the norm of the SAE-filtered shift. It reports AUROC $70\%$3 at $70\%$4 contamination, with $70\%$5 confidence interval $70\%$6, zero false positives on benign fine-tuning, and prompt-level ranking that yields $70\%$7 red-teaming lift (Parekh, 1 Jun 2026). This suggests that prompt-based contamination audit is no longer restricted to corpus overlap; it can also operate on hidden-state geometry when harmful behavior remains dormant in output space.

Across these extensions, the common pattern is that contamination audit has moved from exact duplicate detection toward multi-signal evaluation: provenance metadata, semantic search, behavioral diversity, perturbation sensitivity, negative controls, manual adjudication, and, increasingly, representation-level analysis. The shared conclusion is narrower than a blanket rejection of benchmarks. Benchmarks remain useful, but their scores are interpretable only when partial-prompt contamination has been audited with methods matched to the modality, scale, and training opacity of the system under study (Spiesberger et al., 12 Feb 2026, Zarzecki et al., 2 Jun 2026).