CoDeC: Contamination Detection via Context

• The paper introduces a statistical method where shifts in model confidence indicate contamination by comparing in-context and baseline log-likelihood scores.
• CoDeC operationalizes in-context learning perturbations to generate interpretable contamination scores with near-perfect AUC metrics across various LLMs.
• The methodology is model- and dataset-agnostic, providing actionable insights and dynamic thresholding guidelines for robust contamination auditing.

Contamination Detection via Context (CoDeC) is a statistical methodology for detecting and quantifying data contamination in LLMs. The approach operationalizes the hypothesis that LLMs exhibit differential behavior when evaluated on in-distribution (potentially memorized) versus out-of-distribution (unseen) datasets under in-context learning (ICL) perturbations. CoDeC is formulated as a model- and dataset-agnostic, gray-box diagnostic tool, providing interpretable contamination scores derived from changes in model confidence under contextual conditioning.

1. Theoretical Basis and Motivation

CoDeC is motivated by the observation that LLMs trained on a dataset tend to heavily memorize distributional priors associated with that dataset—vocabulary, structure, and style—resulting in sharp loss minima. When prompted on truly novel data, incorporating a few in-context demonstrations from the same distribution typically boosts the model’s output log-likelihood by providing relevant cues. However, when the queried dataset has already been seen during training, the addition of similar context disrupts memorized patterns and reduces log-likelihood, due to the model’s reliance on specific overfit structures. This directionality friction—context helping on unseen data and hindering on contaminated data—is the critical CoDeC signal (Zawalski et al., 30 Oct 2025).

2. Formalization and Algorithm

Let $M$ denote an LLM producing token-level log-probabilities, and $\mathcal{D} = \{x_1, \dots, x_N\}$ be a candidate dataset.

• Baseline log-likelihood: For each $x \in \mathcal{D}$, compute

$$\ell_{\rm baseline}(x) = \frac{1}{T} \sum_{t=1}^T \log P_M(x[t] \mid x[<t])$$

where $x[1:T]$ are the tokens in $x$.

• In-context log-likelihood: Sample $n$ context examples $x_1', ..., x_n' \in \mathcal{D} \setminus \{x\}$ and compose $z = x_1' \Vert ... \Vert x_n' \Vert x$. Compute

$$\ell_{\rm context}(x) = \frac{1}{T} \sum_{t=1}^T \log P_M(x[t] \mid x_1',...,x_n',\,x[<t]).$$

• Sample-level CoDeC signal: $\mathcal{D} = \{x_1, \dots, x_N\}$0.
• Dataset-level contamination score:

$\mathcal{D} = \{x_1, \dots, x_N\}$1

i.e., the proportion of dataset samples whose in-context log-likelihood decreases.

The protocol involves $\mathcal{D} = \{x_1, \dots, x_N\}$2 by default, $\mathcal{D} = \{x_1, \dots, x_N\}$3 target samples, randomizing context draws (with $\mathcal{D} = \{x_1, \dots, x_N\}$4 seeds for stability), and ignoring the first $\mathcal{D} = \{x_1, \dots, x_N\}$510 target tokens to mitigate context transition artifacts (Zawalski et al., 30 Oct 2025, Zarzecki et al., 2 Jun 2026).

3. Experimental Validation and Comparative Results

CoDeC has been evaluated across diverse LLM families (Pythia, GPT-Neo, RWKV, OLMo, Nemotron, Qwen2/3, Llama-Nemotron hybrids), models covering 0.4B–253B parameters, and numerous both seen (in-training) and unseen (post-training) datasets.

Quantitative findings include:

• Near-perfect separation: AUC ≈ 99.9% for distinguishing seen and unseen datasets at the benchmark (not split) level. Example scores: for Pythia 1.4B, CoDeC AUC is 99.9%; baselines (vanilla loss, min-K%, Zlib ratio) achieve ≤71.2% (Zawalski et al., 30 Oct 2025).
• Sensitivity to contamination: Training on or finetuning with a suspect dataset rapidly pushes $\mathcal{D} = \{x_1, \dots, x_N\}$6 above 90% on that data (strong contamination flag).
• Transfer effect: Finetuning on a related dataset raises scores on similar benchmarks but not on unrelated corpora.
• Ablation robustness: Single in-context example suffices (n=1); increasing n marginally improves discrimination but increases compute.
• True positive and false positive rates: High detection accuracy for coarse-grained corpus-level contamination; lower reliability when classifying fine-grained splits (see Section 5).

Table: Sample CoDeC Results (Zarzecki et al., 2 Jun 2026)

Scenario	$\mathcal{D} = \{x_1, \dots, x_N\}$7	Detection
Pretraining corpus	0.97	+
Benchmark train split	0.91	+
Benchmark test split	0.96	+ (FP)
Unseen benchmark	0.55	–

4. Interpretability, Thresholds, and Practical Use

$\mathcal{D} = \{x_1, \dots, x_N\}$8 is interpreted as the fraction of “ICL-hurt” samples. Thresholds:

• $\mathcal{D} = \{x_1, \dots, x_N\}$9 = strong evidence of contamination,
• $x \in \mathcal{D}$0 = likely unseen data,
• $x \in \mathcal{D}$1 = ambiguous/partial (possibly related or diverse dataset).

Absolute thresholds can be brittle across model size/distribution. Best practice is to compare $x \in \mathcal{D}$2 across multiple models and datasets to identify outlier signals indicative of dataset-specific memorization (Zawalski et al., 30 Oct 2025, Zarzecki et al., 2 Jun 2026).

CoDeC analysis is model- and dataset-agnostic, requiring only logit access for scoring pretokenized prompts—no need for reference/test splits, synthetic data, or specialized classifiers.

5. Limitations, Failure Modes, and Reliability Gaps

Subsequent large-scale auditing (Zarzecki et al., 2 Jun 2026) demonstrates that CoDeC, while highly effective for coarse-grained, corpus-level contamination, encounters several limitations:

• Resolution: Difficulty in reliably distinguishing near-IID train/test splits (high false positives on held-out test).
• Variance: Scores vary systematically with model size and training pipeline; fixed thresholds do not adapt well.
• Edge cases: Highly duplicated, extremely heterogeneous, or adversarial datasets distort $x \in \mathcal{D}$3 (e.g., a dataset of 1,000 identical samples will not behave as expected).
• Chat models: Persistent templating or prompt-specific tokens can induce artifacts.
• Coarse provenance: Statistical detection is insufficient for certifying provenance at the single-sample or small-benchmark-split level.

Recommended mitigation strategies include dynamic thresholding calibrated on known pretrain/held-out sets, comparative cross-model analysis, and augmentation with other detection or provenance tools.

6. Relation to Alternative and Complementary Techniques

CoDeC belongs to the in-context perturbation family of detectors. In contrast:

• LLM Dataset Inference and Post-Hoc Dataset Inference require reference or synthesized validation sets and involve statistical testing or classification, posing higher data and computational requirements (Zarzecki et al., 2 Jun 2026).
• ConStat defines contamination as artificially inflated, non-generalizing benchmark performance and relies on performance differences between original and synthetic/rephrased/cross-benchmark references relative to a suite of reference (presumed uncontaminated) models. It quantifies the contamination effect $x \in \mathcal{D}$4 and supplies p-values, providing robust detection even under paraphrased or reworded contamination (Dekoninck et al., 2024).
• Zero-CoT Probe (ZCP) deliberately truncates all reasoning during inference and compares zero-CoT performance on original versus structure-preserving reference datasets. ZCP introduces a Bayesian confidence metric quantifying the likelihood and severity of contamination, and is especially resilient to paraphrased or evasive contamination (Lan et al., 21 May 2026).
• Cross-Context Verification/Session Isolation directly observes model generation under randomized, session-isolated trials and leverages behavioral indicators (e.g., diversity in outputs, absence of reasoning structure) to achieve perfect empirical separation in code-generation settings, particularly under the Hierarchical Cross-Context Architecture (Song, 23 Mar 2026).

Integration of CoDeC-style features into multimodal settings has been proposed (e.g., for VLMs), potentially via embedding divergence or logit agreement under context perturbation (Park et al., 5 Nov 2025).

7. Implications, Best Practices, and Future Directions

CoDeC is a practical, low-cost, and automated “gray-box” contamination detector. It is suitable for large pretraining corpora and benchmarking generalization across model lineages. For high-stakes or leaderboard evaluation:

• Dynamic, cross-model thresholding is advised.
• Score interpretation should be comparative, not absolute.
• Hybrid statistical + provenance-based auditing is essential for split-level certification.
• Resistance to adversarial dataset design and extension to non-causal architectures remain open research areas.

Extensions include tighter membership inference, robustness calibration under distribution shift, augmentation with multi-metric and human-in-the-loop protocols, and adaptation for multimodal and sequence-to-sequence architectures (Zawalski et al., 30 Oct 2025, Zarzecki et al., 2 Jun 2026, Dekoninck et al., 2024, Lan et al., 21 May 2026).

In summary, Contamination Detection via Context (CoDeC) is a highly interpretable, model-agnostic tool for robustly identifying data contamination in LLMs at scale, providing actionable insights when integrated into multi-layered benchmark and model auditing pipelines (Zawalski et al., 30 Oct 2025, Zarzecki et al., 2 Jun 2026).