---
title: 'CoDeC: Contamination Detection via Context'
url: https://www.emergentmind.com/topics/contamination-detection-via-context-codec
type: topic
---

# CoDeC: Contamination Detection via Context

Contamination Detection via Context (CoDeC) is a statistical methodology for detecting and quantifying data contamination in large language models (LLMs). The approach operationalizes the hypothesis that LLMs exhibit differential behavior when evaluated on in-distribution (potentially memorized) versus out-of-distribution (unseen) datasets under in-context learning (ICL) perturbations. CoDeC is formulated as a model- and dataset-agnostic, gray-box diagnostic tool, providing interpretable contamination scores derived from changes in model confidence under contextual conditioning.

## 1. Theoretical Basis and Motivation

CoDeC is motivated by the observation that LLMs trained on a dataset tend to heavily memorize distributional priors associated with that dataset—vocabulary, structure, and style—resulting in sharp loss minima. When prompted on truly novel data, incorporating a few in-context demonstrations from the same distribution typically boosts the model’s output log-likelihood by providing relevant cues. However, when the queried dataset has already been seen during training, the addition of similar context disrupts memorized patterns and reduces log-likelihood, due to the model’s reliance on specific overfit structures. This directionality friction—context helping on unseen data and hindering on contaminated data—is the critical CoDeC signal [2510.27055].

## 2. Formalization and Algorithm

Let $M$ denote an LLM producing token-level log-probabilities, and $\mathcal{D} = \{x_1, \dots, x_N\}$ be a candidate dataset.

- **Baseline log-likelihood:** For each $x \in \mathcal{D}$, compute
  $$
  \ell_{\rm baseline}(x) = \frac{1}{T} \sum_{t=1}^T \log P_M(x[t] \mid x[<t])
  $$
  where $x[1:T]$ are the tokens in $x$.

- **In-context log-likelihood:** Sample $n$ context examples $x_1', ..., x_n' \in \mathcal{D} \setminus \{x\}$ and compose $z = x_1' \Vert ... \Vert x_n' \Vert x$. Compute
  $$
  \ell_{\rm context}(x) = \frac{1}{T} \sum_{t=1}^T \log P_M(x[t] \mid x_1',...,x_n',\,x[<t]).
  $$

- **Sample-level CoDeC signal:** $\Delta(x) = \ell_{\rm context}(x) - \ell_{\rm baseline}(x)$.

- **Dataset-level contamination score:**
  $$
  S_{\rm CoDeC}(\mathcal{D}) = \frac{1}{N} \sum_{i=1}^N \mathbf{1}[\Delta(x_i) < 0]
  $$
  i.e., the proportion of dataset samples whose in-context log-likelihood decreases.

The protocol involves $n=1$ by default, $N\gtrsim 100$ target samples, randomizing context draws (with $S\geq 1$ seeds for stability), and ignoring the first $\sim$10 target tokens to mitigate context transition artifacts [2510.27055, 2606.03305].

## 3. Experimental Validation and Comparative Results

CoDeC has been evaluated across diverse LLM families (Pythia, GPT-Neo, RWKV, OLMo, Nemotron, Qwen2/3, Llama-Nemotron hybrids), models covering 0.4B–253B parameters, and numerous both seen (in-training) and unseen (post-training) datasets.

Quantitative findings include:

- **Near-perfect separation**: AUC ≈ 99.9% for distinguishing seen and unseen datasets at the benchmark (not split) level. Example scores: for Pythia 1.4B, CoDeC AUC is 99.9%; baselines (vanilla loss, min-K%, Zlib ratio) achieve ≤71.2% [2510.27055].
- **Sensitivity to contamination:** Training on or finetuning with a suspect dataset rapidly pushes $S_{\rm CoDeC}$ above 90% on that data (strong contamination flag).
- **Transfer effect:** Finetuning on a related dataset raises scores on similar benchmarks but not on unrelated corpora.
- **Ablation robustness:** Single in-context example suffices (n=1); increasing n marginally improves discrimination but increases compute.
- **True positive and false positive rates:** High detection accuracy for coarse-grained corpus-level contamination; lower reliability when classifying fine-grained splits (see Section 5).

Table: Sample CoDeC Results ([2606.03305])

| Scenario               | $S_{\rm CoDeC}$ | Detection |
|------------------------|-----------------|-----------|
| Pretraining corpus     | 0.97            | +         |
| Benchmark train split  | 0.91            | +         |
| Benchmark test split   | 0.96            | + (FP)    |
| Unseen benchmark       | 0.55            | –         |

## 4. Interpretability, Thresholds, and Practical Use

$S_{\rm CoDeC} \in [0,1]$ is interpreted as the fraction of “ICL-hurt” samples. Thresholds:
- $>0.8$ = strong evidence of contamination,
- $<0.2$ = likely unseen data,
- $0.2–0.8$ = ambiguous/partial (possibly related or diverse dataset).

Absolute thresholds can be brittle across model size/distribution. Best practice is to compare $S_{\rm CoDeC}$ across multiple models and datasets to identify outlier signals indicative of dataset-specific memorization [2510.27055, 2606.03305].

CoDeC analysis is model- and dataset-agnostic, requiring only logit access for scoring pretokenized prompts—no need for reference/test splits, synthetic data, or specialized classifiers.

## 5. Limitations, Failure Modes, and Reliability Gaps

Subsequent large-scale auditing [2606.03305] demonstrates that CoDeC, while highly effective for coarse-grained, corpus-level contamination, encounters several limitations:

- **Resolution:** Difficulty in reliably distinguishing near-IID train/test splits (high false positives on held-out test).
- **Variance:** Scores vary systematically with model size and training pipeline; fixed thresholds do not adapt well.
- **Edge cases:** Highly duplicated, extremely heterogeneous, or adversarial datasets distort $S_{\rm CoDeC}$ (e.g., a dataset of 1,000 identical samples will not behave as expected).
- **Chat models:** Persistent templating or prompt-specific tokens can induce artifacts.
- **Coarse provenance:** Statistical detection is insufficient for certifying provenance at the single-sample or small-benchmark-split level.

Recommended mitigation strategies include dynamic thresholding calibrated on known pretrain/held-out sets, comparative cross-model analysis, and augmentation with other detection or provenance tools.

## 6. Relation to Alternative and Complementary Techniques

CoDeC belongs to the in-context perturbation family of detectors. In contrast:
- **LLM Dataset Inference** and **Post-Hoc Dataset Inference** require reference or synthesized validation sets and involve statistical testing or classification, posing higher data and computational requirements [2606.03305].
- **ConStat** defines contamination as artificially inflated, non-generalizing benchmark performance and relies on performance differences between original and synthetic/rephrased/cross-benchmark references relative to a suite of reference (presumed uncontaminated) models. It quantifies the contamination effect $\hat \delta$ and supplies p-values, providing robust detection even under paraphrased or reworded contamination [2405.16281].
- **Zero-CoT Probe** (ZCP) deliberately truncates all reasoning during inference and compares zero-CoT performance on original versus structure-preserving reference datasets. ZCP introduces a Bayesian confidence metric quantifying the likelihood and severity of contamination, and is especially resilient to paraphrased or evasive contamination [2605.21856].
- **Cross-Context Verification/Session Isolation** directly observes model generation under randomized, session-isolated trials and leverages behavioral indicators (e.g., diversity in outputs, absence of reasoning structure) to achieve perfect empirical separation in code-generation settings, particularly under the Hierarchical Cross-Context Architecture [2603.21454].

Integration of CoDeC-style features into multimodal settings has been proposed (e.g., for VLMs), potentially via embedding divergence or logit agreement under context perturbation [2511.03774].

## 7. Implications, Best Practices, and Future Directions

CoDeC is a practical, low-cost, and automated “gray-box” contamination detector. It is suitable for large pretraining corpora and benchmarking generalization across model lineages. For high-stakes or leaderboard evaluation:
- Dynamic, cross-model thresholding is advised.
- Score interpretation should be comparative, not absolute.
- Hybrid statistical + provenance-based auditing is essential for split-level certification.
- Resistance to adversarial dataset design and extension to non-causal architectures remain open research areas.

Extensions include tighter membership inference, robustness calibration under distribution shift, augmentation with multi-metric and human-in-the-loop protocols, and adaptation for multimodal and sequence-to-sequence architectures [2510.27055, 2606.03305, 2405.16281, 2605.21856].

In summary, Contamination Detection via Context (CoDeC) is a highly interpretable, model-agnostic tool for robustly identifying data contamination in LLMs at scale, providing actionable insights when integrated into multi-layered benchmark and model auditing pipelines [2510.27055, 2606.03305].

Source: https://www.emergentmind.com/topics/contamination-detection-via-context-codec