---
title: N-gram Contamination in Language Models
url: https://www.emergentmind.com/topics/n-gram-based-contamination-definitions
type: topic
---

# N-gram Contamination in Language Models

N-gram-based contamination definitions refer to a class of formal criteria for identifying when entries from evaluation sets (benchmarks) are present verbatim or near-verbatim in the pre-training corpus of language models. Such overlap directly threatens the validity of performance claims for modern language models (LMs) by confounding generalization with memorization. Although the field has moved toward more sophisticated, semantic-aware metrics, n-gram-based methods remain foundational for contamination auditing due to their formal simplicity, scalability, and interpretability.

## 1. Formal Definitions of N-gram-based Contamination

The core construct of n-gram-based contamination is the set of all contiguous subsequences of length $n$ (n-grams) extracted from a token sequence. Formally, for a sequence $X = (t_1, ..., t_{|X|})$ and integer $n$, the n-grams are
$$
\mathrm{NG}(X, n) = \{ (t_i, ..., t_{i+n-1}) : 1 \leq i \leq |X| - n + 1 \}
$$

Given an evaluation example $E$ and corpus document $d$, two standard overlap metrics are:

- **N-gram Precision:**  
  $$
  \mathrm{prec}_n(d, E) = \frac{|\mathrm{NG}(d, n) \cap \mathrm{NG}(E, n)|}{|\mathrm{NG}(E, n)|}
  $$
- **Jaccard Index:**  
  $$
  \mathrm{Jacc}_n(d, E) = \frac{|\mathrm{NG}(d, n) \cap \mathrm{NG}(E, n)|}{|\mathrm{NG}(d, n) \cup \mathrm{NG}(E, n)|}
  $$

Entry $E$ is typically labeled "contaminated" with respect to $d$ if the overlap metric exceeds a threshold $\lambda$.

Several prominent formalizations have been adopted in practice:
- **Direct n-gram overlap:** Contaminated if $\mathrm{NG}(d, n) \cap \mathrm{NG}(E, n) \neq \varnothing$.
- **PaLM criterion:** $n=8$, $\lambda=0.70$; $E$ is contaminated if at least $70\%$ of its 8-grams are found in any corpus document.
- **Llama 2 criterion:** $n>10$ (typically $n=15$), slide a window across $E$, labeling tokens as contaminated if they belong to any matching $n$-gram; define contamination percentage per sample and use thresholds to classify as Clean, Not Clean, Not Dirty, or Dirty [2401.06059].

Another common instantiation is the per-entry overlap rate:
$$
\eta_n(x; C) = \frac{1}{|G_n(x)|} \sum_{g \in G_n(x)} \mathbb{1}[\mathrm{count}_C(g) > 0]
$$
where $G_n(x)$ is the set of n-grams in $x$ and $\mathrm{count}_C(g)$ is the frequency of $g$ in the corpus $C$ [2506.12229].

## 2. Practical Contamination Metrics and Classification

Pipelines for measuring n-gram contamination in large benchmarks and corpora typically implement:

| Metric               | Definition                                                                   | Usage Example                         |
|----------------------|------------------------------------------------------------------------------|---------------------------------------|
| Union (“match”)      | $\frac{|\mathcal{N}_n(E) \cap \mathcal{N}_n^\tau(\mathcal{C})|}{|\mathcal{N}_n(E)|}$ | [2411.03923]                          |
| Tokenwise (“chunk”)  | Fraction of tokens in $E$ in any matching n-gram                             | Llama 2 contamination percentage      |
| Longest-substring    | $M(E)/|E|$ with $M(E)$ the longest substring (in tokens) in corpus           | [2411.03923], recommended by ConTAM   |
| Character n-gram     | As above, but over raw bytes or characters                                   | Infini-gram mini [2506.12229]         |

Classification is made by applying thresholds to these scores. For example:
- **Infini-gram mini:** $n=50$ characters; entries classified as Clean ($\eta < 0.2$), Suspicious ($0.2 \leq \eta < 0.8$), Dirty ($\eta \geq 0.8$) [2506.12229].
- **Llama 2:** Clean if contamination percentage $\leq 0.75$, Dirty if above $0.85$ [2401.06059].
- **ConTAM:** Thresholds empirically chosen per model/benchmark via maximizing estimated performance gain (EPG), not fixed a priori [2411.03923].

## 3. Algorithmic Approaches and Hyperparameter Sensitivity

Computing n-gram overlap at internet scale relies on compressed, index-based search. Infini-gram mini uses FM-indexes to allow substring search on petabyte-scale corpora, extracting overlapping n-grams with fine stride for sensitivity [2506.12229].

Key hyperparameters affecting both sensitivity and specificity:
- **n-gram length ($n$):** Small $n$ increases recall but raises false positives (spurious matches); large $n$ ensures precision but induces false negatives due to minor edits or paraphrases.
- **Frequency threshold ($\tau$):** Imposing $\tau > 1$ (i.e., discarding rare n-grams) can suppress noise but increases false negatives [2411.03923].
- **Skip budget ($k$):** Allowing mismatches (e.g., token substitutions) along the substring increases resilience to minor edits but shows marginal practical value [2411.03923].

An empirically robust configuration is $n=8$ tokens, $\tau=1$, skip budget $k=0$; longer n-grams or stricter frequency thresholds systematically reduce the measured contamination rate but risk missing "memorized" examples.

## 4. Limitations and Failure Modes

N-gram-based contamination metrics exhibit several well-documented limitations:

- **Susceptibility to paraphrase:** Any non-exact rewording evades detection, leading to high false negative rates. Embedding-based or syntax-based measures are required to capture such soft contamination [2401.06059, 2602.12413].
- **False positives due to generic content:** Short n-grams are likely to be matched by chance, especially in high-volume web text, inflating the contamination estimate with semantically unrelated passages.
- **Lack of context sensitivity:** Mere token sequence overlap does not distinguish between semantic alignment and coincidental reuse (e.g., "bank account" versus "river bank").
- **Threshold arbitrariness:** Different choices of $n$ or contamination percentage threshold $\lambda$ can substantially alter the fraction of test items labeled contaminated, with little impact on actual measured generalization [2401.06059].
- **Ground-truth blindness:** N-gram rules ignore leakage of gold outputs (answers/labels), which can be memorized and significantly boost apparent model capability even when inputs do not overlap textually.

Consequently, n-gram-based statistics provide an upper bound on surface-level leakage yet do not guarantee detection of all harmful forms of memorization.

## 5. Empirical Findings and Benchmark Analyses

Systematic large-scale contamination audits reveal variable but often substantial test overlap in public corpora:

| Benchmark   | Corpus         | Dirty Rate (n-gram, typical n)         | Reference         |
|-------------|----------------|----------------------------------------|-------------------|
| ZebraLogic  | Olmo3          | 49.5% (n=13, exact 13-gram)            | [2602.12413]      |
| CodeForces  | Olmo3          | 77.5% (semantic + exact)               | [2602.12413]      |
| SQuAD       | DCLM-baseline  | 40.1% (n=50 char, dirty $\geq$ 0.8)    | [2506.12229]      |
| MMLU        | DCLM-baseline  | 27.7%                                  | [2506.12229]      |
| MBPP        | Olmo3          | 100% (semantic, no exact)              | [2602.12413]      |

Infini-gram mini found that reading-comprehension and commonsense benchmarks (e.g., SQuAD, ARC) regularly exceed 30–40% dirty rate in large, web-scale corpora; knowledge-reasoning and code benchmarks exhibit variable rates [2506.12229].

Multiple studies report that filtering large fractions of entries flagged by n-gram overlap alters downstream model performance only minimally, evidencing the low specificity of these filters for actual memorization [2401.06059]. However, high dirty rates raise strong concerns about the inflation of benchmark scores in evaluations, emphasizing the need for de-duplication and the development of more resilient benchmarks.

## 6. Methodological Extensions and Alternatives

A clear trend is the development of complementary or alternative contamination definitions:

- **Semantic contamination** (a.k.a. soft contamination): Entries are flagged as contaminated if a semantic encoding (e.g., via sentence embeddings) achieves cosine similarity $\geq \theta_\mathrm{soft}$ with any pre-training example [2602.12413].
- **Substring edit distance:** Certain studies replace n-gram overlap with substring-level Levenshtein similarity or AST k-gram metrics, especially for code benchmarks, to capture near-duplicates [2403.04811].
- **Performance-grounded metrics (ConTAM):** The contamination threshold is selected not by surface heuristics, but by the empirical increase in model accuracy when contaminated examples are included, providing an effect-size-guided method for threshold optimization [2411.03923].

These methods preserve interpretability while improving robustness to superficial text alterations and aligning contamination designations more closely with observed model behavior.

## 7. Best Practices and Future Directions

Current best-practice recommendations to ensure rigorous contamination analysis include:

- Prefer the longest-match score (i.e., the normalized length of the longest matching substring) as a primary metric, with n-gram union as a sanity check [2411.03923].
- Hyperparameters: use n=8 tokens, frequency filter $\tau=1$, skip budget zero.
- Empirically determine contamination thresholds by profiling the actual performance gain for flagged items (ConTAM), not by arbitrary or inherited convention.
- Manually review flagged/unflagged samples near the chosen threshold.
- Supplement surface-level overlap with semantic or editing-robust similarity to capture paraphrased leaks [2602.12413].
- Proactively de-duplicate or sanitize both training and evaluation sets using scalable substring search (e.g., FM-indexing) when working with large web-scale corpora [2506.12229].

These combined approaches form the current methodological backbone for contamination detection and reporting in language model pretraining and evaluation.

Source: https://www.emergentmind.com/topics/n-gram-based-contamination-definitions