---
title: Translation-Aware Contamination Detection
url: https://www.emergentmind.com/topics/translation-aware-contamination-detection
type: topic
---

# Translation-Aware Contamination Detection

Translation-Aware Contamination Detection refers to a class of contamination detection methodologies that explicitly account for the possibility of benchmark data leakage through translations, rather than only considering surface-level overlap in the source language. This is of critical importance in multilingual large language model (LLM) evaluation, as translations of benchmarks can enter model pre-training corpora, thereby artificially inflating performance on the original task in the source language, while escaping traditional detection techniques focused on monolingual or surface-form overlap. Recent studies have demonstrated both the existence of cross-lingual contamination effects and the necessity of translation-aware detection protocols to ensure rigorous, fair, and reproducible assessment of LLM generalization [2406.13236][2601.14994].

## 1. Foundations: Cross-Lingual Contamination and Its Formalization

Data contamination occurs when evaluation items or close variants are present in a model’s training data, thereby undermining claims of generalization. Traditional detection focuses on direct overlap, e.g., n-gram or token matching. In the translation-aware context, the contamination indicator is revised to account for translations:

\[
\mathrm{cont}(x) = 1\left[\exists\,\ell\in\mathcal{L} : T_\ell(x) \in \mathcal{S}_{\mathrm{train}}\right]
\]

Here, $x$ is a benchmark instance, $T_\ell(x)$ its translation into language $\ell$, $\mathcal{L}$ the set of languages considered, and $\mathcal{S}_{\mathrm{train}}$ the (possibly opaque) training corpus of the model [2601.14994]. Under this threat model, "cross-lingual contamination" is said to occur when a model is exposed to a translated version of the evaluation data, but not the original, allowing memorization of answers without direct English overlap [2406.13236]. Empirical injection of such contamination, for instance by fine-tuning LLMs on translated test sets in French or Arabic, results in measurable performance gains on the original English benchmarks despite the model never seeing the English form [2601.14994].

## 2. Failure Modes of Traditional Contamination Detection

Conventional detection techniques are based on surface similarity between evaluation and training (n-gram overlap, shared-likelihood, semantic matching). For cross-lingual contamination, these probes are fundamentally inadequate:

- **N-Gram Overlap:** For T^ℓ contamination (T^ℓ being the translated set), n-gram recall between training and evaluation in the source language remains at "clean-model" baseline levels, even as performance is spuriously inflated [2406.13236].
- **Likelihood-Based Permutation Tests:** Shared-likelihood p-values for models contaminated on translated benchmarks show no significant reduction, failing to flag contamination [2406.13236].
- **Behavioral and Distributional Probes:** Methods such as Tested Slot Guessing (TS-Guessing), Index Recall Rate (IDR), and Min-K% probability analysis exhibit dramatic suppression of signal when the model is contaminated via translation. For instance, on Mistral-7B, IDR in English collapses to baseline even with 100% contamination through Arabic, and Min-K% AUROC values indicate near-chance detection [2601.14994].

These failures arise because translated contamination evades surface-form matching and disrupts positional or probabilistic cues commonly leveraged in monolingual contamination detection.

## 3. Translation-Aware Contamination Detection Methodologies

Translation-Aware Contamination Detection (TACD) seeks to recover hidden contamination signals by leveraging multilingual evaluation and cross-variant perturbations:

- **Evaluation Views:** For each instance $x_j$, TACD generates translations $x_j^\ell = T_\ell(x_j)$ for all $\ell \in \mathcal{L}$ (e.g., EN, AR, FR). For every language, a random permutation of the answer choices is applied, yielding perturbed views $\tilde{x}_j^\ell$ [2601.14994].
- **Model Querying and Signal Extraction:**
    - **IDR (Index Recall Rate):** Proportion where the model correctly recalls the original answer’s index under randomized choice orderings and across all languages. Under pure generalization, this value should be near uniform guessing, $1/K$ for $K$-way multiple choice.
    - **CLC (Cross-Lingual Consistency):** The fraction of examples where predictions agree across all tested languages. For $L$ languages and $K$ answer choices, the baseline is $1/K^{L-1}$ under independent guessing.
- **Statistical Thresholding:** Contamination is flagged if either signal significantly exceeds its baseline, as determined by binomial tests or thresholds calibrated on clean data.

The TACD algorithm can be summarized as follows:

1. For each evaluation item and each of $L$ languages, compute a translation and randomize answer order.
2. Query the model on each variant, recording both the predicted index per view and prediction agreement across views.
3. Aggregate IDR and CLC scores; flag contamination if these diverge from expectation under independence [2601.14994].

## 4. Empirical Results: Effectiveness and Limitations

TACD has been empirically validated across various backbone models (LLaMA3, Gemma, Qwen, Mistral) and benchmarks (MMLU, ARC-Challenge, MathQA, XQuAD), with contamination introduced via multiple languages (French, Arabic, Spanish, etc.):

- **Recovery of Hidden Signals:** While English-only detection probes fail (IDR, Min-K% both near chance), TACD's cross-lingual CLC yields high AUROC (>0.9) for distinguishing contaminated from non-contaminated models [2601.14994].
- **Cross-Lingual Consistency Metric:** For models contaminated on Arabic translations, CLC climbs far above baseline as contamination proportion increases (e.g., for Gemma-3-1B-it, CLC goes from 0.278 to 0.634 across p=0→100%). Qwen3-1.7B exhibits CLC=1.00 at all p, indicating degenerate collapse observed only via TACD [2601.14994].
- **Generalization Benchmark Approach:** The generalization-gap metric, Δ = $f(T^g) - f(T)$, where $T^g$ is a version of the test set with all distractor choices replaced by other correct answers, sharply distinguishes genuine generalization from contamination-induced memorization. Clean models show large $Δ ≫ 0$, while contaminated models exhibit $Δ ≈ 0$ or $Δ < 0$ [2406.13236].
- **Language-Dependent Effects:** European language contamination (French, Spanish, Italian) yields stronger cross-lingual leakage than East Asian languages (Chinese, Japanese, Korean). This suggests a link to subword/token overlap and model input/output interfaces [2406.13236].

A summary of key signals and their detection power across methods is provided in the following table:

| Detection Method     | Responds to X-Lingual Contam? | Detection Power in TACD Setting |
|---------------------|:-----------------------------:|:------------------------------:|
| N-Gram Overlap      | No                            | Baseline/Chance                |
| Shared-Likelihood   | No                            | Baseline/Chance                |
| TS-Guessing         | No (if evaluated in EN only)  | Baseline/Chance                |
| Min-K% Probability  | No (EN only); Yes in TACD     | High (CLC AUC > 0.9)           |
| TACD (CLC/IDR)      | Yes                           | High (AUC > 0.9)               |

## 5. Impact on LLM Evaluation and Model Development

Translation-aware contamination detection compels a re-examination of evaluation protocols:

- **Multilingual and Translation-Variant Pipelines:** Reliable assessment mandates benchmarking across multiple translations and perturbations, not solely in English or a single language [2601.14994].
- **Understanding LLM Knowledge Representation:** Discrepancies in contamination transfer across languages (e.g., higher for French than for Arabic or Chinese) offer interpretive leverage for studying how LLMs store and access factual and linguistic information [2406.13236].
- **Low-Cost Multilingual Tuning:** The same training pipelines that induce cross-lingual contamination can be repurposed to enhance multilinguality, as continual pre-training on $\tau_\ell(T)$ yields boosts not only in the target language but also in English and other intermediate languages [2406.13236].

## 6. Related Approaches and Generalizations

The TACD philosophy is broadly applicable beyond QA and MCQ evaluation:

- **Text-to-Code:** Masking structural names (e.g., API calls, column names) and measuring reconstruction accuracy (as in DC-Accuracy for text-to-SQL) can serve as a contamination probe across software engineering benchmarks [2402.08100].
- **Generative/Extractive QA:** Masked-token prediction and answer perturbation can be generalized to translation-adjusted evaluation for extractive tasks, using metrics such as exact match and ROUGE-L in each language [2601.14994].
- **Machine Translation Benchmarks:** Adapting the generalization-gap or choice-confusion paradigms allows robustness analysis in translation tasks where canonical sentence pairs can be masked or structurally altered [2402.08100][2406.13236].

In all cases, a strong gap between performance on “public” (potentially contaminated) versus “fresh” splits, or under TACD-style cross-lingual perturbations, constitutes a robust indicator of contamination, surpassing monolingual overlap-based approaches [2402.08100][2406.13236][2601.14994].

Source: https://www.emergentmind.com/topics/translation-aware-contamination-detection