Spiking the training data to correct for test set contamination

Abstract: The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our core proposal is to spike the training data by intentionally contaminating some test examples at known rates. The spiked examples can then be used to calibrate predictors of model memorization which enable principled statistical correction of inflated test scores. To evaluate different correction estimators, we first present a simulation framework based on the Hubble models. Hubble models come in minimal pairs, where the perturbed model was deliberately contaminated with several test sets, while the standard model was not, serving as the counterfactual and correction target. We consider estimators that use information from a memorization predictor, correctness predictor, or both. In simulation, we establish basic statistical intuitions and show that estimators leveraging memorization and correctness information are better than naive estimation which makes no correction at all. We then instantiate several memorization and correctness predictors, and find that simple predictors such as Platt-scaled membership inference metrics provide good signal for correction. Finally, we examine the practical considerations of spiking. Simple memorization predictors need no more than 10 examples for calibration and often transfer from one dataset to another. Taken together, spiking is a promising solution for test set contamination.

• The paper introduces a spiking protocol to statistically correct test set contamination by calibrating memorization and correctness predictors.
• The methodology uses simulation with Llama-based models to compare four estimators, revealing optimal performance with IPW and combined approaches.
• The approach shows high sample efficiency, reducing naive error by up to 13.1 points and enhancing benchmark reliability in LLM evaluation.

Spiking the Training Data to Correct for Test Set Contamination

Motivation and Conceptual Framework

Test set contamination in LLMs systematically inflates benchmark scores, undermining the validity of empirical evaluations. The chronic presence of benchmark or test set data in large-scale pretraining corpora necessitates robust correction strategies beyond detection. This work proposes a principled correction protocol: intentionally "spiking" a subset of test examples into the training set at known rates, thereby enabling calibration of memorization predictors and facilitating statistical adjustment of observed scores.

The approach reframes the contaminated test score as a mixture of true performance and contamination-induced memorization. Correction requires answering two counterfactual questions for each test item: (a) was the item memorized due to contamination, and (b) what would the model have answered absent contamination. Spiking provides randomized ground truth (via known insertions), satisfying causal inference requirements and enabling calibration of memorization and correctness predictors.

Figure 1: Illustration of correcting contaminated test scores. Each point represents a test example and its $P(\mathrm{correct})$ under Hubble models. Panels show no contamination, random contamination, and contamination correlated with example difficulty.

Simulation Methodology and Estimator Design

The evaluation leverages the Hubble suite: pairs of Llama-based models, with perturbed variants deliberately contaminated by benchmark test items at controlled duplication rates. The simulation protocol samples test sets under various contamination regimes, enabling direct comparison between the contaminated (perturbed) and clean (standard) models.

Correction estimators are constructed using (a) a memorization predictor estimating $P(\text{contam}|i)$, and (b) a correctness predictor estimating $P(\text{correct}|i)$ (the probability the clean model would answer correctly). Four estimators are formulated:

• Naive estimator: Uncorrected average of observed outcomes
• IPW (Inverse Propensity Weighting): Downweights observed correctness by estimated contamination probability
• Imputation estimator: Replaces all outcomes with correctness predictor estimates
• Combined estimator: Weighted interpolation between observed outcome and predictor estimate, based on contamination probability

Phase diagrams reveal estimator selection is regime-dependent: IPW is optimal for random contamination, imputation for correlated contamination, and the combined estimator is robust across strong, correlated regimes.

Figure 2: Phase diagrams showing optimal estimator selection by RMSE, as function of memorization and correctness predictor quality and contamination regime.

Empirical Benchmarking of Predictors

Memorization predictors are built from membership inference attacks leveraging token-level statistics (sequence log-likelihood, Min-K\%, Min-K\%++, zlib-normalized likelihood, and reference likelihood ratios). Calibration is performed via Platt scaling using spiked calibration items. Predictive discrimination improves substantially with contamination strength, achieving near-perfect AUROC for heavily duplicated items.

Correctness predictors are instantiated using both fine-tuned classifiers (RoBERTa) and paired pretrained LLMs (Llama-3.1, Pythia, Qwen3) with Platt scaling. Calibration is performed solely on clean calibration split items, and absolute bias is evaluated across correlated difficulty bins—results show low bias for all methods, supporting their utility for correction.

Estimator Performance and Practical Considerations

Estimates are evaluated by RMSE against clean standard model accuracy. Heavy contamination induces substantial score inflation (naive error up to +13.1 points under random, high-dose contamination), and correction drastically reduces RMSE. Combined and imputation estimators effectively address selection bias under correlated contamination, outperforming heuristic reweighting baselines.

Spiking efficiency is examined by varying calibration set size. Memorization predictors calibrated via Platt scaling require only ~10 spiked items for effective correction, due to minimal parameter fitting and calibration balance. Correctness predictors are less sample efficient, typically needing hundreds of examples. Memorization predictors transfer well across benchmarks, with Wikipedia-calibrated predictors performing comparably to dataset-specific ones.

Figure 3: RMSE of IPW and imputation estimators versus spiked calibration set size, demonstrating high sample efficiency for IPW with Min-K\%++.

Figure 4: RMSE analysis for mid random contamination across datasets, further substantiating sample efficiency and estimator robustness.

Implications and Future Directions

Statistical correction via spiked training data allows rigorous isolation of true model performance, facilitating cleaner benchmarking and more reliable empirical conclusions. The low cost and transferability of memorization predictors recommend this protocol for operational deployment. Spiking also establishes randomized ground truth, augmenting model auditing capacity for privacy, provenance, and governance analyses—potentially serving as foundation for technical regulatory frameworks.

Practically, adoption of spiking-based evaluation requires cooperation from model developers but yields substantial benefits in auditability and benchmarking integrity. Theoretical extensions include adapting the simulation framework for paraphrased or near-duplicate contamination scenarios.

Conclusion

This work formalizes statistical correction for test set contamination using randomized spiking, calibrated memorization and correctness predictors, and principled estimators. Simulation and empirical benchmarking demonstrate substantial error reduction compared to naive baselines, with extremely sample-efficient memorization predictors. Spiking offers a robust path to rigorous, audit-ready LLM evaluation, opening new avenues for technical governance and empirical reproducibility.