Determine data contamination on HumanEval and MBPP for large code generation models
Determine whether large language models for code generation have training data contamination with respect to the HumanEval and MBPP benchmarks, specifically whether either benchmark’s problems or solutions are included in the models’ pretraining or fine-tuning corpora, and if so quantify the extent of contamination to ensure fair and reliable evaluation.
References
However, due to limited public knowledge about model training data, it is unclear whether models suffer from data contamination on HumanEval and MBPP~\citep{jain2024livecodebench}.
— CodeRAG-Bench: Can Retrieval Augment Code Generation?
(2406.14497 - Wang et al., 2024) in Section 2.1 (Programming Problems: Basic programming problems)
HumanEval is also a widely used benchmark, so we cannot rule out that some of its problems appeared in the pretraining data of the model we fine-tune.
— Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift
(2610.26416 - Batjargal et al., 19 Oct 2026) in Limitations