Contamination-Free Rolling Benchmark

• Contamination-free rolling benchmarks are evaluation frameworks that continuously renew test datasets to prevent model exposure to outdated information.
• They integrate strict temporal separation, synthetic instance generation, and controlled access mechanisms to ensure unbiased and reproducible assessments.
• Empirical evidence shows significant performance drops on post-cutoff data across domains, highlighting the necessity for dynamic, continuously updated evaluation protocols.

A contamination-free rolling benchmark is a benchmark framework that combines explicit anti-contamination mechanisms with recurrent or continuous refresh, so that evaluation items remain outside plausible model exposure windows while still supporting reproducible measurement over time. Recent work does not converge on a single formal definition. Instead, it operationalizes the concept through strict temporal separation, synthetic instance generation, hidden or private evaluation protocols, post-cutoff slicing, and automated update pipelines, with different emphases in protein science, code, medicine, function calling, and general knowledge evaluation (Rong et al., 24 Dec 2025, Wu et al., 2024, White et al., 2024, Maekawa et al., 30 Sep 2025, Yan et al., 10 Feb 2026).

1. Conceptual foundations

In this literature, contamination usually denotes prior model exposure to benchmark items or to information sufficiently close to them that benchmark scores become inflated. The most common operationalization is temporal: a test item is treated as contaminated if it was publicly available early enough to have plausibly entered model pretraining or post-training corpora. LiveProteinBench makes this explicit by treating contamination as prior exposure to protein records through public databases, literature, or web content before model cutoffs, and it prevents this by selecting only experimentally validated proteins first released after 2025-01-01 and cross-validating entry creation dates against publication dates (Rong et al., 24 Dec 2025). AntiLeakBench sharpens the distinction by arguing that newly collected data are not necessarily new knowledge; its criterion is not just post-cutoff publication, but post-cutoff change in a subject–relation–object fact, recovered from Wikidata/Wikipedia histories (Wu et al., 2024).

The adjective rolling is also used with different degrees of strictness. Some systems are fully time-indexed and explicitly refreshed, such as LiveBench with monthly updates, LiveMedBench with weekly harvesting, and ArxivRoll with six-month cycles (White et al., 2024, Yan et al., 10 Feb 2026, Liang et al., 25 Jul 2025). Others are better described as live in principle: LiveProteinBench includes an automated update pipeline intended to keep the benchmark ahead of future models, but the published work evaluates one post-2025 snapshot rather than a sequence of released benchmark waves (Rong et al., 24 Dec 2025).

Recent papers also distinguish contamination-free, contamination-aware, and contamination-resistant. Know2Guess is explicitly contamination-aware rather than contamination-free, because it uses frozen labels and contamination-risk metadata rather than a guarantee of zero exposure (Meng et al., 30 Apr 2026). MMLU-CF combines decontamination transformations with a hidden test split to resist both accidental and malicious leakage, but it is still a static release rather than a continuously refreshed benchmark (Zhao et al., 2024). More programmatic proposals argue that public benchmarks should become contamination-resistant or even unlearnable at release time, shifting attention from post hoc detection to benchmark architecture itself (Al-Lawati et al., 19 May 2026, Musawi et al., 13 May 2025).

2. Construction strategies

The benchmark-design space is now structured around several recurring construction logics. Some benchmarks derive novelty from time-stamped real-world sources; some from on-demand synthetic generation; some from governance and access control; and some from combinations of all three.

Benchmark	Domain	Core anti-contamination mechanism
LiveProteinBench	Protein science	Post-2025 experimentally validated UniProt entries, cross-validated with publication dates
AntiLeakBench	Real-world QA	Post-cutoff Wikidata knowledge changes plus post-change Wikipedia revisions
LiveCodeBench	Code	Newly released contest problems after model cutoff dates
LiveMedBench	Medicine	Weekly harvesting of recent real-world clinical cases and post-cutoff subsets
FuncBenchGen	Tool use	Runtime synthetic DAG-generated function schemas and tasks
MMLU-CF	MCQ knowledge	Rephrasing, choice shuffling/replacement, plus hidden test / public validation
ArxivRoll	Scientific text	Private SCP tasks generated from recent arXiv papers, used once and then retired

Temporal-source benchmarks rely on authoritative or timestamped corpora. LiveCodeBench continuously collects new problems from LeetCode, AtCoder, and CodeForces, then evaluates models only on problems released after their cutoffs (Jain et al., 2024). LiveMedBench weekly harvests physician-answered cases from iCliniq, SDN, DXY, and MedLive, retains only posts published on or after 2023-01-01, and defines post-cutoff subsets per model (Yan et al., 10 Feb 2026). AntiLeakBench uses Wikidata claim transitions and later Wikipedia revisions to create QA tuples grounded in knowledge that changed after a model cutoff, rather than merely in newly posted text (Wu et al., 2024).

Synthetic-generation benchmarks avoid static-item leakage by generating tasks at evaluation time. FuncBenchGen casts multi-step function calling as traversal over a hidden function-dependency DAG $G=(\mathcal{F},\mathcal{E})$, with randomly generated function names, variable names, and values, so the task instances do not exist publicly beforehand (Maekawa et al., 30 Sep 2025). The Caesar-cipher benchmark proposed in the contamination-resistance literature uses procedurally generated plaintext/shift combinations so that the benchmark family is effectively unbounded, even though the underlying task is simple (Musawi et al., 13 May 2025).

Governance-based designs reduce leakage by controlling benchmark access. MMLU-CF releases a public validation split but keeps the 10,000-question test split closed-source, using the validation–test gap $\Delta$ as an integrity monitor over time (Zhao et al., 2024). TRUCE generalizes this into private benchmarking architectures based on trusted dataset owners, trusted third parties, confidential computing, or secure multiparty computation, so that evaluation can occur without exposing hidden test sets to model providers (Rajore et al., 2024). C$^2$LEVA adds active benchmark protection through licensing, encryption, and watermarking on a subset of items, complementing renewable passive generation (Li et al., 2024).

3. What makes a benchmark genuinely rolling

A rolling benchmark is not merely a fresh benchmark. It requires an update process, a temporal policy, and some account of comparability across releases. The recent literature shows substantial variation in how fully these elements are specified.

The most explicit refresh policies are found in LiveBench, LiveMedBench, and ArxivRoll. LiveBench adds and updates questions monthly from recent information sources, releases all questions and answers, and intends to introduce harder task versions over time so that the benchmark remains discriminative as models improve (White et al., 2024). LiveMedBench harvests new clinical cases weekly and simultaneously publishes frozen snapshots such as LiveMedBench-v2026.01, combining a live stream with versioned reproducibility (Yan et al., 10 Feb 2026). ArxivRoll constructs a new private benchmark every six months from recent arXiv papers, evaluates models once, then releases that benchmark publicly and retires it, explicitly adopting a one-time-use lifecycle (Liang et al., 25 Jul 2025).

Other systems are rolling-compatible rather than fully operationalized. AntiLeakBench is designed to be rerun whenever newer Wikidata dumps and Wikipedia revisions become available; this makes it low-cost to refresh, but the paper does not specify a fixed cadence such as monthly or quarterly (Wu et al., 2024). C$^2$LEVA describes periodic crawling, synthetic regeneration, contamination detection, and a continuously maintained leaderboard, but does not present a mature multi-version release history or a formal refresh protocol (Li et al., 2024). LiveProteinBench has an automated update pipeline that “periodically scans” source databases, yet the published work does not define version control, archival policy, or frozen test-set release intervals (Rong et al., 24 Dec 2025).

This distinction matters because a benchmark can be contamination-resistant without being a fully specified rolling infrastructure. The literature repeatedly suggests that robust rolling evaluation needs four operational components beyond freshness: a declared cadence, release-version identifiers, a freezing or snapshot policy, and longitudinal comparability rules. Where these are absent, “live” often means continuously collectable rather than continuously benchmarked.

4. Task formats and evaluation protocols

Contamination-free rolling benchmarks do not share a single task format. Their commonality lies in how novelty is enforced, not in how ability is scored. As a result, the evaluation layer varies from exact-match multiple choice to execution-based grading and rubric-driven open-ended assessment.

Objective grading is strongly preferred wherever possible. LiveProteinBench casts all 12 protein tasks as single-choice questions and uses accuracy as the core metric (Rong et al., 24 Dec 2025). LiveCodeBench uses execution-based correctness for code generation and self-repair, alongside exact output checking for code execution and test-output prediction (Jain et al., 2024). LiveBench restricts itself to tasks with objective ground truth and then averages question scores into task, category, and overall benchmark scores, explicitly avoiding LLM judging on hard objective problems (White et al., 2024). FuncBenchGen scores end-task success and average number of function calls, with a bounded interactive budget, rather than requiring a unique exact call sequence (Maekawa et al., 30 Sep 2025).

Where answerability itself is the target, the protocol becomes more structured. Know2Guess defines each item as a tuple with zone label, abstention expectation, origin, and contamination-risk tag, then parses outputs into $\text{ANSWER}$, $\text{ABSTAIN}$, or $\text{REFUSE}$. Its primary metric is reliability,

$$\mathrm{Rel}=\frac{1}{N}\sum_{i=1}^{N}\Big[\mathbb{I}(e_i=0 \land \hat y_i=\mathrm{ANSWER} \land \hat a_i \in G_i) + \mathbb{I}(e_i=1 \land \hat y_i=\mathrm{ABSTAIN})\Big],$$

which gives equal credit to correct answers on answer-expected items and correct abstentions on abstention-expected items (Meng et al., 30 Apr 2026).

Open-ended domains require richer scoring than lexical overlap. LiveMedBench decomposes each physician response into case-specific rubric criteria $R=\{(c_1,w_1),\dots,(c_m,w_m)\}$, with weights $w_j \in [-10,10]$, and scores a response by weighted criterion satisfaction clipped to $\Delta$0: $\Delta$1 This rubric-based evaluator is intended to align better with physicians than generic LLM-as-a-Judge scoring (Yan et al., 10 Feb 2026).

The result is methodological heterogeneity rather than convergence. Taken together, these systems suggest that a contamination-free rolling benchmark is best understood as a benchmark lifecycle architecture, not a fixed metric family.

5. What rolling benchmarks have revealed empirically

Across domains, fresh or post-cutoff evaluation consistently lowers apparent model performance and changes qualitative conclusions. The most direct effect is that models often look much weaker on temporally novel data than on older public benchmarks.

In protein science, LiveProteinBench reports that general-purpose proprietary models outperform open-source and biology-specialized models by over 20 percentage points of accuracy on post-cutoff proteins, with GPT-5 reaching 60.65 average accuracy and the best specialized model, BioMedGPT-R1, reaching 31.83 (Rong et al., 24 Dec 2025). Its temporal validation experiment reinforces the contamination argument: on a comparison set built from 2020 UniProt entries, several models score far higher than on the live post-2025 set, with drops such as GPT-4o from 59.91 to 32.59 and Claude-3.7-sonnet from 66.76 to 36.41 (Rong et al., 24 Dec 2025). The benchmark therefore attributes a large portion of prior apparent capability to historical exposure.

In code, LiveCodeBench shows that rankings from older static benchmarks do not transfer cleanly to new contest problems. The correlation between HumanEval+ and LCB-Easy generation is only 0.77, and several fine-tuned open models that look strong on HumanEval+ degrade sharply on the fresh benchmark (Jain et al., 2024). The benchmark also surfaces task-level profile differences: Claude-3-Opus leads GPT-4-Turbo on test-output prediction, while GPT-4-Turbo is strongest on overall generation and self-repair in the post-cutoff slice (Jain et al., 2024).

In medicine, LiveMedBench reports that even the best model, GPT-5.2, reaches only 39.2% overall, and that 84% of tested models degrade on post-cutoff cases (Yan et al., 10 Feb 2026). Its January 2026 freshness analysis further shows that open-book retrieval partially recovers performance on very recent cases, which suggests that temporal degradation is not reducible to random noise; newly emerging clinical information materially changes outcomes (Yan et al., 10 Feb 2026).

Synthetic rolling frameworks expose a different failure mode: limited compositional reasoning under controlled novelty. FuncBenchGen shows GPT-5 success dropping from 72.5% at 5 core nodes to 15.0% at 20 core nodes, with connected distractor functions proving especially difficult and “Value Not Yet Known” dominating failure analysis (Maekawa et al., 30 Sep 2025). These results indicate that contamination-free generation can separate planning depth and state-tracking ability from benchmark familiarity.

More broadly, the literature suggests that the largest gains from rolling evaluation are not merely lower absolute scores, but different interpretations of what models are good at. LiveProteinBench argues for reasoning depth over domain memorization; LiveCodeBench shows that static code benchmarks overstate some fine-tuned models; LiveMedBench shifts the error narrative from factual deficiency to contextual application failure; FuncBenchGen isolates brittle multi-step state tracking rather than tool-schema understanding.

6. Limits, controversies, and future directions

Despite the label, most current systems are not contamination-free in an absolute sense. Their guarantees are operational rather than formal. LiveProteinBench’s post-2025 timestamp does not logically exclude earlier homologous information, partial records, or indirect leakage through related literature (Rong et al., 24 Dec 2025). AntiLeakBench depends on assumptions about claim timestamps, model cutoff metadata, and the absence of earlier mentions of the updated fact (Wu et al., 2024). Know2Guess openly states that contamination-risk labels are proxies and do not prove whether a specific model memorized a specific item (Meng et al., 30 Apr 2026). LiveMedBench reduces contamination risk through temporal separation but cannot exclude all possible exposure channels to recent web content (Yan et al., 10 Feb 2026).

Another limitation is that benchmark refreshing can itself damage benchmark continuity. A controlled large-scale study of 20 contamination-mitigation strategies finds that no existing benchmark-update method significantly improves resistance over the no-update baseline across all benchmarks, and none balances fidelity with contamination resistance reliably (Sun et al., 20 Mar 2025). This suggests that rolling benchmarks cannot assume that paraphrasing, translation, or LLM-generated successor items preserve construct validity.

Post hoc contamination detection is also fragile. For large reasoning models, contamination introduced during SFT can be visible initially yet become hard to detect after brief GRPO-style RL, because the member/non-member confidence gaps used by many detectors contract while benchmark score inflation remains (Wang et al., 30 Sep 2025). This pushes benchmark design away from retrospective contamination forensics and toward stronger ex ante controls.

At the same time, newer auditing proposals indicate that structural verification can help. Cross-Context Verification solves the same coding benchmark problem in multiple session-isolated trials and measures solution diversity and gold proximity, while the Hierarchical Cross-Context Architecture enforces analyst independence through restricted information flow; on a nine-problem SWE-bench Verified slice, this combination separates contaminated from genuine reasoning behavior in the reported sample and identifies contamination–flaw composite cases (Song, 23 Mar 2026). The implication is not that contamination detection is solved, but that benchmark governance must become process-aware.

Two future directions recur in the literature. One is private benchmarking, where hidden tests are evaluated via trusted infrastructure, confidential computing, or secure multiparty computation rather than by publishing raw items outright (Rajore et al., 2024). The other is contamination-resistant release formats, where benchmarks are published in representations intended to support inference but be unlearnable under standard training pipelines, such as KV-cache-based latent encodings or other transformed datasets (Al-Lawati et al., 19 May 2026). Both approaches shift the problem from “refresh faster” to “make publication itself less harmful.”

Taken together, these works suggest that the most stable meaning of a contamination-free rolling benchmark is not “a benchmark that can never leak,” but “a benchmark system that continuously regenerates or renews evaluation data, timestamps and versions that data, constrains or audits exposure channels, and treats contamination as an ongoing lifecycle problem rather than a one-time cleaning step.”