When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Abstract: AI benchmarks play a central role in measuring progress in model development and guiding deployment decisions. However, many benchmarks quickly become saturated, meaning that they can no longer differentiate between the best-performing models, diminishing their long-term value. In this study, we analyze benchmark saturation across 60 LLM benchmarks selected from technical reports by major model developers. To identify factors driving saturation, we characterize benchmarks along 14 properties spanning task design, data construction, and evaluation format. We test five hypotheses examining how each property contributes to saturation rates. Our analysis reveals that nearly half of the benchmarks exhibit saturation, with rates increasing as benchmarks age. Notably, hiding test data (i.e., public vs. private) shows no protective effect, while expert-curated benchmarks resist saturation better than crowdsourced ones. Our findings highlight which design choices extend benchmark longevity and inform strategies for more durable evaluation.

• The paper introduces an uncertainty-aware saturation index and finds that 29 of 60 LLM benchmarks show high or very high saturation, where leading models become statistically indistinguishable.
• Bayesian regression explains benchmark saturation strongly, with an estimated R² of 0.884, while benchmark age and test-set size consistently outperform adoption, privacy, language, and output format as predictors.
• The findings suggest researchers should report uncertainty, enlarge or stratify test sets, use adversarial or dynamically updated data, and retire benchmarks that lose reliable discriminative power.

This paper presents the first systematic empirical study of benchmark saturation in LLM evaluation, analyzing 60 widely used text-based benchmarks drawn from technical reports by major developers (OpenAI, Anthropic, Google, Meta, Alibaba) (2602.16763). The authors formalize saturation as the loss of reliable discriminative power among state-of-the-art models, introduce an uncertainty-aware saturation index computed from leaderboard data, and test five hypotheses about which benchmark properties drive or delay saturation. Their central findings are that saturation is widespread—nearly half of the analyzed benchmarks exhibit high or very high saturation—and that it is driven primarily by benchmark age and test set scale rather than by commonly assumed design safeguards such as private test sets or open-ended formats.

Operationalizing saturation

Prior discussions of saturation have been largely descriptive, and there has been no agreed quantitative criterion. The paper defines a benchmark as saturated when two conditions hold: (1) top-performing models are statistically indistinguishable on it, and (2) top performance approaches the benchmark's empirically inferred ceiling. If only the first condition holds, the benchmark is stagnated rather than saturated—indistinguishability may reflect evaluation noise or model-level limitations that future advances could overcome. This distinction is important but admittedly hard to draw in practice, because reliable estimates of evaluation noise and ceilings are rarely available; the authors concede this directly.

The operationalization is designed to be model-relative, metric-agnostic, data-driven, and reproducible. For each benchmark, they consider the top $k=5$ leaderboard scores $s_1 \ge \dots \ge s_k$. For accuracy-like metrics over a test set of nominal size $n$, the standard error uses an effective sample size $n_{\mathrm{eff}} = n^{\alpha}$ with default $\alpha = 0.5$, deliberately downweighting nominal dataset size to avoid letting large test sets dominate the calculation. Top-model spread is normalized by the standard error of the difference between first and fifth place, yielding $R_{\mathrm{norm}} = (s_1 - s_k)/\sigma_\Delta$, and the continuous saturation index is defined as

$$S_{\text{index}} = \exp(-R_{\mathrm{norm}}^2) \in [0,1],$$

with interpretive bins at very low (<0.01), low ([0.01, 0.3)), moderate ([0.3, 0.7)), high ([0.7, 0.9)), and very high ($\ge$ 0.9). Using the highest observed score as an empirical ceiling proxy avoids reliance on human baselines, which are inconsistently measured, though this choice means the index captures convergence relative to observed performance rather than absolute task completion.

Corpus construction and annotation

Benchmark selection proceeded in stages: candidate benchmarks were harvested from 61 developer reports published between January 2022 and November 2025 (190 unique benchmarks identified) plus highly cited papers via the Semantic Scholar API; filtering required public documentation, appearance in at least five developer reports for report-derived benchmarks, unambiguous scoring protocols, text-only scope, and adequate longitudinal leaderboard data. Hypothesis-driven refinement then supplemented the set to balance categories such as multilingual, templated, and open-ended benchmarks. Twenty-three researchers annotated each benchmark under a structured schema covering temporality, top-5 scores, documented data-quality issues, task structure, and curation properties, with dual verification and a final consistency audit.

Saturation prevalence and hypothesis tests

Of the 60 benchmarks, 29 exhibit high or very high saturation ($S_{\text{index}} \ge 0.7$), including 14 in the very high category ($\ge$ 0.9). Saturation prevalence rises with age, from 42.9% for benchmarks released within the past 24 months to 54.5% for those older than 60 months (mean indices of 0.51, 0.52, and 0.60 across bins). The authors note this trend is directionally consistent but not statistically significant at conventional thresholds.

The five hypotheses fare unevenly:

Hypothesis	Claim	Outcome
H1	Public benchmarks saturate faster than private	Rejected
H2	English-only saturates faster than multilingual	Rejected
H3	Human-authored resists saturation better than synthetic/hybrid	Partially supported
H4	Closed-ended formats saturate faster than open-ended	Rejected
H5	Older, more-adopted benchmarks saturate faster	Supported

Test-set hiding shows no protective effect: public ($s_1 \ge \dots \ge s_k$0) and private ($s_1 \ge \dots \ge s_k$1) benchmarks have statistically indistinguishable saturation distributions, contradicting the assumption that holding out test data preserves discriminative power once distributional characteristics become widely known. Language coverage appears protective only superficially: English-only benchmarks show higher raw saturation, but multilingual benchmarks average 32.9 months since release versus 48.9 for English ones, so their apparent robustness largely reflects recency—a caveat that applies to several raw comparisons here. Output format, which was age-balanced ($s_1 \ge \dots \ge s_k$2), shows essentially no difference between closed-ended ($s_1 \ge \dots \ge s_k$3) and open-ended ($s_1 \ge \dots \ge s_k$4) benchmarks; generation-based tasks do not preserve discriminative power relative to multiple-choice.

Curation strategy is the one design dimension showing signal: crowdsourced benchmarks are oldest (mean 66.7 months) and nearly all saturated, while expert-curated benchmarks such as ARC-AGI and BIG-Bench Hard remain unsaturated despite prolonged exposure. However, the curation categories differ significantly in age ($s_1 \ge \dots \ge s_k$5), so age remains a confounder even here, and fully synthetic benchmarks' currently low saturation may simply reflect youth. Templated benchmarks do not differ significantly from non-templated ones ($s_1 \ge \dots \ge s_k$6). Documented quality issues correlate with higher saturation, but issue-flagged benchmarks are significantly older (51.5 vs. 30.9 months; $s_1 \ge \dots \ge s_k$7), leaving artifact exploitation, improved construction practices, and post-hoc scrutiny as competing explanations that observational data cannot disentangle.

Joint regression and structural interpretation

A Bayesian regression predicting $s_1 \ge \dots \ge s_k$8 from age, test set size, adoption proxies, accessibility, format, templating, language coverage, curation, and quality issues achieves $s_1 \ge \dots \ge s_k$9, with a posterior AUROC median near 0.98 for classifying saturated versus non-saturated benchmarks. Across specifications, only benchmark age and test set size show consistent effects. Adoption metrics lose significance once age is controlled: citation counts yield $n$0 ($n$1), citation growth $n$2 ($n$3), and technical-report appearances $n$4 ($n$5). The raw correlation between citations and saturation is therefore attributable to shared dependence on time, not popularity per se.

The synthesis frames saturation as a structural interaction between cumulative exposure and finite measurement resolution. Larger test sets lower saturation indices by reducing evaluation uncertainty, meaning score compression can occur well below task mastery—when noise exceeds true performance gaps, top models appear identical despite substantive differences. An important conceptual point follows: the authors argue saturation itself is neutral. Convergence at a valid benchmark's ceiling signals genuine capability attainment; problematic saturation arises from loss of measurement resolution and calls for revision or retirement. Distinguishing the two cases empirically remains difficult given the limited noise estimates acknowledged earlier.

Implications for benchmark practice

The results motivate uncertainty-aware reporting (confidence intervals alongside peak scores), increased evaluation resolution through larger or stratified test sets and multi-metric analysis, dynamic or adversarial data collection to disrupt optimization stability, and explicit retirement criteria when saturation indices remain high. Because private test sets and open-ended formats confer no measurable protection, resources spent on secrecy alone appear misdirected relative to investments in scale and monitoring. The finding that expert-curated, adversarially designed benchmarks resist saturation longer suggests deliberate difficulty engineering has value, though the confounding with age tempers causal claims.

Limitations

The corpus overrepresents widely adopted benchmarks by construction, and top-N leaderboard snapshots may miss dynamics for sparsely evaluated benchmarks. Benchmark properties are assumed time-invariant even though attributes like annotation diversity evolve after release. Restriction to text-only benchmarks limits generalizability to multimodal settings. Leaderboard heterogeneity—inconsistent LLM-judge setups, irregular updates, third-party maintenance—required subjective trustworthiness judgments before data extraction. The observational design cannot establish causality for any single factor, particularly for curation and quality issues where age confounds dominate.

Conclusion

By grounding a formal definition of saturation in leaderboard uncertainty and testing design hypotheses across 60 benchmarks, this work replaces anecdote about benchmark aging with quantified evidence: saturation affects roughly half of contemporary LLM benchmarks, scales with age and shrinks with test set size, and is unaffected by private test sets, output format, or template diversity. Expert curation shows promise as a durability lever but requires disentangling from benchmark maturity. The central open questions the paper leaves are whether apparent saturation is transient—recoverable through paradigm shifts as seen with reasoning-centric models—and how to reliably separate stagnation from genuine task mastery when evaluation noise and ceilings are themselves uncertain.