The Capability Frontier: Benchmarks Miss 82% of Model Performance

Abstract: Existing benchmarks typically report accuracy for a single model on a single run. This systematically understates real-world LLM capabilities, particularly under heterogeneous data distributions: (i) different models get different questions correct according to their specializations, and (ii) given a budget, multiple generations can be sampled and selectively retained. To quantify this gap, we introduce the Capability Frontier: a Pareto frontier over a set of models that characterizes the best achievable performance at each cost level under optimal selection across models and generations (i.e., via an oracle). Our construction corrects for two opposing biases: underestimation from single-model evaluation and overestimation from taking maxima over noisy samples. We study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks, comparing Capability Frontier performance at matched cost to each benchmark's top-performing model. Correcting for single-model evaluation yields a 54% error rate reduction; additionally correcting for single runs yields an 82% improvement, with SOTA accuracy matched at 85% cost reduction. Complementing these empirical results, we use controlled probabilistic simulations to show that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model. Our findings suggest collective LLM capabilities are substantially underestimated, with implications for evaluation and deployment in data-heterogeneous, multi-domain settings.

• The paper introduces a debiased Capability Frontier that estimates the best quality achievable at each cost through optimal per-prompt selection across 21 LLMs and 16 benchmarks.
• The paper finds that cross-model routing reduces error rates by 53.7% on average and achieves comparable accuracy with 85.2% lower costs than the best single model, although these are oracle bounds.
• The paper shows that posthoc selection across 10 generations can reduce errors by 82.4%, while naive oracle methods overestimate performance—especially cost—highlighting the need for practical routers and reliable judges.

Motivation and problem statement

Standard LLM benchmarking reports the accuracy of a single model on a single sampled generation per prompt. The paper argues this convention systematically understates what is achievable in heterogeneous, multi-domain workloads, for two reasons: different models specialize in different query types, and, given a budget, multiple generations can be sampled and selectively retained. Prior routing studies — oracle routing (Shnitzer et al., 2023), RouterBench (Hu et al., 2024), RouteLLM (Ong et al., 2024) — estimated oracle gains from finite samples of generations per prompt by taking maxima over noisy sample means. This introduces a positive bias (the "optimizer's curse," well documented in economics and decision analysis [2305.xxxx-adjacent literature; see (2606.26836)'s citations of Andrews et al. and Smith & Winkler]), inflating reported headroom.

The paper's central contribution is the Capability Frontier: a Pareto frontier over a pool of models characterizing the best achievable quality at each cost level under optimal per-prompt model selection (and optionally per-prompt generation selection), with explicit debiasing so that the frontier is neither an underestimate (single-model evaluation) nor an overestimate (naive maxima).

Formal setup

For prompt $n$, model $l$, and generation $g$, observations $\phi_{nlg}$ encode quality and cost. The true oracle value is $$\mathcal{O}^{true} = \frac{1}{N}\sum_n \max_l \mu_{nl}$$ where $\mu_{nl} = \mathbb{E}[\phi_{nlg}]$; the standard estimator replaces $\mu_{nl}$ with the sample mean $\bar{\phi}_{nl}$ and takes the maximum, yielding $\mathcal{O}^{biased}$. Under Gaussian assumptions with equal means across models, the bias has closed form approximately $\bar{\sigma}\sqrt{2\log L / G}$: it decays as $l$0 and grows with pool size $l$1 and observation variance. For Bernoulli correctness metrics, synthetic study shows decay as $l$2 with $l$3 when generations are correlated across models, and that roughly $l$4 generations are required before the asymptotic form dominates. At realistic budgets ($l$5), the asymptotic form does not hold, motivating a smooth-transition extrapolation curve fitted to finite-$l$6 data.

Two debiasing methods are proposed:

• Extrapolation: fit $l$7 (or its smooth-transition variant) and take the intercept $l$8 as $l$9. With $g$0 this carries risk, validated against synthetic ground truth and PGM estimates.
• Probabilistic graphical model (PGM): latent prompt difficulty $g$1, topic assignment $g$2, and per-topic model aptitude $g$3 generate success probabilities $g$4 via a learned link function (a feedforward network outperformed a multiplicative form). Stochastic variational inference yields debiased oracles via $g$5. The authors concede the PGM involves ad-hoc structural choices and assumes conditional independence across generations given latents — an assumption that may fail when models share training data.

The multi-objective frontier is traced by sweeping a scalarization weight $g$6 over normalized quality and cost, applying debiasing separately to each axis (LogNormal likelihood for cost). A posthoc oracle extends selection to $g$7 retained generations per model under a perfect, free judge — an assumption the paper flags explicitly as unrealistic in deployment.

Experimental results

The evaluation covers 21 frontier LLMs (GPT-5 family, Claude 4.5, Gemini 2.5, Llama 4, Mistral, Qwen, Kimi K2, DeepSeek R1, GLM-4.6) across 16 benchmarks with binary verifiable metrics, spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks, with $g$8 generations per prompt-model pair.

Finding 1 — Routing gains are large. Against each benchmark's SOTA single model, the debiased oracle reduces error rate by 53.7% on average (e.g., BigCodeBench 35.8% → 49.1% accuracy; MedCalcBench 70.5% → 86.4%; GPQA Diamond 94.0% → 99.0%). Conversely, matching SOTA accuracy achieves 85.2% average cost savings (e.g., TruthfulQA cost drops from 0.38 to ~0 cents; Terminal-Bench 2.0 from 260.84 to 25.39 cents). These are strong claims: they assert that collective capability is dramatically understated by single-model leaderboards, though they describe oracle upper bounds rather than realized router performance.

Finding 2 — Posthoc selection compounds gains. With a perfect free judge selecting among $g$9 generations across all models, error reduction rises to 66.1% at $\phi_{nlg}$0 and 82.4% at $\phi_{nlg}$1, with several benchmarks reaching 100% error reduction (all prompts solved by some model-generation pair). The paper notes this naive formulation queries every model for every prompt and that sequential prompting with early exit could attain similar gains at lower cost — but does not demonstrate it.

Finding 3 — Naive oracles overestimate. Comparing biased to debiased estimates: quality is inflated by ~1.2% on average (up to 9.2% on BigCodeBench), while cost is inflated by up to 88% (average ~37.5% in the text; the appendix figure reports ~23.8% mean). Cost bias dominates because cost distributions are heavily skewed, amplifying selection effects. This directly qualifies prior routing literature built on biased estimators.

Finding 4 — Reliability varies independently. Defining reliability as $\phi_{nlg}$2, GPT-5-mini is most reliable (90.2%) and GLM-4.6 least (76.3%), with no significant correlation between reliability and either quality or cost — implying reliability is an independent axis for router design.

Finding 5 — Diversity drives gains. In controlled PGM simulations (10 LLMs, 30 latent topics, 10 million observations), oracle uplift over the best single model increases near-monotonically with topic-distribution entropy. This provides a mechanistic account: achievable routing gains are fundamentally driven by workload heterogeneity, explaining variation in gains across benchmarks.

Limitations and open questions

The paper is candid about several constraints. First, $\phi_{nlg}$3 generations sit below the regime where the $\phi_{nlg}$4 bias law is reliable, so extrapolation-based debiasing carries residual uncertainty, mitigated only by synthetic validation and cross-checks against the PGM. Second, agentic benchmarks fix one model per trajectory because per-step optimal routing is combinatorially hard — meaning reported agentic gains likely understate achievable performance. Third, posthoc results assume an error-free, zero-cost judge; real verifiers would erode the headline 82% figure substantially, and quantifying that erosion is left open. Fourth, the PGM's structural choices and independence assumptions could influence estimates. Finally, the paper measures oracle bounds, not deployable routers; whether practical policies can approach the frontier under realistic constraints remains unaddressed, as do the effects of system-prompt and hyperparameter variation on frontier shape.

Conclusion

This paper reframes LLM evaluation around a debiased Pareto frontier rather than single-model, single-run scores. Its empirical claims are substantial: a 54% average error reduction from cross-model selection alone, 82% with multi-run posthoc selection, and 85% cost savings at matched accuracy — alongside evidence that prior oracle estimates were themselves inflated, particularly on cost. The entropy simulation grounds these gains in data heterogeneity. The principal caveat is that all headline numbers are oracle bounds contingent on perfect selection and, for posthoc results, perfect judging; the gap between these bounds and realizable routing systems remains the field's outstanding question.