How Inference Compute Shapes Frontier LLM Evaluation

Abstract: AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving. As a result, performance is increasingly sensitive to the amount and allocation of compute available at test time ("inference compute"). Yet many evaluations still report performance at a single restrictive budget, meaning that low scores may reflect the evaluation setup rather than the model's underlying capability. To test this, we evaluate up to 12 frontier LLMs on seven challenging benchmarks spanning software engineering, mathematics, medicine, and cybersecurity. We use a controlled setup combining three simple inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts, guided either by the model itself or by minimal correctness feedback. We find three main results. First, larger token budgets substantially improve performance on benchmarks across multiple domains, including cybersecurity, FrontierMath, Humanity's Last Exam, and TerminalBench. Second, fixed-budget evaluations can increasingly understate frontier capability as models advance. Newer models reach higher performance at large budgets, where they unlock harder tasks and solve them more reliably. Third, benchmarks differ in which inference-scaling methods help most: repeated submission broadly improves performance, but the value of larger token budgets, external feedback, and parallel attempts varies by benchmark. Overall, our results show that benchmark scores are protocol-dependent. We therefore argue that evaluations should report capability as a function of inference-time compute, specify protocol choices explicitly, and compare model generations over a large shared compute range at matched budgets, especially in safety- or policy-relevant settings.

• The paper demonstrates that expanded token budgets and iterative submission protocols significantly boost LLM performance on complex, long-horizon tasks.
• The paper systematically evaluates 12 state-of-the-art LLMs across diverse benchmarks, revealing that fixed compute protocols can underestimate true model capabilities.
• The paper highlights generational improvements in reach and reliability while emphasizing that evaluation outcomes are critically shaped by inference-time compute allocation.

Inference-Time Compute as a Determinant in Frontier LLM Benchmarking

Motivation and Context

Recent advancements in LLM evaluation emphasize increasingly complex, long-horizon tasks that rely on extended trajectories, tool use, and iterative problem-solving. This shift induces a heightened sensitivity of model performance to inference-time compute allocations. Yet, evaluation practices often rely on fixed, restrictive compute budgets—token limits, turn caps, or timeouts—that may artificially suppress observed capabilities. Such protocol-dependent measurement risks misrepresenting a model’s potential, especially as frontier LLMs progress. The paper systematically interrogates inference scaling through a uniform protocol across 12 state-of-the-art LLMs on seven benchmarks—including software engineering (TerminalBench, SWE-Bench Pro), mathematics (FrontierMath), medicine (HealthBench), expert knowledge (Humanity's Last Exam), and cybersecurity scenarios (Cyber CTFs, The Last Ones) (2606.17930).

Experimental Design

The investigation employs three general inference-scaling interventions:

• Expanded Token Budgets: Total trajectory budgets are increased by 1–3 orders of magnitude over typical published defaults, allowing test-time compute to reach up to 100M tokens in some settings.
• Context Compaction: Summarization of earlier context by the model enables serial scaling—circumventing window constraints and preserving information for long-horizon tasks.
• Iterative Submission/Resubmission: Repeat attempts per task, governed either by self-guided exploration or minimal correctness feedback, create opportunities for refinement.

Evaluations are fully crossed: each model is assessed under both no feedback and oracle score feedback conditions, using a ReAct-style agent scaffold. For each benchmark-model-condition combination, five independent trajectories are run, enabling robust statistical analysis of scaling effects.

Quantitative Performance Insights

Benchmark Sensitivity to Inference Scaling

Scaling with larger token budgets yields substantial performance improvements, with strong benchmark-dependent variability:

• FrontierMath and HLE exhibit pronounced headroom. Increasing token budgets from typical values (1M to 10M for FrontierMath; 64k to 5M for HLE) raises success by $+11.7 \pm 11.0$ and $+9.3 \pm 12.0$ percentage points, respectively.
• TerminalBench and SWE-Bench Pro, evaluated at already permissive budgets, only marginally benefit: increases are $<1.5$ percentage points even when budgets are doubled.
• HealthBench is nearly saturated; performance increases are negligible ($+0.3 \pm 0.4$ points).
• Cyber benchmarks (CTFs, The Last Ones) and selected stateful environments continue to improve within tested ranges—indicating protocol-induced headroom, not inherent saturation.

Plateau analysis establishes that diminishing returns from increased compute arise at task-specific locations; in many cases, evaluated token caps reveal only partial reachable performance.

Model Generational Effects

Decomposition of generational gains (across three generations per provider—Anthropic and OpenAI) demonstrates:

• Reach: Successive model iterations unlock harder tasks (with negative generation-by-difficulty interaction for reach, $p < 0.05$ on five out of six benchmarks).
• Reliability: Newer models solve unlocked tasks more consistently across repeated attempts, a robust trend except HealthBench.
• Efficiency: Token usage per solved task is reduced in newer models, but improvements are uneven and conditional on reach—most pronounced in Cyber CTFs and HealthBench.

Aggregate trends show that newer models attain higher capped performance and begin succeeding at lower budgets, but the principal advances are in reach and reliability rather than pure token efficiency.

Protocol Dependence and Submission Dynamics

Repeated submission materially enhances performance across all studied benchmarks, with uplift multipliers ranging from 1.11x (FrontierMath) to 1.70x (HLE). Oracle score feedback is particularly potent in settings where it guides continued search (notably HLE and SWE-Bench Pro), whereas self-guided refinement dominates on benchmarks close to saturation.

Parallelization—allocation of fixed compute budgets across multiple independent trajectories—benefits stateless tasks (HLE, HealthBench) far more than stateful tool-based environments. Gains from parallel sampling diminish markedly in newer models, reflecting increased ability to leverage deep, single trajectories for problem solving.

Theoretical and Practical Implications

The findings challenge the adequacy of fixed-protocol, single-budget evaluation practices. Benchmark scores are not static reflections of model architecture but are critically shaped by inference-time compute and submission rules. The results call for:

• Reporting capability as a function of inference budget, not a single number.
• Explicit protocol design and documentation—including tool use, feedback, and trajectory/task allocation.
• Computing cross-generational comparisons at matched budgets over broad ranges, especially in safety- or policy-relevant contexts.

These recommendations align with recent literature emphasizing compute-optimal inference scaling [Snell et al., 2024], time-horizon analyses [Kwa et al., 2025], and the necessity of protocol-aware benchmarking [Balachandran et al., 2025].

Practically, underestimating model performance through compute-starved evaluation poses governance risks, as well-resourced actors may unlock substantial additional capabilities in real-world deployments. For theoretical analysis, inference scaling curves reveal latent capabilities and should be a staple in capability tracking.

Limitations and Future Directions

Limitations include use of a single, general scaffold and relatively simple inference-scaling techniques: more sophisticated elicitation (e.g., verifier-guided branching, adaptive width/depth) may shift results. Benchmark-specific saturation may reflect protocol or scaffold mismatches, not inherent model boundaries. LLM-judged repetition guards and continuous-score benchmarks (HealthBench) introduce measure noise, but parallel-scaling effects are validated to be robust against judge unreliability.

Future research should investigate optimal inference scaling strategies, wider protocol variations, and more granular capability elicitation—especially as LLMs become more agentic and interactive. Extending these protocols to broader domains (e.g., RL agents, multimodal models) and integrating real-world feedback and safety constraints remains vital.

Conclusion

Performance assessment of frontier LLMs is inseparably linked to inference-time compute allocation and submission protocol. The investigated scaling interventions demonstrate that fixed-budget scores systematically understate model reach, reliability, and potential on complex benchmarks, particularly as models progress. Evaluation protocols must evolve to capture threshold unlocking, iterative search, and compute-limited headroom—establishing a more accurate and operation-relevant measure of AI capability (2606.17930).