Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification (2502.01839v2)

Abstract: Sampling-based search, a simple paradigm for utilizing test-time compute, involves generating multiple candidate responses and selecting the best one -- typically by having models self-verify each response for correctness. In this paper, we study the scaling trends governing sampling-based search. Among our findings is that simply scaling up a minimalist implementation of sampling-based search, using only random sampling and direct self-verification, provides a practical inference method that, for example, elevates the reasoning capabilities of Gemini v1.5 Pro above that of o1-Preview on popular benchmarks. We partially attribute the scalability of sampling-based search to a phenomenon of implicit scaling, where sampling a larger pool of responses in turn improves self-verification accuracy. We further identify two useful principles for improving self-verification capabilities with test-time compute: (1) comparing across responses provides helpful signals about the locations of errors and hallucinations, and (2) different model output styles are useful for different contexts -- chains of thought are useful for reasoning but harder to verify. We also find that, though accurate verification can be elicited, frontier models demonstrate remarkably weak out-of-box verification capabilities and introduce a benchmark to measure progress on these deficiencies.

• The paper demonstrates that scaling a minimalist sampling-based search strategy with self-verification yields continuous performance gains in language model reasoning tasks through implicit scaling.
• Authors propose two strategies to improve verification, such as comparing candidates, and highlight that current frontier models still show inadequate out-of-box verification abilities requiring new benchmarks.
• The research emphasizes the practical advantages of sampling-based search due to its embarrassingly parallel nature and its increasing importance for tackling complex problems with growing test-time compute resources.

The paper "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" introduces a paper on scaling trends in sampling-based search, a paradigm that leverages test-time compute for improving the performance of LLMs. The authors explore the concept of sampling-based search, which involves generating multiple candidate responses at inference-time and selecting the most appropriate one by verifying each response for correctness. Their investigation highlights several critical observations regarding the scaling behavior of such methodologies.

Key findings include:

1. Performance Improvements through Scaling: The paper demonstrates that scaling a minimalist implementation of sampling-based search, which relies on random sampling and direct self-verification, yields continuous performance gains. Notably, the Gemini v1.5 Pro model surpasses the o1-Preview model's reasoning capabilities on well-known benchmarks like LiveBench and AIME due to sampling-based search. The improvement is attributed to an implicit effect known as "implicit scaling," where evaluating a larger pool of generated responses enhances verification accuracy.
2. Verification Strategies: The authors identify two strategies to bolster verification capabilities during test-time:
   • Comparing candidate responses to derive helpful signals about error locations.
   • Adopting different model output styles for various contexts, such as using chain-of-thought for reasoning, which are beneficial for generation but complex for verification.
3. Verification Weakness in Frontier Models: Despite scaling trends, the current frontier models exhibit inadequate out-of-box verification abilities. To address this, the paper introduces a benchmark aimed at measuring and tracking progress in overcoming these deficiencies.
4. Scaling Test-Time Compute: The research further elucidates the potential advantages of using sampling-based search, particularly due to its embarrassingly parallel nature. This method becomes increasingly crucial as compute availability for inference tasks continues to grow, particularly for solving intricate mathematical and scientific problems.
5. Technical Insights: Insights into efficient implementation are shared through detailed evaluation of search and verification trends across different datasets. Experiments reveal that sampling more responses (search scaling) or verification (verification scaling) leads to non-trivial power-law scaling in reasoning tasks.

The overall contribution of this work lies in highlighting both the scalability and practicality of sampling-based search. It extends the understanding of how intricate trade-offs between search space exploration and verification fidelity can be managed to optimize reasoning performance, especially when verification is scaled effectively with additional test-time compute resources. The paper emphasizes the need for systematic strategies beyond random sampling and simpler verification when working towards large-scale applications and establishing robust baselines for model evaluation.