Inference Scaling fLaws: The Limits of LLM Resampling with Imperfect Verifiers (2411.17501v2)

Abstract: Recent research has generated hope that inference scaling could allow weaker LLMs to match or exceed the accuracy of stronger models, such as by repeatedly sampling solutions to a coding problem until it passes unit tests. The central thesis of this paper is that there is no free lunch for inference scaling: indefinite accuracy improvement through resampling can only be realized if the "verifier" (in this case, a set of unit tests) is perfect. When the verifier is imperfect, as it almost always is in domains such as reasoning or coding (for example, unit tests have imperfect coverage), there is a nonzero probability of false positives: incorrect solutions that pass the verifier. Resampling cannot decrease this probability, so it imposes an upper bound to the accuracy of resampling-based inference scaling even with an infinite compute budget. We find that there is a very strong correlation between the model's single-sample accuracy (i.e. accuracy without unit tests) and its false positive rate on coding benchmarks HumanEval and MBPP, whose unit tests have limited coverage. Therefore, no amount of inference scaling of weaker models can enable them to match the single-sample accuracy of a sufficiently strong model (Fig. 1a). When we consider that false positives have a negative utility compared to abstaining from producing a solution, it bends the inference scaling curve further downward. Empirically, we find that the optimal number of samples can be less than 10 under realistic assumptions (Fig. 1b). Finally, we show that beyond accuracy, false positives may have other undesirable qualities, such as poor adherence to coding style conventions.

• The paper shows that inference scaling is limited by the accuracy of verifiers, as imperfect checks lead to persistent false positives.
• The study demonstrates that weaker LLMs with lower single-sample accuracy incur higher false positive rates on benchmarks like HumanEval and MBPP.
• The authors highlight that optimal resampling and improved verifier design are essential for enhancing model performance and overall code quality.

Inference Scaling FLaws: The Limits of LLM Resampling with Imperfect Verifiers

This paper explores the domain of LLMs, particularly focusing on the concept of inference scaling, a technique proposed to improve model accuracy by resampling outputs and using verifiers to identify correctness. The paper posits a significant thesis: scaling accuracy indefinitely through resampling is constrained by the quality of the verifier. Specifically, when verifiers are imperfect—as they often are in complex tasks such as coding or reasoning—the probabilities of false positives (incorrectly accepted solutions) limit the performance improvements achievable by resampling, even with infinite computational resources.

Key Findings

The authors challenge the assumption that weaker LLMs can match the accuracy of stronger models through inference scaling alone. Their paper underscores several critical points:

1. Verifier Imperfection: An imperfect verifier, typified by common unit tests with limited coverage, cannot distinguish perfectly between correct and incorrect solutions. This imperfection introduces a persistent false positive rate that cannot be reduced through additional sampling.
2. Single-Sample Accuracy Correlation: The paper finds a strong correlation between a model's single-sample accuracy and its false positive rate on benchmarks like HumanEval and MBPP. Weaker models, defined by lower single-sample accuracy, are more prone to generating false positives.
3. Effect on Code Quality: Beyond functional correctness, false positives also negatively impact other code quality aspects, such as adherence to naming conventions and readability. This suggests a broader issue with the outputs generated under the influence of imperfect verifiers.
4. Optimal Sampling Techniques: Empirical analysis presented in the paper suggests that the optimal number of resampling attempts is often small. For tasks where false positives incur a significant cost, the advisable strategy might even be to avoid resampling altogether.

Implications and Speculations

The findings present several practical and theoretical implications:

• Verification as a Subfield: The necessity for robust verifiers highlights a potential avenue for research focused on improving their accuracy and reliability. This could involve developing verifiers as specialized components with distinct metrics and benchmarks.
• Training-time Verification: Feedback from imperfect verifiers during model training might lead models to exploit verifier weaknesses rather than truly solve tasks, raising safety and generalizability concerns.
• Model Strategy and Deployment: Given the constraints seen in using resampling as a strategy, the emphasis might shift towards either improving single-sample performance through other means or bolstering the verification process.
• Future Research Directions: The contrast between verification problems in controlled benchmarks versus real-world applications suggests that future research could explore creating more realistic evaluation environments. Moreover, integrating inference scaling with fine-tuning strategies might provide a path to improving model performance without over-relying on computational resources.

Conclusion

This paper brings to light critical limitations in the application of inference scaling in LLMs, tempered by the accuracy of verifiers. The research emphasizes that repeated resampling alone cannot bridge the performance gap between weaker and stronger models when verifiers are not reliable. These findings invite the community to reconsider the balance between computational resource allocation and verifier design, as well as to explore novel directions for enhancing both the training and inference capabilities of LLMs in diverse application domains.