An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint (2504.14350v3)

Abstract: Recent work has demonstrated the remarkable potential of LLMs in test-time scaling. By making models think before answering, they are able to achieve much higher accuracy with extra inference computation. However, in many real-world scenarios, models are used under time constraints, where an answer should be given within a certain output length. It is unclear whether and how the reasoning ability of different LLMs remain effective under strict constraints. We take a first look at this problem by conducting an in-depth empirical study. Specifically, we test 30 LLMs on common reasoning datasets under a wide range of output length budgets, and we analyze the correlation between the inference accuracy and various properties including model type, model size, prompt style, etc. We also consider the mappings between token budgets and actual on-device latency budgets. The results have demonstrated several interesting findings regarding the budget-aware LLM reasoning ability that differ from the unconstrained situation, e.g. the optimal choices of either model size or prompt style change under different budgets. These findings offer timely evaluation to this area and practical guidance for users to deploy LLMs under real-world latency constraints.

Time-Constrained Reasoning in LLMs: An Empirical Study

The paper "Time Up! An Empirical Study of LLM Reasoning Ability Under Output Length Constraint" presents a thorough examination of the reasoning capabilities of LLMs under output length constraints. In contexts where models have to provide quick, timed responses, the intricate relationship between model size, prompt design, and reasoning performance necessitates further exploration. This study aims to fill this gap by evaluating various open-source models across established reasoning benchmarks.

Core Investigations and Methodology

The authors investigate over 25 LLMs on datasets such as GSM8K and MATH500, employing diverse prompting styles—step-by-step (sbs), coarse-to-fine (c2f), and answer-and-verify (aav)—to maximize effectiveness under token limitations. Two primary methods are introduced: direct termination at token budget, and an innovative early stopping approach. The latter involves constructive intervention before reaching the token limit, allowing models to formulate a structured conclusion even with truncated inference processes.

Key Findings

1. Impact of Early Stopping: Across different datasets and prompting styles, early stopping consistently enhances LLM reasoning performance versus direct termination. This improvement highlights the significance of structured conclusion incentives, which allow models to deliver meaningful answers without completing extended reasoning chains.
2. Prompt Design and Its Effectiveness: While no singular prompt style universally excels, c2f and aav prompts often outperform sbs under token constraints. This suggests that strategies encouraging succinct preliminary answers could improve reasoning efficiency, particularly for models predisposed to generating lengthy outputs.
3. Model Size vs. Performance: Counter to traditional scaling laws, larger models do not uniformly surpass smaller ones under strict token budgets. This anomaly indicates potential inefficiencies in larger models' reasoning processes when concise thinking is required, urging a reevaluation of proportionate scaling in relation to reasoning capabilities.
4. Specialized vs. Instruction-Tuned Models: In several scenarios, generic instruction-tuned models outperform those specialized in reasoning tasks when operating under token constraints. The paper notes that sometimes concise and direct reasoning, typical of instruction-tuned models, is preferable within stringent output limits.
5. Latency Considerations: Mid-sized models often offer optimal performance in latency-sensitive settings, making them preferable for deployments with real-world time constraints despite available resources to support larger models.

Implications and Future Directions

The findings provide practical insights for deploying LLMs in environments with tight latency and token constraints, such as real-time decision-making applications. Moreover, they challenge existing paradigms about the role of model size in determining reasoning efficiency and open pathways for prompt engineering tailored to varied reasoning demands.

This study advocates for further exploration into adaptive reasoning techniques that account for the inherent challenge of reasoning under temporal limitations. Future research may explore domain-specific applications, widen the scope of prompt stylistic variations, and refine tuning strategies that optimize the balance between inference depth and output brevity.

The authors propose that as LLMs continue to evolve, understanding and optimizing their behavior in constrained environments will be pivotal, offering benefits that extend beyond academic benchmarks to impactful real-world tasks. This paper serves as a foundational step towards comprehensively characterizing the compute-constrained reasoning abilities of LLMs.