Efficiently Scaling LLM Reasoning with Certaindex (2412.20993v2)

Abstract: Test-time reasoning algorithms such as chain-of-thought, self-consistency, and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy. At the same time, we observe that these algorithms exhibit answer stabilization: their intermediate solutions often cease to change after a certain point, and further investment of compute does not change their final answer. To quantify this phenomenon, we introduce Certaindex, an algorithm-agnostic metric measuring this evolving stability, signaling when further computation is unlikely to alter the final result. Certaindex is lightweight, can accelerate reasoning program inference via early exit, and further enables dynamic token allocation, gang scheduling, and many opportunities when integrated with real-world LLM serving systems. To quantify real-world benefits, we built Certaindex as a scheduler into Dynasor, our reasoning-aware LLM serving system, and demonstrate up to 50% compute savings and 3.3x higher throughput in real workloads with no accuracy drop. Our code is available at https://github.com/hao-ai-lab/Dynasor.git

• The paper introduces Dynasor, which uses certaindex to dynamically allocate compute for LLM reasoning, reducing token usage by up to 50% in batch processing.
• It details a unified system design that integrates multiple reasoning algorithms with resource-aware, program-specific scheduling to sustain higher online request rates.
• Evaluations confirm that Dynasor outperforms state-of-the-art systems by effectively balancing accuracy, cost, and latency across diverse reasoning queries.

This paper introduces Dynasor, a novel system designed to efficiently serve LLM reasoning programs. The core challenge addressed is the inefficiency of existing serving systems in handling the compute-intensive nature of inference-time reasoning algorithms, which are crucial for advanced reasoning tasks. These algorithms, like Self-Consistency (SC), Monte Carlo Tree Search (MCTS), and Rebase, explore multiple solution paths, leading to increased compute demands and latency. Existing systems fail to adapt to the scaling behavior of these algorithms and the varying difficulty of queries, resulting in wasted resources and unmet latency targets.

Dynasor addresses these limitations by introducing a system that tracks and schedules requests within reasoning queries and uses a "certaindex," a proxy that measures statistical reasoning progress based on model certainty, to guide compute allocation dynamically. The system adaptively allocates more compute to difficult queries, reduces compute for simpler ones, and terminates unpromising queries early, balancing accuracy, latency, and cost.

The key contributions of the paper are:

1. Identification of Trade-offs: The paper identifies the crucial trade-offs between inference-time compute, question difficulty, and task latency and accuracy in serving reasoning queries. It introduces certaindex as an effective, simple, and general signal for reasoning progress across diverse tasks and algorithms.
2. Dynasor System Design: The paper presents the design of Dynasor, an adaptive inference-serving system that leverages certaindex as a key interface between its scheduler and diverse reasoning applications. Dynasor dynamically allocates compute and schedules reasoning requests across queries to optimize accuracy, cost, latency, and fairness.
3. Comprehensive Evaluation: Extensive evaluations across various datasets and reasoning algorithms demonstrate that Dynasor significantly outperforms state-of-the-art systems like SGLang and ParrotServe. It achieves up to 50% fewer tokens to reach similar accuracy in batch processing and sustains significantly higher request arrival rates and stricter latency SLOs in online serving.

Key Concepts and Techniques:

• LLM Reasoning Algorithms: The paper discusses several common reasoning algorithms (SC, Rebase, MCTS, ICoT), highlighting their inference-time scaling properties and a control parameter (knob) that governs the trade-off between compute and accuracy.
• Inference-Time Scaling: The paper emphasizes that increasing the computational budget (e.g., number of generated tokens) improves accuracy up to a point, but there are opportunities to optimize resource allocation based on query difficulty.
• Certaindex: The central idea is the use of "certaindex" as a measure of reasoning progress. It quantifies the LLM's certainty in approaching a final answer. It is calculated differently depending on the algorithm used. For example, in Self-Consistency, it can be the entropy of the generated answers. A higher certaindex suggests the LLM is closer to a solution or that the query may be unsolvable.
• Dynasor Architecture: Dynasor comprises three main components:
  • A Reasoning Program Abstraction that provides a unified interface for different reasoning algorithms, including methods for updating certaindex and executing the core logic.
  • An Application Runtime responsible for dynamically allocating resources based on certaindex values and implementing resource allocation policies.
  • A System Runtime managing request-level scheduling, including gang scheduling to prioritize requests from the same reasoning program and an approximation of Shortest Job First (SJF) scheduling.
• Resource Allocation Policies: The paper explores two main certaindex-based resource allocation policies: thresholding (terminating queries with certaindex above a threshold) and Pareto-frontier allocation (dynamically capping resources based on certaindex values).
• Program-Aware Scheduling: Dynasor's scheduler is aware of the parent LLM reasoning query of individual input/output sequences, enabling it to optimize for the end-to-end latency SLOs of those parent queries.

Experimental Results:

• Offline (Batch Processing): Dynasor reduces token usage by up to 50% while maintaining accuracy compared to baselines that allocate resources uniformly or based on output length.
• Online Serving: Dynasor achieves significantly higher sustainable request rates (up to 3.3x) and meets stricter SLOs (up to 4.7x tighter) compared to SGLang and ParrotServe.
• Ablation Studies: Ablation experiments confirm the effectiveness of both certaindex-based resource allocation and gang scheduling. The paper also analyzes the impact of different certaindex threshold values. The paper also performed experiments comparing certaindex with a length-based method and a predictor based on LLM activations.
• Fairness Analysis: Compared certaindex-based scheduling, gang scheduling, and SJF regarding finish-time fairness.

In conclusion, the paper presents a well-designed and thoroughly evaluated system that effectively addresses the challenges of serving LLM reasoning programs by dynamically allocating resources based on a novel "certaindex" metric.