Optimizing Speculative Decoding for Serving Large Language Models Using Goodput (2406.14066v2)

Abstract: Reducing the inference latency of LLMs is crucial, and speculative decoding (SD) stands out as one of the most effective techniques. Rather than letting the LLM generate all tokens directly, speculative decoding employs effective proxies to predict potential outputs, which are then verified by the LLM without compromising the generation quality. Yet, deploying SD in real online LLM serving systems (with continuous batching) does not always yield improvement -- under higher request rates or low speculation accuracy, it paradoxically increases latency. Furthermore, there is no best speculation length work for all workloads under different system loads. Based on the observations, we develop a dynamic framework SmartSpec. SmartSpec dynamically determines the best speculation length for each request (from 0, i.e., no speculation, to many tokens) -- hence the associated speculative execution costs -- based on a new metric called goodput, which characterizes the current observed load of the entire system and the speculation accuracy. We show that SmartSpec consistently reduces average request latency by up to 3.2x compared to non-speculative decoding baselines across different sizes of target models, draft models, request rates, and datasets. Moreover, SmartSpec can be applied to different styles of speculative decoding, including traditional, model-based approaches as well as model-free methods like prompt lookup and tree-style decoding.

Analyzing Speculative Decoding Optimization via Goodput in LLMs

The paper "Optimizing Speculative Decoding for Serving LLMs Using Goodput" presents an insightful analysis of speculative decoding (SD) methods applied to LLMs in real-time serving systems. The primary focus of this work is to mitigate inference latency—a pivotal performance aspect for applications such as search engines, chatbots, and virtual assistants. The authors propose a dynamic framework, SmartSpec, which utilizes a novel performance metric termed "goodput," to optimize speculative decoding under varying workloads and system demands.

Speculative Decoding and its Challenges

Speculative decoding aims to overcome the sequential data dependency bottleneck inherent in autoregressive model generation. It uses lightweight proxy models to propose multiple output tokens, which are then concurrently evaluated for acceptance by a more computationally intensive target model. While this procedure can significantly reduce token generation latency, its efficacy in practice is conditional. With high request rates or low speculative accuracy, SD could potentially inflate latency due to increased computational overhead in evaluating unaccepted tokens.

Goodput: A New Metric for Latency Optimization

The paper introduces the concept of goodput, defined as the number of successfully generated tokens per second that are accepted and verified by the target model. This metric surpasses traditional throughput metrics by factoring in token acceptance accuracy and actual system load conditions. Goodput, as delineated, is affected by proposed lengths and batch size optimizations, offering actionable insights into optimizing speculative execution.

SmartSpec: Dynamic Optimization Framework

SmartSpec leverages goodput for real-time adaptive control over speculative decoding lengths. This framework dynamically adjusts speculation according to current computational availability and historical token acceptance rates. Such adaptivity allows SmartSpec to maximize performance in varying system environments—either by speculating less under high load conditions or increasing speculation when system resources permit.

Evaluation and Possibilities

The evaluation results demonstrate SmartSpec's efficacy across numerous configurations: different LLM sizes, speculative decoding strategies both draft-based and model-free, and task-specific workloads. A significant 3.2$\times$ reduction in latency is noted under optimal conditions, reinforcing the strengths of adaptive speculation guided by system-specific goodput. Furthermore, through integrating SD with existing production systems like vLLM, the practicality of SmartSpec for real-world, latency-sensitive deployments is underscored.

Implications and Future Directions

The implications of this paper are twofold: firstly, emphasizing the dynamic nature of computationally efficient serving models, and secondly, positing goodput as a tailored metric for aligning speculative decoding efficiency with heterogeneous workload demands. Future directions may explore further refinement of token acceptance predictions, enhancing the granularity of control in SmartSpec, and broadening its applicability to advanced speculative techniques and newer LLM architectures. Additionally, reinforcing goodput's predictive accuracy with advanced learning-based models could drive further latency improvements.

In closing, this paper offers a comprehensive examination of the symbiotic relationship between speculative decoding strategies and system-level resource management, advocating for adaptive inference frameworks to improve LLM latency performance in practice.