Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2401.07851v3)

Abstract: To mitigate the high inference latency stemming from autoregressive decoding in LLMs, Speculative Decoding has emerged as a novel decoding paradigm for LLM inference. In each decoding step, this method first drafts several future tokens efficiently and then verifies them in parallel. Unlike autoregressive decoding, Speculative Decoding facilitates the simultaneous decoding of multiple tokens per step, thereby accelerating inference. This paper presents a comprehensive overview and analysis of this promising decoding paradigm. We begin by providing a formal definition and formulation of Speculative Decoding. Then, we organize in-depth discussions on its key facets, such as drafter selection and verification strategies. Furthermore, we present a comparative analysis of leading methods under third-party testing environments. We aim for this work to serve as a catalyst for further research on Speculative Decoding, ultimately contributing to more efficient LLM inference.

Introduction to Speculative Decoding

In the context of LLMs, efficiency during the inference phase is critical. Conventionally, autoregressive decoding, where tokens are generated one by one, has been the norm. However, this sequential generation leads to high latency, especially as the models and generated sequences grow larger. To address this challenge, Speculative Decoding has been introduced, offering a paradigm shift by first efficiently drafting several future tokens and then simultaneously verifying them.

Speculative Decoding Paradigm

Speculative Decoding stands out by allowing the simultaneous decoding of multiple tokens per step, which substantially accelerates inference. The paradigm employs two key strategies: drafting potential output tokens in advance using a "drafter" model and then validating these tokens in parallel with the target LLM. The drafter model is typically a smaller or specialized version of the LLM which can make predictions more quickly, albeit with potentially less accuracy. This drafted output is then screened meticulously, with only those tokens that pass the LLM's verification being accepted to ensure the overall quality of the sequence generated.

Technical Insights and Challenges

Despite its promise, Speculative Decoding opens up several technical questions, such as selecting or designing an appropriate drafter model to achieve a balance between speed and accuracy. Maintaining high-quality outputs while encouraging generation diversity is also critical. Integrating the drafter model with the target LLM is another hurdle that needs to be maneuvered for successful implementation. The field continues to evolve with various strategies explored to refine the speculative decoding process for maximum efficiency without compromising output quality.

Future Research Directions

Speculative Decoding is a rapidly expanding research area with a focus on improving LLM inference efficiency. As researchers continue to delve into speculative decoding, the main direction aims at achieving better alignment between the drafter and the target LLM for improved speculation accuracy. Moreover, there's an ongoing exploration of combining Speculative Decoding with other advanced techniques, expanding applications beyond text-only models into multimodal arenas. The ultimate goal is to catalyze future research in this domain for broader and more effective deployment of LLMs across various applications.