Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge (2405.00263v1)

Abstract: LLMs suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically, billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation. Recently, parallel decoding, a type of speculative decoding algorithms, is becoming more popular and has demonstrated impressive efficiency improvement in generation. It introduces extra decoding heads to large models, enabling them to predict multiple subsequent tokens simultaneously and verify these candidate continuations in a single decoding step. However, this approach deviates from the training objective of next token prediction used during pre-training, resulting in a low hit rate for candidate tokens. In this paper, we propose a new speculative decoding algorithm, Clover, which integrates sequential knowledge into the parallel decoding process. This enhancement improves the hit rate of speculators and thus boosts the overall efficiency. Clover transmits the sequential knowledge from pre-speculated tokens via the Regressive Connection, then employs an Attention Decoder to integrate these speculated tokens. Additionally, Clover incorporates an Augmenting Block that modifies the hidden states to better align with the purpose of speculative generation rather than next token prediction. The experiment results demonstrate that Clover outperforms the baseline by up to 91% on Baichuan-Small and 146% on Baichuan-Large, respectively, and exceeds the performance of the previously top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on Baichuan-Large, respectively.

Exploring SeqarHead: Enhancing Speculative Decoding in LLMs

The Problem with Existing Speculative Decoding Techniques

In the world of AI and machine learning, efficiency remains a key challenge, especially when deploying LLMs like GPT on GPUs. These models are notorious for their "chatty" nature during the decoding phase, where they generate one token at a time, making inefficient use of GPU capabilities. This inefficiency primarily arises from the mismatch between auto-regressive token generation and the GPU’s parallel processing strengths.

Recent advancements have aimed to rectify this through speculative decoding techniques, such as the Medusa system, which introduces parallelism by speculating multiple future tokens at once. However, there’s a catch: these systems often neglect the sequential dependencies crucial for maintaining the contextual accuracy of generated texts, leading to a lower hit rate of correct token predictions.

Introducing SeqarHead

SeqarHead proposes to take speculative decoding a notch higher by integrating what's known as a Regressive Connection, an Attention Decoder, and an Augmenting Block. The essence here is to not only predict multiple future tokens at once but also ensure that these predictions respect the sequential flow of information, which is paramount for context coherence in text generation.

Core Components of SeqarHead

• Regressive Connection: This feature allows the model to consider tokens that have already been speculated when predicting the next ones. This means each new token prediction carries forward the context from its predecessors, unlike in systems like Medusa where each token is predicted in isolation.
• Attention Decoder: This component operates at the heart of SeqarHead, effectively merging the speculated tokens’ influences with the ongoing inputs. It ensures that the sequential dependencies are not just carried over but are actively influencing the next token predictions.
• Augmenting Block: Positioned as an enhancement tool within the LLM, this block tweaks the hidden states of the model such that they are better aligned for predictive tasks that extend beyond the next immediate token.

Performance Gains

The practical benefits of SeqarHead are robust, as evidenced by exhaustive testing on models of different sizes. For instance, when deployed on Baichuan-Small and Baichuan-Large models, SeqarHead outperformed the existing speculative decoding benchmarks significantly, achieving improvements of up to 146% over baseline predictions in larger models. Not only does it enhance the speed (tokens per second), but it also boosts the accuracy and the number of correctly predicted tokens in extended sequences.

Future Implications and Developments

The introduction of SeqarHead is a promising step toward more efficient use of hardware in deploying LLMs, especially in real-time scenarios where speed and accuracy are crucial. The ability to maintain context integrity while speculating multiple tokens could pave the way for more interactive and instantaneous AI-driven applications.

Moreover, as AI research delves deeper into the realms of efficiency and effectiveness, techniques like SeqarHead set the stage for speculative decoding to evolve into a more context-aware, intelligent process. This could potentially reduce the computational overheads associated with LLMs while ensuring that the generative capabilities of these models are not compromised.

In essence, SeqarHead not only advances the technical scope of LLM efficiency but also aligns practical AI deployments closer to real-time responsiveness and contextual accuracy, enriching the interaction between humans and AI-generated content.