Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2203.16487v6)

Abstract: We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around $5\times$ speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

Speculative Decoding: Utilizing Speculative Execution for Speeding Up Seq2seq Generation

The field of sequence-to-sequence (seq2seq) generation is an essential component within NLP, and the Transformer architecture has become the backbone for numerous applications such as machine translation and abstractive summarization. Nonetheless, the efficiency of Transformer’s autoregressive (AR) decoding is hampered by limited parallelism, resulting in significant latency and computation costs when deployed in real-time scenarios. The paper introduces Speculative Decoding (SpecDec), aiming to significantly enhance seq2seq generation speed by drawing inspiration from speculative execution techniques used in computer architectures.

Overview of Speculative Decoding

Speculative Decoding encapsulates two primary components: Spec-Drafter and Spec-Verification. Spec-Drafter is an independent model meticulously optimized to draft output sequences efficiently and accurately. Spec-Verification, on the other hand, reliably corroborates the drafted tokens, ensuring fidelity to the generation quality comparable to beam search decoding.

To empirically validate this methodology, the authors conduct extensive experiments on multiple seq2seq tasks, including machine translation across English-German and English-Romanian datasets, as well as abstractive summarization. The outcomes indicate that SpecDec achieves a performance speedup of approximately $5\times$ over traditional Transformer architectures. This outcome significantly surpasses previous draft-then-verify techniques, which only yielded a speedup between $1.4\times$ and $2.0\times$. Furthermore, SpecDec maintains robust generation quality, contrapuntally challenging the notion that the draft-then-verify paradigm offers limited acceleration potential.

Innovations in Speculative Decoding

• Spec-Drafter: The Spec-Drafter is designed following two core principles. The Capability Principle ensures its competence in producing accurate drafts, while the Latency Principle focuses on minimizing iteration latency. This design employs a deep encoder and shallow decoder architecture, making it a lightweight yet highly effective drafting model.
• Spec-Verification: The verification strategy is enhanced beyond strict AR top-1 matching, allowing drafted tokens to be different yet close to top-1 results. This modification trusts high-quality drafts more, thus embracing higher parallelism in verification.

These components collectively yield significant improvements in decoding speed without sacrificing the quality of seq2seq tasks.

Contributions and Implications for Future AI Research

The introduction of Speculative Decoding brings forth several practical implications:

• Real-World Applicability: The $5\times$ speedup facilitates the deployment of computationally-intensive Transformer models for real-time applications where quick responses and cost savings are critical.
• Theoretical Advancement: By redefining the draft-then-verify paradigm, the research opens avenues for subsequent improvements and adaptations of speculative execution within NLP and beyond.
• Future Developments: Speculative Decoding's success prompts further investigation into speculative execution techniques to enhance other facets of Transformer models, possibly combining them with cutting-edge parallel computing developments.

While the paper delivers impressive numerical results, it also provokes further exploration of speculative execution as a viable path towards optimized computational resource utilization in AI systems. The paper demonstrates that there is substantial untapped potential in such paradigms to vastly improve the efficiency of state-of-the-art LLMs.