SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification (2305.09781v4)

Abstract: This paper introduces SpecInfer, a system that accelerates generative LLM serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8x for distributed LLM inference and by 2.6-3.5x for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/

• The paper introduces SpecInfer, which employs tree-based speculative inference and small models to predict and verify LLM outputs efficiently.
• It demonstrates a parallel token tree decoding strategy that improves verification rates from 57% to 97%, achieving 1.5–3.5× speedups.
• SpecInfer optimizes resource utilization and scalability, paving the way for deploying large LLMs in production settings.

Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification

The paper introduces SpecInfer, a novel system designed to accelerate the serving of generative LLMs. SpecInfer leverages tree-based speculative inference and verification to significantly reduce the end-to-end latency and computational resource requirements for LLM inference. The main innovation of SpecInfer lies in its use of small speculative models (SSMs) to predict the outputs of LLMs and organize these predictions in a token tree, which is then verified against the LLM in parallel. This approach retains the generative performance of the LLM while accelerating the inference process.

Key Contributions

SpecInfer's primary contributions can be summarized as follows:

1. Token Tree-Based Speculative Inference:
   • SpecInfer introduces a tree-based approach rather than relying on sequence-based speculative inference. The speculative models generate a diverse set of candidate token sequences organized into a token tree. By considering multiple candidates simultaneously, SpecInfer maximizes the likelihood of aligning the speculative predictions with the LLM's outputs.
2. Tree-Based Parallel Decoding:
   • The system utilizes an innovative tree-based parallel decoding mechanism. SpecInfer transforms the problem of sequential token generation into parallel verification of multiple token sequences within a token tree. This reduces the dependency on sequential computation, thus lowering latency.
3. Multi-Step Speculative Sampling:
   • The system ensures that the probabilistic nature of stochastic decoding is preserved. SpecInfer's multi-step speculative sampling algorithm balances alignment accuracy and sampling efficiency, maintaining the model quality while improving verification success rates from approximately 57% to 97% for stochastic decoding.
4. Performance Improvement:
   • SpecInfer demonstrates substantial performance gains across various configurations:
     • Distributed Inference: The system outperforms existing LLM serving systems by 1.5-2.8× for distributed LLM inference.
     • Offloading-based Inference: SpecInfer achieves 2.6-3.5× speedup for offloading-based inference on a single GPU.

Practical and Theoretical Implications

The findings from SpecInfer's experiments imply several transformative improvements in the domain of LLM inference:

1. Efficiency in Resource Utilization:
   • By significantly reducing the number of LLM decoding steps, SpecInfer optimizes GPU and memory utilization. This is particularly crucial for large models like GPT-3 and GPT-4, which are known for their extensive parameter requirements and stringent memory demands.
2. Scalability:
   • SpecInfer's parallel decoding approach facilitates better scalability of LLM serving systems, making it feasible to deploy large LLMs in resource-constrained environments. The reduction in computational and memory overheads permits more efficient use of hardware, potentially lowering the barrier for deploying powerful models in production settings.
3. Implications for Future AI Developments:
   • This work sets a precedent for future research in speculative execution for machine learning applications. The strategies for model alignment and speculative sampling can be extended to other generative AI domains, inspiring new techniques for accelerating inference while ensuring output quality.

Detailed Experimental Analysis

The paper carefully evaluates SpecInfer using a range of LLMs and SSMs, including the LLaMA and OPT model families, on diverse datasets. The experimental setup covers various configurations, demonstrating the robustness of SpecInfer's design. For instance, using speculative widths and tree structures significantly improves the rate of successful token verification, highlighting the importance of speculative diversity. The empirical results exhibit compelling evidence of the predicted and verified tokens aligning with the LLM's natural outputs, a testament to the effectiveness of SpecInfer's methodology.

Conclusion and Future Directions

SpecInfer offers a promising approach to accelerate the inference of generative LLMs through speculative inference and token tree verification. By reshaping the inference process into a more parallelizable and efficient framework, SpecInfer stands as a significant step forward in serving large-scale AI models. Future research could explore dynamic token tree expansion strategies and further optimizations in speculative model tuning to refine the proposed system. Additionally, integrating SpecInfer's techniques with other optimization strategies like quantization and pruning could yield even more striking performance improvements, opening up new avenues for deploying advanced AI capabilities in real-world applications.