Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CLLMs: Consistency Large Language Models (2403.00835v4)

Published 28 Feb 2024 in cs.CL and cs.AI
CLLMs: Consistency Large Language Models

Abstract: Parallel decoding methods such as Jacobi decoding show promise for more efficient LLM inference as it breaks the sequential nature of the LLM decoding process and transforms it into parallelizable computation. However, in practice, it achieves little speedup compared to traditional autoregressive (AR) decoding, primarily because Jacobi decoding seldom accurately predicts more than one token in a single fixed-point iteration step. To address this, we develop a new approach aimed at realizing fast convergence from any state to the fixed point on a Jacobi trajectory. This is accomplished by refining the target LLM to consistently predict the fixed point given any state as input. Extensive experiments demonstrate the effectiveness of our method, showing 2.4$\times$ to 3.4$\times$ improvements in generation speed while preserving generation quality across both domain-specific and open-domain benchmarks.

PDF HTML Abstract

Consistency LLMs (CLLMs): Enhancing Efficiency in LLM Inference

The paper introduces Consistency LLMs (CLLMs), aimed at optimizing the inference speed of LLMs through an innovative approach called Jacobi decoding. While traditional autoregressive (AR) decoding has been the standard for LLMs, its sequential nature often leads to high latency, particularly when generating long responses. Jacobi decoding, in contrast, offers a parallelizable alternative that has the potential to significantly reduce inference time.

Understanding the Bottleneck in Existing Parallel Decoding

Jacobi decoding involves processing token sequences in parallel, predicting an entire sequence at once rather than updating one token at a time. Despite this theoretical efficiency, practical speedups have been limited due to LLMs' inability to accurately predict multiple tokens in a single iteration due to dependencies created by the attention mechanisms.

The Innovation of CLLMs

The authors propose refining the LLMs to consistently predict the fixed point of a Jacobi trajectory from any input state. This approach modifies the model to predict multiple correct tokens at once, enhancing the efficacy of Jacobi decoding.

  1. Training Methodology: CLLMs are trained using a dataset generated from Jacobi trajectories exhibiting states ranging from initial guesses to fixed points. Two types of consistency losses are employed:
    • Global Consistency Loss: Directly maps any point in the Jacobi trajectory to its fixed point.
    • Local Consistency Loss: Ensures that consecutive points in the trajectory map to each other, implicitly guiding the sequence towards the fixed point.
  2. Empirical Results: The experimental results are substantial, reporting improvements in generation speed by factors of 2.4×\times to 3.4×\times across various benchmarks, all while retaining generation quality.

Implications and Observations

The acceleration stems from two phenomena identified in CLLMs:

  • Fast Forwarding: The ability to predict several correct subsequent tokens in one step.
  • Stationary Tokens: Correctly predicted tokens that remain fixed despite incorrect preceding tokens.

These capabilities imply that CLLMs have learned implicit linguistic structures or collocations that are predictable in groups rather than individually. This discovery might not only expedite inference but also provide insights into more efficient model training and design.

Comparative Analysis

CLLMs present an advantageous alternative to existing methods such as speculative decoding and certain architectural augmentations. They do not necessitate additional model components or significant architectural changes, thus maintaining memory efficiency and ease of integration into existing systems.

Future Directions

This research proposal can potentially influence both theoretical and practical domains:

  • Theoretical: Enhancing understanding of parallel token prediction and implicit language structures within LLMs.
  • Practical: Enabling faster LLM inference could transform real-time applications across industries where latency is critical.

In conclusion, CLLMs represent a significant step towards more efficient LLM deployment. Their ability to accelerate inference without compromising on quality is particularly promising for large-scale applications. Future work could explore extending these methods to various LLM architectures and further refining the training process to accommodate diverse linguistic patterns.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.13649, 2023.
  3. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023.
  4. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  5. Slicegpt: Compress large language models by deleting rows and columns, 2024.
  6. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.
  7. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024.
  8. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
  9. Generative ai for math: Abel. URL https://github.com/GAIR-NLP/abel.
  10. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  11. Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
  12. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022.
  13. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022.
  14. Sparsegpt: Massive language models can be accurately pruned in one-shot. 2023.
  15. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
  16. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057, 2024.
  17. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023.
  18. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  19. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019.
  20. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019.
  21. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947, 2016.
  22. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.  611–626, 2023.
  23. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.  19274–19286. PMLR, 2023.
  24. Eagle: Speculative sampling requires rethinking feature uncertainty, 2024.
  25. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023.
  26. Online speculative decoding, 2023.
  27. Llm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11627, 2023.
  28. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  29. Iterative solution of nonlinear equations in several variables. SIAM, 2000.
  30. Meta-kd: A meta knowledge distillation framework for language model compression across domains. arXiv preprint arXiv:2012.01266, 2020.
  31. Sentence repetition: What does the task measure? International Journal of Language & Communication Disorders, 50(1):106–118, 2015.
  32. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023.
  33. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427, 2023.
  34. Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.
  35. Smadja, F. From n-grams to collocations: An evaluation of xtract. In 29th Annual Meeting of the Association for Computational Linguistics, pp.  279–284, 1991.
  36. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189, 2023.
  37. Accelerating feedforward computation via parallel nonlinear equation solving. In International Conference on Machine Learning, pp.  9791–9800. PMLR, 2021a.
  38. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. URL https://openreview.net/forum?id=PxTIG12RRHS.
  39. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  40. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023.
  41. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  42. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  43. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023.
  44. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408, 2022.
  45. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023.
  46. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.  38087–38099. PMLR, 2023.
  47. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887, 2018.
  48. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
  49. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
  50. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461, 2023b.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Siqi Kou (8 papers)
  2. Lanxiang Hu (9 papers)
  3. Zhezhi He (31 papers)
  4. Zhijie Deng (58 papers)
  5. Hao Zhang (947 papers)
Citations (16)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube