Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
72 tokens/sec
GPT-4o
61 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LLM Inference Unveiled: Survey and Roofline Model Insights (2402.16363v6)

Published 26 Feb 2024 in cs.CL and cs.AI

Abstract: The field of efficient LLM inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise framework that analyzes the various methods of LLM Inference to provide a clear understanding of this domain. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems, such as why LLMs are memory-bound, how much memory and computation they need, and how to choose the right hardware. We systematically collate the latest advancements in efficient LLM inference, covering crucial areas such as model compression (e.g., Knowledge Distillation and Quantization), algorithm improvements (e.g., Early Exit and Mixture-of-Expert), and both hardware and system-level enhancements. Our survey stands out by analyzing these methods with roofline model, helping us understand their impact on memory access and computation. This distinctive approach not only showcases the current research landscape but also delivers valuable insights for practical implementation, positioning our work as an indispensable resource for researchers new to the field as well as for those seeking to deepen their understanding of efficient LLM deployment. The analyze tool, LLM-Viewer, is open-sourced.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (158)
  1. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.13649.
  2. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding.
  3. N-ode transformer: A depth-adaptive variant of the transformer using neural ordinary differential equations.
  4. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774.
  6. Dialogue chain-of-thought distillation for commonsense-aware conversational agents. arXiv preprint arXiv:2310.09343.
  7. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems.
  8. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.
  9. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism.
  10. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085.
  11. Palm: Scaling language modeling with pathways.
  12. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference.
  13. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28.
  14. Fido: Fusion-in-decoder optimized for stronger performance and faster inference.
  15. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339.
  16. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.
  17. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078.
  18. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pages 7750–7774. PMLR.
  19. Bert: Pre-training of deep bidirectional transformers for language understanding.
  20. Matformer: Nested transformer for elastic inference.
  21. Jump to conclusions: Short-cutting transformers with linear transformations.
  22. The efficiency spectrum of large language models: An algorithmic survey. arXiv preprint arXiv:2312.00678.
  23. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293–302.
  24. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547–5569. PMLR.
  25. Depth-adaptive transformer.
  26. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667.
  27. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.(2021). arXiv preprint cs.LG/2101.03961.
  28. Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML.
  29. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.
  30. Breaking the sequential dependency of llm inference using lookahead decoding.
  31. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726.
  32. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.
  33. Mask-predict: Parallel decoding of conditional masked language models.
  34. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pages 291–326. Chapman and Hall/CRC.
  35. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789–1819.
  36. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281.
  37. Fully non-autoregressive neural machine translation: Tricks of the trade.
  38. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543.
  39. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–15.
  40. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023.
  41. Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376–385, Online. Association for Computational Linguistics.
  42. Compresso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015.
  43. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252.
  44. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  45. Training compute-optimal large language models.
  46. Dynabert: Dynamic bert with adaptive width and depth.
  47. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301.
  48. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  49. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933.
  50. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291.
  51. In-context learning distillation: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.10670.
  52. Binarized neural networks. Advances in neural information processing systems, 29.
  53. Automoe: Heterogeneous mixture-of-experts with adaptive computation for efficient neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9116–9132.
  54. Early exit with disentangled representation and equiangular tight frame. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 14128–14142, Toronto, Canada. Association for Computational Linguistics.
  55. Mixtral of experts. arXiv preprint arXiv:2401.04088.
  56. Lion: Adversarial distillation of closed-source large language model. arXiv preprint arXiv:2305.12870.
  57. Scaling laws for neural language models.
  58. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint arXiv:2305.14152.
  59. Token-scaled logit distillation for ternary weight generative language models. arXiv preprint arXiv:2308.06744.
  60. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629.
  61. Full stack optimization of transformer inference: a survey.
  62. Speculative decoding with big little decoder.
  63. Literature survey on low rank approximation of matrices. Linear and Multilinear Algebra, 65(11):2212–2244.
  64. Optimizing mixture of experts using dynamic recompilations. arXiv preprint arXiv:2205.01848.
  65. Copy is all you need.
  66. Optimal brain damage. Advances in neural information processing systems, 2.
  67. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.02272.
  68. Deterministic non-autoregressive neural sequence modeling by iterative refinement.
  69. Sparse mixers: Combining moe and mixing to build a more efficient bert. arXiv preprint arXiv:2205.12399.
  70. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668.
  71. Fast inference from transformers via speculative decoding.
  72. Norm tweaking: High-performance low-bit quantization of large language models. arXiv preprint arXiv:2309.02784.
  73. Cascadebert: Accelerating inference of pre-trained language models via calibrated complete models cascade.
  74. Symbolic chain-of-thought distillation: Small models can also” think” step-by-step. arXiv preprint arXiv:2306.14050.
  75. Fptq: Fine-grained post-training quantization for large language models. arXiv preprint arXiv:2308.15987.
  76. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077.
  77. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659.
  78. Hint-based training for non-autoregressive machine translation. arXiv preprint arXiv:1909.06708.
  79. Qft: Quantized full-parameter tuning of llms with affordable resources. arXiv preprint arXiv:2310.07147.
  80. Homodistil: Homotopic task-agnostic distillation of pre-trained transformers. arXiv preprint arXiv:2302.09632.
  81. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pages 20852–20867. PMLR.
  82. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403.
  83. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978.
  84. Apar: Llms can do auto-parallel auto-regressive decoding.
  85. Fastbert: a self-distilling bert with adaptive inference time.
  86. Online speculative decoding. arXiv preprint arXiv:2310.07177.
  87. Towards efficient NLP: A standard evaluation and a strong baseline. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3288–3303, Seattle, United States. Association for Computational Linguistics.
  88. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888.
  89. Deja vu: Contextual sparsity for efficient llms at inference time.
  90. Llm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11627.
  91. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234.
  92. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781.
  93. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13581.
  94. Skeleton-of-thought: Large language models can do parallel decoding.
  95. Gpt-4 technical report.
  96. Iterative solution of nonlinear equations in several variables. SIAM.
  97. Oseledets, I. V. (2011). Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295–2317.
  98. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  99. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. arXiv preprint arXiv:2206.09557.
  100. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International Conference on Machine Learning, pages 18332–18346. PMLR.
  101. Finding the sweet spot: Analysis and improvement of adaptive inference in low resource settings.
  102. Promptmix: A class boundary augmentation method for large language model distillation. arXiv preprint arXiv:2310.14192.
  103. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427.
  104. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.06749.
  105. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
  106. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. Advances in Neural Information Processing Systems, 35:20051–20063.
  107. Confident adaptive language modeling.
  108. Consistent accelerated inference via confident adaptive transformers.
  109. The right tool for the job: Matching model and instance complexities. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640–6651, Online. Association for Computational Linguistics.
  110. Lipschitz continuity guided knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10675–10684.
  111. Pb-llm: Partially binarized large language models. In ICLR.
  112. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 198–205.
  113. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558.
  114. How many layers and why? An analysis of the model depth in transformers. In Kabbara, J., Lin, H., Paullada, A., and Vamvas, J., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 221–228, Online. Association for Computational Linguistics.
  115. Accelerating feedforward computation via parallel nonlinear equation solving. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9791–9800. PMLR.
  116. Powerinfer: Fast large language model serving with a consumer-grade gpu.
  117. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31.
  118. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning.
  119. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695.
  120. Fast structured decoding for sequence models. Advances in Neural Information Processing Systems, 32.
  121. Spectr: Fast speculative decoding via optimal transport. arXiv preprint arXiv:2310.15141.
  122. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  123. Llama 2: Open foundation and fine-tuned chat models.
  124. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  125. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558.
  126. Scott: Self-consistent chain-of-thought distillation. arXiv preprint arXiv:2305.01879.
  127. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748.
  128. Non-autoregressive machine translation with auxiliary regularization. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5377–5384.
  129. Imitation learning for non-autoregressive neural machine translation. arXiv preprint arXiv:1906.02041.
  130. Lamini-lm: A diverse herd of distilled models from large-scale instructions. arXiv preprint arXiv:2304.14402.
  131. Ad-kd: Attribution-driven knowledge distillation for language model compression. arXiv preprint arXiv:2305.10010.
  132. Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats. arXiv preprint arXiv:2307.09782.
  133. Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309.10285.
  134. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694.
  135. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR.
  136. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  137. Deebert: Dynamic early exiting for accelerating bert inference.
  138. Llmcad: Fast and scalable on-device large language model inference. arXiv preprint arXiv:2309.04255.
  139. Tensorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition. arXiv preprint arXiv:2307.00526.
  140. A survey on knowledge distillation of large language models.
  141. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717.
  142. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487.
  143. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183.
  144. Edgemoe: Fast on-device inference of moe-based large language models. arXiv preprint arXiv:2308.14352.
  145. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175.
  146. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089.
  147. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821.
  148. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12065.
  149. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177–1193.
  150. Consistentee: A consistent and hardness-guided early exiting method for accelerating language models inference.
  151. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168.
  152. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  153. A survey of large language models. arXiv preprint arXiv:2303.18223.
  154. Bert loses patience: Fast and robust inference with early exit.
  155. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461.
  156. Zhu, W. (2021). LeeBERT: Learned early exit for BERT with cross-level optimization. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2968–2980, Online. Association for Computational Linguistics.
  157. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633.
  158. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (14)
  1. Zhihang Yuan (45 papers)
  2. Yuzhang Shang (35 papers)
  3. Yang Zhou (311 papers)
  4. Zhen Dong (87 papers)
  5. Chenhao Xue (9 papers)
  6. Bingzhe Wu (58 papers)
  7. Zhikai Li (24 papers)
  8. Qingyi Gu (25 papers)
  9. Yong Jae Lee (88 papers)
  10. Yan Yan (241 papers)
  11. Beidi Chen (61 papers)
  12. Guangyu Sun (47 papers)
  13. Kurt Keutzer (199 papers)
  14. Zhe Zhou (33 papers)
Citations (45)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets