Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CBQ: Cross-Block Quantization for Large Language Models (2312.07950v4)

Published 13 Dec 2023 in cs.LG and cs.CL

Abstract: Post-training quantization (PTQ) has played a key role in compressing LLMs with ultra-low costs. However, existing PTQ methods only focus on handling the outliers within one layer or one block, which ignores the dependency of blocks and leads to severe performance degradation in low-bit settings. In this paper, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. CBQ employs a cross-block dependency using a homologous reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation. Furthermore, CBQ incorporates a coarse-to-fine preprocessing (CFP) strategy for suppressing weight and activation outliers, coupled with an adaptive LoRA-Rounding technique for precise weight quantization. These innovations enable CBQ to not only handle extreme outliers effectively but also improve overall quantization accuracy. Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across various LLMs and datasets. Notably, CBQ quantizes the 4-bit LLAMA1-65B model within only 4.3 hours on a single GPU, achieving a commendable tradeoff between performance and quantization efficiency.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.  7432–7439, 2020a.
  2. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, pp.  7432–7439, Jun 2020b. doi: 10.1609/aaai.v34i05.6239. URL http://dx.doi.org/10.1609/aaai.v34i05.6239.
  3. Language models are few-shot learners. arXiv: Computation and Language,arXiv: Computation and Language, May 2020.
  4. Zeroq: A novel zero shot quantization framework. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020. doi: 10.1109/cvpr42600.2020.01318. URL http://dx.doi.org/10.1109/cvpr42600.2020.01318.
  5. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv preprint arXiv:1807.06964, 2018.
  6. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  7. Mutual: A dataset for multi-turn dialogue reasoning. arXiv preprint arXiv:2004.04494, 2020.
  8. Llm.int8(): 8-bit matrix multiplication for transformers at scale.
  9. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  10. Learned step size quantization. arXiv preprint arXiv:1902.08153, 2019.
  11. Optimal brain compression: A framework for accurate post-training quantization and pruning. Aug 2022.
  12. Gptq: Accurate post-training quantization for generative pre-trained transformers. Oct 2022a.
  13. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022b.
  14. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
  15. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275, 2020a.
  16. Measuring massive multitask language understanding. Cornell University - arXiv,Cornell University - arXiv, Sep 2020b.
  17. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  18. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  19. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.10518, 2020.
  20. Accurate post training quantization with small calibration sets. International Conference on Machine Learning,International Conference on Machine Learning, Jul 2021.
  21. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951.
  22. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.  1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  23. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809–31826, 2022.
  24. The bigscience roots corpus: A 1.6tb composite multilingual dataset. Le Centre pour la Communication Scientifique Directe - HAL - Diderot,Le Centre pour la Communication Scientifique Directe - HAL - Diderot, Nov 2022.
  25. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426, 2021.
  26. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023.
  27. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041, 2023.
  28. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103, 2021.
  29. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  30. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197–7206. PMLR, 2020.
  31. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021.
  32. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016.
  33. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557, 2022.
  34. Language models are unsupervised multitask learners.
  35. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  36. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137, 2023.
  37. Llama: Open and efficient foundation language models.
  38. Emergent abilities of large language models. Jun 2022a.
  39. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402–17414, 2022b.
  40. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145, 2023.
  41. Smoothquant: Accurate and efficient post-training quantization for large language models. Nov 2022.
  42. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183, 2022.
  43. Opt: Open pre-trained transformer language models.
  44. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  45. Neural architecture search with reinforcement learning. International Conference on Learning Representations,International Conference on Learning Representations, Nov 2016.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now