Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations (2407.05690v1)

Published 8 Jul 2024 in cs.CL and cs.AI

Abstract: Structured pruning fundamentally reduces computational and memory overheads of LLMs and offers a feasible solution for end-side LLM deployment. Structurally pruned models remain dense and high-precision, highly compatible with further tuning and compression. However, as the coarse-grained structured pruning poses large damage to the highly interconnected model, achieving a high compression ratio for scaled-up LLMs remains a challenge. In this paper, we introduce a task-agnostic structured pruning approach coupled with a compact Transformer architecture design. The proposed approach, named TransAct, reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules, while preserving the inter-module activations that are sensitive to perturbations. Hence, the LLM is pruned into an intra-module low-rank architecture, significantly reducing weights, KV Cache and attention computation. TransAct is implemented on the LLaMA model and evaluated on downstream benchmarks. Results verify the optimality of our approach at high compression with respect to both efficiency and performance. Further, ablation studies reveal the strength of activation-guided iterative pruning and provide experimental analysis on the redundancy of MHA and MLP modules.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press.
  2. Medusa: Simple llm inference acceleration framework with multiple decoding heads. ArXiv preprint, abs/2401.10774.
  3. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  4. Think you have solved question answering? try arc, the ai2 reasoning challenge.
  5. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference.
  6. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. ArXiv preprint, abs/2208.07339.
  7. An algorithm–hardware co-optimized framework for accelerating n:m sparse transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 30(11).
  8. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR.
  9. Gptq: Accurate post-training quantization for generative pre-trained transformers. ArXiv preprint, abs/2210.17323.
  10. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv preprint, abs/2312.00752.
  11. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  12. Compressing llms: The truth is rarely pure and never simple. ArXiv preprint, abs/2310.01382.
  13. Mixtral of experts. ArXiv preprint, abs/2401.04088.
  14. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  15. Efficient memory management for large language model serving with pagedattention.
  16. Relax: Composable abstractions for end-to-end dynamic machine learning. ArXiv preprint, abs/2311.02103.
  17. Bloom: A 176b-parameter open-access multilingual language model.
  18. Optimal brain damage. In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann.
  19. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  20. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR.
  21. Awq: Activation-aware weight quantization for llm compression and acceleration. ArXiv preprint, abs/2306.00978.
  22. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  23. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3622–3628. ijcai.org.
  24. Llm-qat: Data-free quantization aware training for large language models. ArXiv preprint, abs/2305.17888.
  25. Pipellm: Pipeline llm inference on heterogeneous devices with sequence slicing. In Proceedings of the ACM SIGCOMM 2023 Conference, pages 1126–1128.
  26. Llm-pruner: On the structural pruning of large language models. ArXiv preprint, abs/2305.11627.
  27. Pointer sentinel mixture models.
  28. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391.
  29. Relu strikes back: Exploiting activation sparsity in large language models.
  30. A white paper on neural network quantization. ArXiv preprint, abs/2106.08295.
  31. The lambada dataset: Word prediction requiring a broad discourse context. In The 54th Annual Meeting of the Association for Computational Linguistics Proceedings of the Conference: Vol. 1 Long Papers, volume 3, pages 1525–1534. ACL.
  32. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048.
  33. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732–8740. AAAI Press.
  34. Confident adaptive language modeling.
  35. Powerinfer: Fast large language model serving with a consumer-grade gpu.
  36. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  37. A simple and effective pruning approach for large language models. ArXiv preprint, abs/2306.11695.
  38. MLC team. 2023. MLC-LLM.
  39. Together Computer. 2023. Redpajama: an open dataset for training large language models.
  40. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288.
  41. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 94–106. Association for Computational Linguistics.
  42. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations.
  43. Smoothquant: Accurate and efficient post-training quantization for large language models.
  44. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
  45. Opt: Open pre-trained transformer language models.
  46. Unveiling a core linguistic region in large language models. ArXiv preprint, abs/2310.14928.
  47. A survey on model compression for large language models. ArXiv preprint, abs/2308.07633.
  48. Perp: Rethinking the prune-retrain paradigm in the era of llms. ArXiv preprint, abs/2312.15230.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Bowen Shen (23 papers)
  2. Zheng Lin (104 papers)
  3. Daren Zha (5 papers)
  4. Wei Liu (1135 papers)
  5. Jian Luan (50 papers)
  6. Bin Wang (750 papers)
  7. Weiping Wang (123 papers)
Citations (1)
View on Semantic Scholar