Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency (2304.02721v3)

Published 5 Apr 2023 in cs.CL and cs.AI

Abstract: Sequence-to-sequence LLMs can be used to produce abstractive summaries which are coherent, relevant, and concise. Still, model sizes can make deployment in latency-sensitive or web-scale implementations difficult. This paper studies the relationship between model size, structured pruning, inference efficiency, and summarization accuracy on widely used summarization datasets. We show that model accuracy is tied to the encoder size while inference efficiency is connected to the decoder. Using asymmetric pruning can lead to nearly 3x improvement in inference latency with ~1 point loss in Rouge-2. Moreover, we find both the average degradation and the role of asymmetry to be consistent across model sizes and variations in datasets.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Language models are few-shot learners. ArXiv, abs/2005.14165.
  2. Sparse*bert: Sparse models are robust. ArXiv, abs/2205.12452.
  3. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  4. A deep neural network compression pipeline: Pruning, quantization, huffman encoding. ArXiv.
  5. Scaling laws for autoregressive generative modeling. ArXiv, abs/2010.14701.
  6. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.
  7. Training compute-optimal large language models. ArXiv, abs/2203.15556.
  8. Tinybert: Distilling bert for natural language understanding. ArXiv, abs/1909.10351.
  9. Scaling laws for neural language models. ArXiv, abs/2001.08361.
  10. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In International Conference on Learning Representations.
  11. Learned token pruning for transformers. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
  12. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. ArXiv, abs/2203.07259.
  13. Block pruning for faster transformers. ArXiv, abs/2109.04838.
  14. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36:1234–1240.
  15. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098–1115.
  16. Dq-bart: Efficient sequence-to-sequence model via joint distillation and quantization. In Annual Meeting of the Association for Computational Linguistics.
  17. Train large, then compress: Rethinking model size for efficient training and inference of transformers. ArXiv, abs/2002.11794.
  18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  19. Tsvetomila Mihaylova and André F. T. Martins. 2019. Scheduled sampling for transformers. ArXiv, abs/1906.07651.
  20. Train flat, then compress: Sharpness-aware minimization learns more compressible models. ArXiv, abs/2205.12694.
  21. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745.
  22. Oren Neumann and Claudius Gros. 2022. Scaling laws for a multi-agent reinforcement learning model. ArXiv, abs/2210.00849.
  23. Document ranking with a pretrained sequence-to-sequence model. In Findings.
  24. Alec Radford. 2018. Improving language understanding by generative pre-training.
  25. Robust speech recognition via large-scale weak supervision. ArXiv, abs/2212.04356.
  26. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683.
  27. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732.
  28. On the predictability of pruning across scales. ArXiv, abs/2006.10621.
  29. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
  30. Movement pruning: Adaptive sparsity by fine-tuning. ArXiv, abs/2005.07683.
  31. Confident adaptive language modeling. ArXiv, abs/2207.07061.
  32. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  33. Sam Shleifer and Alexander M. Rush. 2020. Pre-trained summarization distillation. ArXiv, abs/2010.13002.
  34. Sequence to sequence learning with neural networks. In NIPS.
  35. Scale efficiently: Insights from pre-training and fine-tuning transformers. ArXiv, abs/2109.10686.
  36. Attention is all you need. ArXiv, abs/1706.03762.
  37. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL.
  38. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652.
  39. Deebert: Dynamic early exiting for accelerating bert inference. In ACL.
  40. A streaming approach for efficient batched beam search. In Conference on Empirical Methods in Natural Language Processing.
  41. Q8bert: Quantized 8bit bert. 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS), pages 36–39.
  42. Prune once for all: Sparse pre-trained language models. ArXiv, abs/2111.05754.
  43. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. ArXiv, abs/1912.08777.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Daniel Campos (62 papers)
  2. ChengXiang Zhai (64 papers)
Citations (2)
View on Semantic Scholar