Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Scavenging Hyena: Distilling Transformers into Long Convolution Models (2401.17574v1)

Published 31 Jan 2024 in cs.CL and cs.LG

Abstract: The rapid evolution of LLMs, epitomized by architectures like GPT-4, has reshaped the landscape of natural language processing. This paper introduces a pioneering approach to address the efficiency concerns associated with LLM pre-training, proposing the use of knowledge distillation for cross-architecture transfer. Leveraging insights from the efficient Hyena mechanism, our method replaces attention heads in transformer models by Hyena, offering a cost-effective alternative to traditional pre-training while confronting the challenge of processing long contextual information, inherent in quadratic attention mechanisms. Unlike conventional compression-focused methods, our technique not only enhances inference speed but also surpasses pre-training in terms of both accuracy and efficiency. In the era of evolving LLMs, our work contributes to the pursuit of sustainable AI solutions, striking a balance between computational power and environmental impact.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (19)
  1. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  2. Gpt-neox-20b: An open-source autoregressive language model, 2022.
  3. The fast Fourier transform. IEEE Spectrum, 4(12):63–70, December 1967. ISSN 0018-9235. doi: 10.1109/MSPEC.1967.5217220.
  4. Longnet: Scaling transformers to 1,000,000,000 tokens, 2023.
  5. Hungry Hungry Hippos: Towards Language Modeling with State Space Models. 2022. doi: 10.48550/ARXIV.2212.14052.
  6. The pile: An 800gb dataset of diverse text for language modeling, 2020.
  7. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 2021.
  8. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  9. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL https://api.semanticscholar.org/CorpusID:7200347.
  10. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. 2020. doi: 10.48550/ARXIV.2006.16236.
  11. Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions. 2023. doi: 10.48550/ARXIV.2310.18780.
  12. Pointer sentinel mixture models, 2016.
  13. OpenAI, Nov 2023. URL https://openai.com/blog/new-models-and-developer-products-announced-at-devday.
  14. Hyena hierarchy: Towards larger convolutional language models, 2023.
  15. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
  16. Mobilebert: a compact task-agnostic bert for resource-limited devices, 2020.
  17. Attention is all you need. 2017. URL https://arxiv.org/pdf/1706.03762.pdf.
  18. Harnessing the power of llms in practice: A survey on chatgpt and beyond, 2023.
  19. An Attention Free Transformer. 2021. doi: 10.48550/ARXIV.2105.14103.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
Citations (1)
View on Semantic Scholar

Summary

Overview of the Distillation Method

Researchers have proposed an innovative knowledge distillation method for training LLMs that aims to bypass the extensive computational costs typically associated with pre-training new architectures. Distilling knowledge from a pre-trained LLM using quadratic multi-headed attention into a model utilizing sub-quadratic Hyena operators, exhibits the potential for higher efficiency and improved performance.

Efficiency Challenges in LLMs

While LLMs have significantly advanced the field of natural language processing, concerns about their environmental and financial impact continue to grow. The Hyena mechanism offers a promising solution, focusing on models that can efficiently process longer contexts - a capability that the traditional transformer models find challenging due to their quadratic attention mechanisms.

Key Distillation Techniques

The researchers' innovative approach involves several key strategies:

  • Cross-Architecture Knowledge Transfer: Knowledge is distilled from transformers into long convolutional models, showcasing the method's novelty.
  • Superiority Over Traditional Pre-Training: Remarkably, this new distillation strategy outperforms the efficiency of conventional pre-training in terms of both speed (inference) and performance (accuracy).

Empirical Results and Model Evaluation

Empirical evaluations reveal the advantage of progressive knowledge transfer against typical pre-training, with the distillation approach appearing promising in terms of performance and computational savings. For the long-context processing, the paper demonstrates the utility of the sub-quadratic Hyena operators, which show potential as a useful replacement for the archetypal transformer attention mechanism.

When quantifying the performance, the distillation process resulted in a model that surpassed the base pre-training model, as indicated by perplexity scores on textual datasets such as OpenWebText and WikiText. Furthermore, after fine-tuning the student model post-distillation, a significant improvement in LLMing capability was observed—showing the added benefits of this step in the distillation process.

Language task assessments further supported these findings, elucidating that a Pythia 70M student model with Hyena operators could retain a substantial portion of its linguistic capabilities when compared to its traditional transformer-based teacher model.

Conclusion and Future Work

The insights drawn from this research point to an encouraging direction. Knowledge distillation, especially when used in conjunction with sub-quadratic attention alternatives like Hyena, can not only enhance the training speed but also retain or even surpass the LLMing skills of the prevalent transformer models. The paper calls for future exploration on scalability and further fine-tuning of the distillation process, potentially paving the way for more efficient and robust LLMs that can maintain high performance while being more environmentally sustainable.