Scavenging Hyena: Distilling Transformers into Long Convolution Models (2401.17574v1)

Abstract: The rapid evolution of LLMs, epitomized by architectures like GPT-4, has reshaped the landscape of natural language processing. This paper introduces a pioneering approach to address the efficiency concerns associated with LLM pre-training, proposing the use of knowledge distillation for cross-architecture transfer. Leveraging insights from the efficient Hyena mechanism, our method replaces attention heads in transformer models by Hyena, offering a cost-effective alternative to traditional pre-training while confronting the challenge of processing long contextual information, inherent in quadratic attention mechanisms. Unlike conventional compression-focused methods, our technique not only enhances inference speed but also surpasses pre-training in terms of both accuracy and efficiency. In the era of evolving LLMs, our work contributes to the pursuit of sustainable AI solutions, striking a balance between computational power and environmental impact.

• The paper introduces a distillation method that transfers transformer knowledge into long convolution models using Hyena operators.
• It demonstrates enhanced efficiency by overcoming quadratic attention limitations, leading to improved inference speed and accuracy.
• Empirical evaluations reveal that the distilled Hyena-based student model retains essential language modeling capabilities compared to its transformer teacher.

Overview of the Distillation Method

Researchers have proposed an innovative knowledge distillation method for training LLMs that aims to bypass the extensive computational costs typically associated with pre-training new architectures. Distilling knowledge from a pre-trained LLM using quadratic multi-headed attention into a model utilizing sub-quadratic Hyena operators, exhibits the potential for higher efficiency and improved performance.

Efficiency Challenges in LLMs

While LLMs have significantly advanced the field of natural language processing, concerns about their environmental and financial impact continue to grow. The Hyena mechanism offers a promising solution, focusing on models that can efficiently process longer contexts - a capability that the traditional transformer models find challenging due to their quadratic attention mechanisms.

Key Distillation Techniques

The researchers' innovative approach involves several key strategies:

• Cross-Architecture Knowledge Transfer: Knowledge is distilled from transformers into long convolutional models, showcasing the method's novelty.
• Superiority Over Traditional Pre-Training: Remarkably, this new distillation strategy outperforms the efficiency of conventional pre-training in terms of both speed (inference) and performance (accuracy).

Empirical Results and Model Evaluation

Empirical evaluations reveal the advantage of progressive knowledge transfer against typical pre-training, with the distillation approach appearing promising in terms of performance and computational savings. For the long-context processing, the paper demonstrates the utility of the sub-quadratic Hyena operators, which show potential as a useful replacement for the archetypal transformer attention mechanism.

When quantifying the performance, the distillation process resulted in a model that surpassed the base pre-training model, as indicated by perplexity scores on textual datasets such as OpenWebText and WikiText. Furthermore, after fine-tuning the student model post-distillation, a significant improvement in LLMing capability was observed—showing the added benefits of this step in the distillation process.

Language task assessments further supported these findings, elucidating that a Pythia 70M student model with Hyena operators could retain a substantial portion of its linguistic capabilities when compared to its traditional transformer-based teacher model.

Conclusion and Future Work

The insights drawn from this research point to an encouraging direction. Knowledge distillation, especially when used in conjunction with sub-quadratic attention alternatives like Hyena, can not only enhance the training speed but also retain or even surpass the LLMing skills of the prevalent transformer models. The paper calls for future exploration on scalability and further fine-tuning of the distillation process, potentially paving the way for more efficient and robust LLMs that can maintain high performance while being more environmentally sustainable.