Algorithmic progress in language models (2403.05812v1)

Abstract: We investigate the rate at which algorithms for pre-training LLMs have improved since the advent of deep learning. Using a dataset of over 200 LLM evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in LLMing, shedding light on the relative contributions from compute and algorithms.

• The paper demonstrates that algorithmic innovations exponentially enhance the effective use of computational resources in LLM pre-training.
• Empirical evaluations across 200+ benchmarks show that compute requirements for a given performance level halve every 8 months.
• The study highlights the transformative impact of the transformer architecture, yielding significant compute-equivalent gains over previous models.

Algorithmic Progress and Compute Scaling in Pre-Training of LLMs: An Empirical Analysis

Introduction

The fast-paced advances in LLMing, propelled by increasingly capable LLMs, have captured widespread attention. These LLMs serve a pivotal role across a broad spectrum of applications, ranging from natural language processing tasks to generating complex textual content. Central to these advancements are not only the burgeoning computational resources but also significant algorithmic developments that optimize the use of such resources. In this context, understanding the interplay between algorithmic innovation and hardware scalability becomes crucial for assessing future directions in the field of LLMing.

Methodology

Our paper employs an empirical approach to dissect the contributions of algorithmic enhancements and compute scaling in the evolution of LLM performance. We ground our analysis in a dataset encompassing over 200 LLM evaluations on benchmarks like WikiText and Penn Treebank over a period from 2012 to 2023. This comprehensive dataset allows for a nuanced exploration of the trajectory of LLMing improvements.

Model Framework

Central to our analysis is the employment of an augmented scaling law, derived from foundational works in the domain, which links model performance unequivocally to the model's scale and the computational resources. Through careful adjustments, we incorporate notions of effective data' andeffective model size'—terms that quantify the algorithmic efficiency gains over time. Our approach posits that consistent algorithmic progress manifests as an exponential enhancement in the `effectiveness' of these resources, thus facilitating performance gains at reduced compute costs.

Empirical Findings

Our findings are revealing. We estimate that the compute requirements for a given performance threshold on LLMing tasks have approximately halved every 8 months since 2012. This rate outpaces hardware improvements following Moore's Law, highlighting the brisk pace of algorithmic innovation in LLMing. Nonetheless, our analysis also discerns that the lion's share of recent performance upswings is attributable to increases in computational resources, with algorithmic improvements playing a lesser, though still significant, role.

The Transformer Architecture: A Case Study

The advent of the transformer architecture marks a watershed moment in LLM development. Our analysis assigns a compute-equivalent gain to the transformer, quantifying its contribution relative to preceding architectures. The transformer is shown to substantially reduce the compute required for a given level of performance, underscoring its pivotal role in the accelerated progress of LLMing capabilities.

Implications and Future Directions

Understanding the relative contributions of algorithmic progress and compute scaling offers valuable insights into the potential trajectories for the development of LLMs. While the rapid scale-up of computational resources has undeniably fueled recent advances, the sustained pace of algorithmic innovation underscores the field's robust foundation in research and development. Looking ahead, continued exploration of novel architectures, optimization techniques, and efficient training methods will be critical in navigating the computational and environmental constraints facing the next generation of LLMs.

Conclusion

Our paper provides a structured empirical analysis of the advancements in pre-training LLMs, emphasizing the symbiotic relationship between algorithmic progress and compute scaling. By illuminating the dynamics shaping the evolution of LLMs, we contribute to a deeper understanding of the past and present trends, laying the groundwork for informed speculation on the future of LLMing.