Densing Law of LLMs (2412.04315v2)

Abstract: LLMs have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases. However, this scaling brings great challenges to training and inference efficiency, particularly for deploying LLMs in resource-constrained environments, and the scaling trend is becoming increasingly unsustainable. This paper introduces the concept of ``\textit{capacity density}'' as a new metric to evaluate the quality of the LLMs across different scales and describes the trend of LLMs in terms of both effectiveness and efficiency. To calculate the capacity density of a given target LLM, we first introduce a set of reference models and develop a scaling law to predict the downstream performance of these reference models based on their parameter sizes. We then define the \textit{effective parameter size} of the target LLM as the parameter size required by a reference model to achieve equivalent performance, and formalize the capacity density as the ratio of the effective parameter size to the actual parameter size of the target LLM. Capacity density provides a unified framework for assessing both model effectiveness and efficiency. Our further analysis of recent open-source base LLMs reveals an empirical law (the densing law)that the capacity density of LLMs grows exponentially over time. More specifically, using some widely used benchmarks for evaluation, the capacity density of LLMs doubles approximately every three months. The law provides new perspectives to guide future LLM development, emphasizing the importance of improving capacity density to achieve optimal results with minimal computational overhead.

• The paper introduces capacity density as a unified metric that measures LLM performance relative to their effective parameter count.
• The study demonstrates that LLM density doubles roughly every 3.3 months, enabling state-of-the-art performance with significantly smaller models and up to a 266.7-fold decrease in inference costs.
• The findings suggest a shift toward a 'Green Scaling Law,' aligning with hardware advancements to promote sustainable, high-efficiency large language models.

Analysis of the "Densing Law of LLMs"

The paper "Densing Law of LLMs" addresses key challenges in the development of LLMs: the inherent trade-off between model effectiveness and efficiency, given the ever-increasing demand for comprehensive AI applications under resource constraints. The authors propose the concept of "capacity density" as a unified metric that quantifies the performance quality of LLMs concerning their parameter efficiency, introducing the Densing Law as an empirical observation of how this metric evolves over time.

Introduction of Capacity Density

Capacity density is introduced as a ratio of a LLM's effective parameter size to its actual parameter size. The effective parameter size is defined as the minimum parameter count required by a reference model to achieve comparable performance to the target LLM. This approach allows for a holistic evaluation across scales, accounting for both raw parameter count and performance output in different constrained environments.

Empirical Observations: The Densing Law

The central claim of the paper is encapsulated by the Densing Law, which demonstrates that the maximum capacity density of open-source LLMs since 2023 follows an exponential growth trend. Through detailed analysis of LLMs’ performances across widely used benchmarks such as MMLU, BBH, and HumanEval, the authors derive an equation suggesting that the density of these models effectively doubles approximately every 3.3 months. This finding posits that it is possible to attain performance equivalent to current state-of-the-art models with models of significantly reduced parameter sizes within a relatively short time span.

Implications and Core Corollaries

The paper extrapolates several significant implications from the Densing Law:

1. Inferred Inference Cost Reduction: As capacity density increases exponentially, equivalent performance can be achieved with smaller models, suggesting a similar exponential reduction in inference costs. The authors empirically show a 266.7-fold decrease in inference costs for models comparable to GPT-3.5-level performance over time.
2. Synergy with Moore’s Law: By combining insights from the Densing Law and Moore’s Law, the authors illustrate that the effective parameter size of LLMs capable of being run on comparable hardware increasingly improves, hinting at the potential for high-efficiency, large-scale models on consumer-grade hardware.
3. Post-ChatGPT Acceleration: A notable acceleration in the growth rate of LLM density post-ChatGPT release is observed, attributed to increased investments and more accessible open-source models.
4. Challenges with Current Compression: The paper outlines limitations of extant pruning and distillation techniques in enhancing model density, advocating for enhanced methodologies that don't diminish a model’s effective density.
5. Density-Centric Development: This leads to a proposition of a "Green Scaling Law," whereby the focus of LLM development should pivot towards density optimization rather than mere parameter scaling, ensuring sustainable, performance-efficient models.

Future Directions and Challenges

Despite its impactful insights, the paper acknowledges lingering challenges, including the need for more comprehensive evaluation benchmarks to ensure accurate density measurements and mitigation of dataset contamination issues. Additionally, the authors point out the burgeoning relevance of extending capacity density evaluation methods to multimodal models and exploring the implications of reasoning-based tasks in the context of density.

Conclusion

The paper offers a significant contribution to the development and deployment of efficient LLMs, particularly underlining the shared trajectory of algorithmic optimization and hardware enhancement likely to dictate future trends. The introduction of the Densing Law provides a robust framework through which LLM quality and efficiency can be assessed, reflecting on both past and prospective advancements in AI capabilities. As the community strives for models that achieve state-of-the-art performance with minimal computational costs, the implications of this research may drive a shift towards more sustainable and equitably developed AI systems.