An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models (2410.01243v2)

Abstract: Recent empirical studies show three phenomena with increasing size of LLMs: compute-optimal size scaling, emergent capabilities, and performance plateauing. We present a simple unified mathematical framework to explain all of these LLM scaling phenomena, building on recent skill-text bipartite graph frameworks for semantic learning. Modeling the learning of concepts from texts as an iterative process yields an analogy to iterative decoding of low-density parity check (LDPC) codes in information theory. Thence, drawing on finite-size scaling characterizations of LDPC decoding, we derive the compute-optimal size scaling (Chinchilla rule) for LLMs. Further, using tools from random network theory, we provide a simple explanation for both emergence of complex skills and plateauing of performance as the size of LLMs scale. We see multiple plateaus.

• The paper introduces a unified framework combining information theory and graph models to derive the compute-optimal size scaling rule, aligning with the Chinchilla benchmark.
• The paper explains emergent capabilities through the formation of a giant connected component in the skill graph, predicting performance leaps beyond a compute threshold.
• The paper interprets performance plateaus as a consequence of diverse skill requirements, offering actionable insights for optimal training resource allocation.

An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in LLMs

Introduction

The pursuit of efficient large-scale LLM training has led to several intriguing empirical observations, such as compute-optimal size scaling, emergent capabilities, and performance plateauing. This paper presents a unified mathematical framework that coherently explains these phenomena using insights from information theory, coding theory, and random graph theory. The approach builds on the skill-text bipartite graph frameworks, connecting them with iterative decoding processes from communication theory, and providing a solid foundation for understanding the scaling properties of modern LLMs.

Key Contributions

This paper makes several important contributions:

1. A unified framework for understanding the learning process of LLMs, focusing on the acquisition of concepts and skills from text data.
2. Derivation of the compute-optimal size scaling rule (the Chinchilla rule) using non-asymptotic information-theoretic tools.
3. An explanation of emergent capabilities in LLMs through random network theory.
4. Analysis of performance plateauing as a natural outcome of the diversity of skills required for complex tasks, predicting the existence of multiple emergent phenomena with further scaling.

Framework Overview

The proposed framework models the learning process using a bipartite graph where the nodes represent concepts and texts, and the edges indicate the presence of relevant information in the texts necessary to learn the concepts. This graph, denoted as $G_1^{(C)}$, undergoes an iterative learning (or decoding) process that simulates the peeling decoding of low-density parity check (LDPC) codes, yielding insights into the optimal allocation of computational resources.

Concepts and Skills

The framework differentiates between basic concepts learned directly from texts and higher-level skills composed of these concepts. The skills are further organized hierarchically, with more advanced skills requiring the mastery of prerequisite lower-level skills. This hierarchical skill acquisition is represented by additional edges in the graph, forming another bipartite graph $G_2$.

Compute-Optimal Size Scaling

By treating the learning process as an iterative decoding problem, the authors derive the compute-optimal scaling rule. They show that the number of concepts learned is maximized when the model size ($R$ for concepts) and the dataset size ($T$ for texts) scale equally with the compute budget ($C$). The optimal scaling is derived from the iterative decoding threshold, where the expected number of concepts learned reaches its peak.

The mathematical derivations leverage finite-size scaling laws from coding theory to argue that smaller or larger deviations from the equal scaling of model and dataset size lead to suboptimal learning outcomes. The results are consistent with the empirically observed Chinchilla rule, showing that this scaling law is indeed optimal.

Excess Entropy

The paper also provides insights into the scaling of excess entropy (a proxy for model inefficiency) as a function of model size. Using Pinsker's inequality, the lower bound on excess entropy is shown to match empirical observations well, though there is room for tighter theoretical bounds or improved architectures.

Emergence Phenomena

Emergent capabilities, where models exhibit qualitatively new behaviors with scaling, are explained through the appearance of a giant connected component (GCC) in the random graphs representing skill composition. For a given level of skills, the probability that a model can perform a complex task (requiring the composition of multiple skills) increases sharply beyond a threshold compute budget, due to the formation of a GCC in the skill graph. This behavior maps closely to empirical observations of sudden jumps in model performance on specific tasks.

Plateaus in Performance

The paper addresses the phenomenon of plateauing, where further increases in model size or compute budget yield diminishing returns in performance. Theoretical analysis using random graph theory shows that this plateauing is due to the diverse levels of skills required for different tasks. In particular, tasks that need a wide variety of skill levels (modeled by a multimodal distribution of skill requirements) exhibit multiple emergence thresholds and performance plateaus.

Implications

Understanding these phenomena helps in designing optimal training regimes for future LLMs. The work suggests that:

• Equal scaling of model and dataset size with compute budget remains optimal.
• Emergence thresholds predict when significant new capabilities will develop.
• Observing plateaus may indicate upcoming emergent phenomena.

These insights are not only theoretically profound but also have practical implications for developing better architectures and datasets. Moreover, understanding these scaling laws can inform policymakers about the limits and potentials of large-scale models in AI, impacting regulatory policies and resource allocation decisions.

Future Directions

Future work can extend this framework by incorporating training epoch effects, exploring hierarchical concept structures, and optimizing degree distributions for better learning outcomes. The approach could also be adapted to other machine learning paradigms beyond transformer-based models, offering a broader applicability of the insights gained here.

This formal and rigorous analysis presents a comprehensive understanding of the key phenomena in the scaling of LLMs, grounded in robust mathematical principles from information theory and graph theory. It serves as an essential reference for researchers aiming to optimize the design and training of large-scale AI systems.