--- title: Neural Scaling Laws url: https://www.emergentmind.com/topics/neural-scaling-law type: topic --- # Neural Scaling Laws Neural scaling laws describe the empirical observation that the performance metrics—most commonly generalization error, loss, or accuracy—of deep neural networks improve predictably as a power-law function of key system resources, such as the number of model parameters, training dataset size, or the available compute budget. Across modalities and architectures, these power laws hold over multiple orders of magnitude, providing a foundational principle for guiding large-scale model design, resource allocation, and forecasting achievable performance. While scaling laws were initially discovered in large language and vision models, rigorous theoretical and empirical work demonstrates their broad applicability and reveals that the scaling exponents and limits are governed by intrinsic properties of the data distribution and learning task. ## 1. Formal Definition and Empirical Universality A neural scaling law expresses a performance metric (e.g., test loss $\mathcal{L}$) as a function of a scaling variable such as model size $N$ or dataset size $\mathcal{D}$: $$ \mathcal{L}(N) \sim N^{-\alpha}, \quad \mathcal{L}(\mathcal{D}) \sim \mathcal{D}^{-\beta} $$ where the exponents $\alpha, \beta > 0$ are empirically determined and encode how effectively the system benefits from additional resources. Multiple studies, including exhaustive empirical analyses of LLMs, vision transformers, diffusion models, and operator networks, consistently report clean power-law fits across regimes of practical interest. This universality suggests a deeper connection to the structure of natural data and underlying learning dynamics [2412.07942]. Scaling behaviors extend to composite dependencies on data, model, and compute, of the form $$ \mathcal{L}(N, \mathcal{D}) \sim A N^{-\alpha} + B \mathcal{D}^{-\beta} + C $$ with $C$ an irreducible noise floor. More sophisticated forms, such as smoothly broken power laws, capture observed inflections and multi-phase regimes in large-scale practical settings [2210.14891]. ## 2. Theoretical Foundations: Data-Distribution Rooted Models Recent theoretical developments frame neural scaling laws as emerging from the interaction between model expressivity and the statistical structure of the learning task. Brill (2025) introduces a percolation-theoretic model of natural data, where effective learning decomposes into "quanta" associated with discrete subtasks or semantic clusters [2412.07942]. The model posits: - After factoring invariances, data reside on an effective high-dimensional lattice. - Local learning subtasks arise as clusters of lattice sites sharing the same target function, with site connectivity governed by a bond percolation process. - At the percolation threshold $p_c$, the cluster size distribution is a power law, $n_s \propto s^{-\tau}$. Criticality in this generative model gives rise to two universal scaling regimes for error as a function of model degrees of freedom $N$ or dataset size $\mathcal{D}$: - **Quanta/subtask regime (critical percolation, $p \approx p_c$):** - Model-limited: $\mathcal{L} \propto N^{-\alpha}$, with $\alpha = (3-\tau)/(\tau-2)$. For Bethe lattice ($\tau = 5/2$), $\alpha=1$. - Data-limited: $\mathcal{L} \propto \mathcal{D}^{-\alpha/(1+\alpha)}$. - **Manifold regime (supercritical, $p > p_c$):** - Error scaling governed by manifold approximation: $\mathcal{L} \propto N^{-c/D}$ or $\mathcal{L} \propto \mathcal{D}^{-c/D}$, where $D$ is intrinsic cluster dimension and $c$ is an architecture-specific constant. This framework unifies and grounds previous phenomenological scaling models, relating "discrete quanta" scaling [Michaud et al.] and data-manifold approximation [Bahri & Sharma] to underlying percolation criticality [2412.07942]. ## 3. Scaling Exponents, Architectural Factors, and Data Geometry The exponents $\alpha$ and $\beta$ reflect the geometric and statistical structure of the data as well as properties of the architecture: - In the limit where the data manifold has intrinsic dimension $d$ and ReLU networks provide piecewise-linear fits, the scaling exponent is $\alpha \approx 4/d$ for both cross-entropy and MSE losses [2004.10802]. - For composite or multi-subtask tasks, modular resource allocation and subtask loss additivity yield $\mathcal{L} \sim 1/N$ neuron scaling, and, via typical deep network parameterizations, $\mathcal{L} \sim N_p^{-1/3}$ for parameter count $N_p$ [2402.05164, 2310.02258]. - The "Criticality" class of scaling laws subsumes a spectrum of behavior from strong manifold effects ($\alpha < 1$) to subtask-dominated regimes ($\alpha \approx 1$), seamlessly interpolated by the percolation-theoretic framework [2412.07942]. Task and architecture variation influence exponents: vision and deep regression tasks can show $\alpha$ in the $1-2$ range for data scaling [2509.10000], while LLMs exhibit exponents $\alpha\sim 0.07-0.3$ and data exponents $\beta\sim 0.1-0.25$ depending on pretraining regime [2502.12051]. ## 4. Emergent Regimes, Limitations, and Extension to Practice Scaling laws can break, saturate, or transition across regimes: - Empirical curves display "smoothly broken" behavior with inflections at critical data or compute thresholds, motivating the Broken Neural Scaling Law (BNSL) formalism to model multi-phase, nonmonotonic trends [2210.14891]. - Fine-tuning, multimodal, reinforcement learning, and data-limited settings can induce regime changes (e.g., plateaux giving way to power-law decay at larger scale) [2502.12051]. - Rigorous theory predicts when to expect sharp transitions, irreducible error floors, or diminishing returns due to finite data or model bottlenecks [2210.14891, 2412.07942]. A summary of typical scaling exponents by domain: | Domain | Model Exponent $\alpha$ | Data Exponent $\beta$ | |-----------|------------------------|-----------------------| | Language | 0.05–0.30 | 0.05–0.25 | | Vision | 0.20–0.30 | ~0.10 | | Regression| 1.0–2.3 | 0.8–2.3 | Empirically, progress along a scaling law curve reliably predicts achievable performance so long as the system remains in a regime where the data model, task, and architecture are consistent with prior power-law fits [2502.12051]. ## 5. Metrics Beyond Cross-Entropy and Rank-Based Scaling Most scaling studies focus on cross-entropy loss, but this omits aspects vital for deployment—such as the rank ordering of correct predictions. The Relative-Based Scaling Law, defined via the Relative-Based Probability (RBP) metric $$ RBP_k = \Pr(\text{true token in top-$k$ predicted}), $$ obeys its own power-law scaling: $$ -\log RBP_k \propto S^{-\alpha_k}, $$ with $S$ the non-embedding parameter count and $\alpha_k$ increasing with $k$ [2510.20387]. RBP scaling closely tracks cross-entropy based scaling, but governs phenomena related to emergence in sequence prediction (e.g., sharp thresholding behavior for long-range accuracy). This provides a quantitative framework for predicting the onset and shape of "emergent" capabilities as models grow. ## 6. Practical Implications, Fitting, and Automated Law Discovery Neural scaling laws now guide the design and resource allocation for large-scale AI systems. Key implications include: - Compute allocation: theory predicts compute-optimal tradeoffs between data size and model size, e.g., scaling both linearly for fixed compute budgets ($N \propto \mathcal{D}$), with the law's exponent dictating cost/benefit [2407.01456]. - Design: empirical or theory-driven exponent estimates allow practitioners to forecast the future gains from increasing data or model capacity and to avoid regimes with sharp diminishing returns [2502.12051]. - Automated discovery: frameworks such as EvoSLD co-evolve symbolic law expressions with domain-specific optimizers, yielding parsimonious, generalizable, and highly accurate scaling laws across experimental settings [2507.21184]. - Caution: Scaling laws are not universal—critical regime changes, architectural innovations (sparse, retrieval-augmented, multimodal models), or domain shifts may break existing power-law trends [2210.14891, 2502.12051]. Rigorous uncertainty estimation and cross-validation are required. ## 7. Open Questions and Directions Fundamental directions for neural scaling law research include: - Extending theoretical frameworks beyond percolation and manifold models to account for data heterogeneity, compositionality, and information-theoretic constraints [2412.07942, 2407.01456]. - Formalizing the limits of predictability due to sharp phase transitions and lawful extrapolation beyond existing data [2210.14891]. - Integrating fairness, robustness, and inference-time scaling into the core scaling law formalism [2502.12051]. - Linking finite-width corrections, training dynamics, and optimization artifacts with asymptotic scaling exponents [2402.01092]. - Measuring and modeling the percolation structure or intrinsic dimension in practical LLM corpora and scientific datasets to forecast scaling performance in new domains [2412.07942, 2004.10802]. ### References - [2412.07942] Neural Scaling Laws Rooted in the Data Distribution - [2510.20387] Relative-Based Scaling Law for Neural Language Models - [2509.10000] Neural Scaling Laws for Deep Regression - [2507.21184] EvoSLD: Automated Neural Scaling Law Discovery With Large Language Models - [2502.12051] How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines - [2410.00357] Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study - [2407.01456] Information-Theoretic Foundations for Neural Scaling Laws - [2402.05164] A Resource Model For Neural Scaling Law - [2402.01092] A Dynamical Model of Neural Scaling Laws - [2210.16859] A Solvable Model of Neural Scaling Laws - [2210.14891] Broken Neural Scaling Laws - [2004.10802] A Neural Scaling Law from the Dimension of the Data Manifold - [2310.02258] A Neural Scaling Law from Lottery Ticket Ensembling Source: https://www.emergentmind.com/topics/neural-scaling-law