--- title: Scaling Laws in Machine Learning url: https://www.emergentmind.com/topics/model-scaling-laws type: topic --- # Scaling Laws in Machine Learning Model scaling laws in machine learning formalize the empirical observation that task performance (typically test loss or error) improves as a smooth power law relative to the resources committed to training—namely, model size, dataset size, and total computation. These laws allow practitioners to predict the performance of larger models or datasets by extrapolating from smaller variants, and they provide principled guidelines for optimizing model development, resource allocation, and architectural choices across diverse domains of supervised, unsupervised, and reinforcement learning. ## 1. Mathematical Formulation and Universality Scaling laws describe the dependence of model performance metrics (e.g., cross-entropy loss, word error rate) on resources such as number of parameters $N$, dataset size $D$, and compute budget $C$. A canonical form underlying many domains is: $$ L(N, D) = \left[ (L_\infty)^{1/\alpha} + (N_C/N)^{\alpha_N/\alpha} + (D_C/D)^{\alpha_D/\alpha} \right]^\alpha $$ where $L_\infty$ is the irreducible loss, $N_C$ and $D_C$ are scale parameters, and $\alpha_N$, $\alpha_D$, $\alpha$ are exponents controlling the rate of improvement with model size and data. When only a single resource dominates, this reduces to simpler power laws: $$ L(N) = L_\infty + (N_C/N)^{\alpha_N} \qquad L(D) = L_\infty + (D_C/D)^{\alpha_D}. $$ Similar forms appear in other settings, such as kernel regression, where the excess risk scales as $$ \mathrm{Excess\ risk} = \Theta\left( M^{-(a-1)} + N^{-(a-1)/a} \right) $$ with $M$ the effective model dimension and $a$ the spectral decay exponent of the data covariance matrix [2406.08466]. This universality across tasks, modalities, and even theoretical models (e.g., random feature models, kernel methods) is rooted in underlying properties such as the eigenvalue spectrum of the covariance of real-world data, which empirically observe a power-law decay [2210.16859, 2509.20721]. ## 2. Empirical Validation Across Domains Scaling laws have been empirically validated in a wide variety of machine learning paradigms and domains: - **Language Modeling:** Transformer-based models, as demonstrated by Kaplan et al. and extended to models with up to 33B parameters, follow scaling laws for test loss in $N$ and $D$, with clear regimes of diminishing returns and predictable performance floors [2403.06563]. - **Acoustic/Speech Models:** Auto-predictive coding loss in acoustic models exhibits power-law scaling over two orders of magnitude in both $N$ and $D$, with precise prediction of performance limits [2106.09488, 2306.15815]. - **Recommendation Systems:** DLRM-style click-through rate prediction models obey a power-law plus constant form in all three axes (data, parameter count, compute), with data scaling retaining efficiency even when parameter scaling saturates [2208.08489]. - **Reinforcement Learning:** In multi-agent RL (e.g., AlphaZero), the Elo rating (playing strength) scales as a power law in both network size and training compute, with the existence of an optimal size–compute allocation and observed sample efficiency gains in larger models [2210.00849]. - **Code Understanding, Time Series, Multilingual LMs:** Code search/clone detection [2402.12813], forecasting with TSFMs [2410.12360], and multilingual pretraining [2410.12883] show robust power-law scaling, occasionally complicated by cross-task interactions or the need for family-level aggregation. - **Material Property Prediction:** Neural models for predicting molecular/material properties follow scaling laws in data, parameter count, and compute, with exponents depending on architecture and resource [2509.21811]. Power-law behaviors have also been demonstrated in kernel regression [2503.01314, 2509.20721], with scaling exponents determined by data spectrum redundancy. ## 3. Interpretation of Scaling Exponents and Limitations The scaling exponents $\alpha_N$, $\alpha_D$, $\beta$, etc., quantify how rapidly task error decreases with increased resources. Larger exponents indicate more efficient scaling: a modest increase in resources yields substantial performance gain, whereas small exponents denote diminishing returns. For example, in speech models, a 5% improvement in loss from increasing $N$ requires a $\sim$25× increase in parameters, while the corresponding data scaling exponent indicates a $\sim$14× data increase for the same reduction [2106.09488]. A universal irreducible loss term $L_\infty$ or asymptotic limit $\gamma$ often emerges, representing the fundamental lower bound achievable by any model given infinite data or parameters. Once the predicted achievable loss value nears this floor, further scaling is futile without architectural innovation or better data. Limitations are domain- and architecture-specific. For some tasks (e.g., discriminative rescoring [2306.15815], recommendation systems [2208.08489]), parameter scaling yields minimal gains beyond a certain point, suggesting a saturated regime. In other settings, scaling laws break down when model size or dataset size exceeds the intrinsic latent dimension of the data (as seen in random feature models, where risk plateaus when $N$ or $D > M$) [2210.16859]. Sensitivity to experimental design is a recurrent theme: The prescriptions derived from scaling laws (e.g., optimal token–parameter ratio, width–depth tradeoffs) can vary sharply with training schedules, hyperparameter settings, or architectural details. The Gemstones dataset [2502.06857] demonstrates that narrow experimental design leads to fragile prescriptions, and practitioners should account for such sensitivity. ## 4. Theoretical Mechanisms: Covariance Spectrum and Redundancy Theoretical work connects scaling exponents to the redundancy in data representations. When the input data covariance matrix has a polynomial tail, $\lambda_i \sim i^{-1/\beta}$, then the effective dimension and hence the error decay is governed by $$ \alpha = \frac{2s}{2s + 1/\beta} $$ where $s$ is the source regularity [2509.20721]. Higher redundancy (flatter spectrum, smaller $\beta$) slows learning (smaller $\alpha$). Random feature models and kernel regression provide analytic tractability to explore how power-law spectra of features or kernels translate into observable scaling laws in predictive accuracy, exposing the role of nonlinear activations in “extending” the effective range of the spectrum [2210.16859, 2503.01314]. Universality is established: the exponent $\alpha$ remains invariant under invertible linear transformation of features and is set by the heaviest-tailed spectral component in mixture distributions. This anchoring of the empirical scaling law to data statistics explains cross-domain ubiquity and why architecture- or resource-based improvements ultimately “run into” the data’s intrinsic redundancy limit. ## 5. Practical Guidance for Model and Resource Allocation The predictive accuracy of scaling laws enables quantitative planning: - **Resource Optimization:** For fixed budgets, scaling laws indicate optimal trade-offs between model size and training data (or compute). For example, in the scaling law $L(N, D)$, the ratio $\alpha_N/\alpha_D$ prescribes how much to grow $D$ when $N$ is increased [2106.09488]: $$ D \propto N^{\alpha_N/\alpha_D} $$ - **Early-Stage Model Selection:** Extrapolation from small-scale models accurately forecasts large-scale performance, enabling efficient pre-selection of architecture, optimization strategy, and pretraining objectives [2202.06387]. - **Hyperparameter Tuning:** Explicit expressions for test loss as a function of batch size, training steps, and learning rate enable estimation of optimal batch sizes and convergence trajectories with minimal experimentation [2403.06563, 2410.05661]. - **Training Regimes:** Results suggest that full convergence is not always computationally optimal; training larger models for fewer updates may reach a performance frontier more efficiently [2106.09488]. For MoE models, the scaling law justifies allocation of budget to increase model width rather than only data [2410.05661]. - **Performance Forecasting:** Scaling laws allow scientists to estimate the required resource expansion to reach specified error thresholds, crucial for long-term research and hardware planning in large-scale AI system development [2208.08489, 2509.21811]. ## 6. Extensions and Open Questions Recent research generalizes scaling laws to new contexts: - **Uncertainty Quantification:** Predictive uncertainty, both epistemic and aleatoric, across vision and language models, is found to follow scaling laws with dataset size, contradicting the popular notion that "so much data" makes Bayesian treatment unnecessary [2506.09648]. - **Multilingual and Multi-domain Models:** A proposed scaling law for multilingual LMs reveals that, for each language family, performance improvement depends only on its own sampling ratio, not on the data distribution of other families—thus, global optimal sampling proportions can be derived from small-scale models and scale to large settings [2410.12883]. - **Architectural Breadth:** Comparative studies show that MoE and dense models share underlying scaling frameworks, but differences in noise scale and regularization affect optimal batch size and learning rate prescriptions [2410.05661]. - **Time Series Models:** Power-law scaling persists for time series foundation models (TSFMs) in both in- and out-of-distribution generalization, with encoder-only architectures displaying superior scalability [2410.12360]. - **Material Science:** Scaling exponents in material property prediction are architecture-dependent; physically-informed models (e.g., EquiformerV2) more efficiently convert added capacity or data into error reduction than unconstrained Transformers [2509.21811]. Open challenges include understanding architectural factors that break or accelerate the observed power law, scaling behaviors in highly multi-modal or imbalanced domains, and universality limits in extremely overparameterized regimes or non-i.i.d. training scenarios. ## 7. Summary Table: Representative Scaling Law Instances | Domain/Task | Scaling Formula | Exponent behavior/Notes | |-------------------------|-----------------------------------------------------|--------------------------------------------------------| | Language Modeling [2403.06563] | $L(N) = (N_c/N)^{\alpha_N}$ | Valid up to 33B params; constants depend on setup | | Acoustic Models [2106.09488] | $L(N, D)$ as above | Irreducible loss $L_\infty$ sets floor | | Recommendation [2208.08489] | $L(x) = \alpha x^{-\beta} + \gamma$ | Parameter scaling saturated; data/compute still help | | Kernel Regression [2509.20721] | $E[\mathcal{E}] \sim n^{-\alpha}$ | $\alpha = 2s/(2s+1/\beta)$; $\beta$ from spec. tail | | Multilingual LMs [2410.12883] | $L_i(N, D, p_i)$ power law in each $p_i$ | Opt. ratios from small models generalize to large | | Uncertainty [2506.09648] | $EU \propto 1/N^\gamma$ | Epistemic uncertainty decays with more data, nonzero | | RL [2210.00849] | Elo scales log-linear in $\log N$ | Strength $\propto N^\alpha$, optimality computable | | Materials [2509.21811] | $L = \alpha N^{-\beta}$ | Exponent $\beta$ strong function of architecture | This table typifies the generality and predictive power of scaling laws. The actual formulas and exponent values are determined empirically or theoretically based on the domain, model, and data spectrum. --- Scaling laws provide a unified, predictive, and theoretically motivated framework for understanding how model performance improves with increased resources. They reveal the central role of data redundancy, architectural inductive biases, and resource allocation in dictating learning efficiency and guiding future advances in large-scale machine learning. Source: https://www.emergentmind.com/topics/model-scaling-laws