--- title: Scaling Laws of Large Language Models url: https://www.emergentmind.com/topics/scaling-laws-of-large-language-models type: topic --- # Scaling Laws of Large Language Models Large Language Models (LLMs) exhibit systematic and highly regular trade-offs between model size, dataset scale, compute, architecture, and performance—patterns empirically captured by scaling laws. These empirical or semi-theoretical relations serve as quantitative maps for model design, resource allocation, and performance forecasting in large-scale pretraining and fine-tuning regimes. While initial laws related upstream (pretraining) loss to size and data through power laws, recent work extends these principles to downstream tasks, architectural sparsity, data domain, temporal dynamics, and even direct benchmark scores. This article surveys core mathematical forms, theoretical underpinnings, domain-specific variants, practical implications, and current research directions as documented in recent literature. ## 1. Mathematical Forms of Scaling Laws Scaling laws describe how key metrics (typically loss or downstream accuracy) depend on design and training variables. The archetypal form is a multi-term power law in model size (N or P: non-embedding parameters or total parameters), dataset size (D: tokens), and sometimes compute (C: FLOPs): | Law / Paper | Formula | Regime | |-------------------------------------|------------------------------------------------------------------------------------------|-----------------| | Hoffmann et al. / Chinchilla | $L(N, D) = E + A N^{-\alpha} + B D^{-\beta}$ ($E$: loss floor) | Dense models | | Farseer (Li et al. 2025) | $L(N, D) = \exp(a_3N^\gamma+b_3) + \exp(a_2N^\beta+b_2) D^{-\exp(a_1N^\alpha+b_1)}$ | Unified | | Code LLMs (Farseer fit) | Same as above, but with distinct coefficients reflecting higher data hunger | Code models | | MoE / Sparse (Hossain et al. 2025) | $L(N, D, S) = e(1-S)^\gamma + [a(1-S)^\alpha + cS]N^{-\alpha} + b D^{-\beta}$ | Dense/Sparse | Parameters $A,B,E,\alpha,\beta$ are empirically fitted. Power-law exponents typically fall in $[0.2, 0.6]$; $E$ is set by data/architecture, and constants reflect domain properties. Mixture-of-Experts (MoE) models admit structurally analogous laws with extra scaling in the number of experts $E$: $$ \hat{L}(N,D,E) = \frac{A}{N^\alpha E^\gamma} + \frac{B}{D^\beta} + \sigma $$ with compute budget $C = N D$ [2410.05661]. Code LLMs obey similar power laws, but optimal data-to-parameter ratios ($D/N$) are $7$–$20\times$ higher than for natural language tasks: $D/N \sim 150$–$400$ vs. $D/N \sim 20$ at equal FLOPs [2510.08702]. Refined laws, such as Farseer, further include non-separable interactions and dynamic data exponents, significantly lowering prediction errors and supporting reliable extrapolation across orders of magnitude in $N,D$ [2506.10972]. ## 2. Theoretical Justifications and Regimes Statistical models shed light on why power laws emerge. Maloney et al. [2210.16859] analytically derive scaling regimes by coupling the eigenspectrum of the natural data distribution (typically power-law-tailed) to model complexity via random features and ridge regression. Key implications: - If the eigenvalue spectrum follows $\lambda_I \sim I^{-(1+\alpha)}$, test loss asymptotically behaves as $L \sim N^{-\alpha}$ or $L \sim D^{-\alpha}$, saturating at a noise floor dictated by data entropy. - Optimal compute allocation occurs at equiparameterization ($N \propto D$), aligning with empirical findings that the lowest loss per FLOP comes when $N$ and $D$ are increased in lockstep. - When model size or data exceed the effective dimension of the latent space (i.e., spectrum support), power-law scaling breaks and loss plateaus. Farseer’s improvement arises from explicitly modeling the non-separability in the $N$–$D$ interaction, accurately capturing the empirical surfaces found in massive LLM training campaigns [2506.10972]. ## 3. Domain- and Architecture-Specific Scaling Patterns Empirical research demonstrates that scaling laws broadly generalize across LLM families but with substantial domain or architecture-induced shifts in coefficients and exponents. - MoE architectures exhibit identical scaling exponents to dense models but larger fractions of compute should be allocated to model scale ($N\sim C^{0.59}$, $D\sim C^{0.41}$ for $E=8$) [2410.05661]. MoE models achieve up to $16$\% better data efficiency at fixed compute than comparable dense models. - Code LLMs demand significantly higher data/parameter ratios: optimal $D/N$ grows super-linearly with compute; mixture experiments confirm that a moderate natural language fraction can help small code models in data-scarce regimes but degrades performance at large scale [2510.08702]. - Sparse/pruned models interpolate between dense and fully sparse regimes, with performance given by $L(N,D,S)$, where $S$ is sparsity. This general law exactly recovers dense scaling at $S=0$ and matches MoE/pruning curves up to $S=0.98$, ensuring optimal resource trade-off between $N$, $D$, and $S$ for a fixed compute budget [2508.06617]. ## 4. Extensions: Temporal, Downstream, and Context-Aware Laws Modern scaling law research now addresses aspects beyond upstream loss: - **Temporal Laws**: Model test loss evolution through training, at both sequence-level and token-position granularity. The temporal scaling law uses a dynamic hyperbolic fit for per-token loss, parameterized by position and step. Unlike power-law fits, the temporal law achieves near-perfect $R^2$ on both in-domain and out-of-distribution sets, enabling early hyperparameter selection and accurate prediction of training trajectories [2404.17785]. - **Downstream Performance**: For metrics such as BLEU (MT), benchmark accuracy (MMLU), or few-shot task scores, specialized scaling forms are required. - Machine translation scales as a log-law in BLEU (if pretraining and target distributions are well aligned), but cross-entropy can monotonically decrease even when BLEU no longer improves, indicating distribution misalignment risk [2402.04177]. - The “Performance Law” directly predicts MMLU from model shape, data scale, and a training-instability penalty, achieving $\sim$3-point average error on held-out models and extending to both dense and MoE [2408.09895]. - **Context-aware Laws**: Downstream task performance depends jointly on training compute and the length of in-context demonstrations. These are captured by a saturating power-law in compute, a saturating power-law in context length, and a context-window penalty. Different tasks exhibit distinct “characteristic context” scales; e.g., arithmetic reasoning benefits from many-shots, translation saturates after a few demonstrations [2510.14919]. - **Loss-to-loss Laws**: The downstream loss on any evaluation task is a universal, shifted power law of pretraining validation loss; the dominant influence is the pretraining data distribution—not model size, architecture, or hyperparameters [2502.12120]. ## 5. Empirical Methodologies and Validation Scaling law analysis unites large-scale grid sweeps, statistical surface fitting, and robust extrapolation validation. - Comprehensive grids of (model size, dataset size) are constructed, with held-out validation runs and test losses measured on high-quality data splits [2506.10972, 2510.08702]. - Nonlinear and log-linear fits are benchmarked by mean-relative error, $R^2$, and the magnitude of extrapolation errors on out-of-domain (off-grid or ultra-large-scale) data points [2506.10972]. - Modern methodologies emphasize piecewise and differential fitting, ensuring that fitted exponents and offset constants generalize across regimes rather than overfit a narrow slice. - Newer laws (e.g., Farseer) demonstrably outperform older power-law models, reducing average relative error by several multiples and maintaining percent-level accuracy when predicting models $>10\times$ larger than any in the training set [2506.10972, 2510.08702]. ## 6. Practical Guidelines and Trade-Offs in LLM Design Scaling laws provide closed-form formulas and explicit recipes for model design, resource allocation, and cost–performance optimization. - For a fixed compute budget, optimal model/data allocations are given by scaling exponents (e.g., $N_{\rm opt} \sim C^{0.464}$, $D_{\rm opt} \sim C^{0.536}$ for vanilla dense models) [2412.01505]. - In code LLMs, always budget for much higher $D/N$; for $C\sim10^{21}$ FLOPs, allocate $D/N\sim 150$–$200$ [2510.08702]. - Sparse/Pruned networks: Use joint optimization of $N$, $D$, and $S$ via the generalized law to maximize gains for a specific workload [2508.06617]. - For merging specialist models (like adapters), performance follows a $L(N,k)=L_\infty(N)+A(N)/(k+b)$ law. Most gains arrive by $k=5$–$6$ experts, and diminishing returns thereafter are mathematically predictable. This enables optimal mix of expert acquisition and base scaling for efficient ensemble construction [2509.24244]. - For downstream score prediction (e.g., MMLU), performance is not a trivial function of loss. Use log-linear regression on model configuration and data scale, and apply hardware/instability discounts. This allows performance planning and leakage/quality diagnosis [2408.09895]. - In few-shot and context-extended deployment, leverage context-aware scaling to decide between investing in model scale or context-window extensions based on the task’s saturation profile [2510.14919]. ## 7. Open Directions and Future Methodologies Recent position work advocates recasting scaling law discovery as an inverse problem: Given a desired performance threshold, find the minimal ingredients (model/data/compute/annotation) to achieve it under resource constraints, potentially exposing stage-wise and hybrid scaling laws that extend well beyond classical power-law regimes [2509.07909]. New target areas include: - **Data selection scaling**: Discovering minimal necessary data for target metrics under optimal curation. - **Inference scaling**: Jointly optimizing over architectures, inference strategies, and context to minimize cost for a given utility. - **Machine unlearning scaling**: Quantifying cost–performance trade-offs for removing (unlearning) data while retaining utility on retained data. Actual derivations, empirical coefficients, and validation in these new inverse-problem contexts remain open for future research. --- Scaling laws in LLMs currently anchor quantitative planning for model training, compute investment, and deployment policy. The research trajectory now moves toward generalizing these laws to cover sparsity, domain shift, explicit downstream utility, temporally evolving training, context-driven inference, and compositional/ensemble intelligence, with the aim to precisely map performance landscapes for next-generation AI systems. Source: https://www.emergentmind.com/topics/scaling-laws-of-large-language-models