Chinchilla Law: Neural Scaling for Transformers

Updated 28 February 2026

Chinchilla Law is a compute-optimal neural scaling law that balances model size and dataset tokens to minimize loss.
It prescribes optimal allocations for parameters (N) and tokens (D) under fixed compute budgets, typically suggesting tokens-per-parameter ratios near 1–20.
Empirical validations across transformer architectures show robust power-law behavior with near-equal scaling exponents, guiding modern language model design.

Chinchilla Law is a compute-optimal neural scaling law that characterizes the asymptotic relationship between model size, dataset size, and achievable loss for large-scale transformer LLMs. The law prescribes the allocation of model parameters ( $N$ ) and training token budget ( $D$ ) as a function of the available training compute budget ( $C$ ) to minimize validation loss, and has become an empirical standard in both academic and industrial LLM design. The law’s mathematical form and exponents arise from comprehensive regression fits to large-scale transformer model families and are robust to architectural and optimizer variations within standard training regimes.

1. Formulation of the Chinchilla Scaling Law

The Chinchilla Law posits that the minimum achievable next-token cross-entropy loss $L(N,D)$ for a transformer model with $N$ parameters trained on $D$ tokens is well-approximated by a two-dimensional power-law sum with irreducible offset: $L(N,D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$ where $E$ , $A$ , $B$ , $\alpha$ , and $\beta$ are fitting parameters, with $E$ typically interpreted as the loss floor (entropy of the data distribution), and $A$ , $B$ , $\alpha$ , $\beta$ describe the rate at which the reducible loss components decay as model and data scale.

Given a total compute budget $C$ (in FLOPs), which is proportional to $N \times D$ , the compute-optimal regime is inferred by solving the constrained optimization problem $\min_{N D = C / \kappa} L(N, D)$ with the Chinchilla fit. This yields: $N_{\mathrm{opt}}(C) \propto C^{\beta/(\alpha + \beta)} \;,\quad D_{\mathrm{opt}}(C) \propto C^{\alpha/(\alpha + \beta)}$ with typical empirical fits giving $\alpha \approx 0.34-0.37$ , $\beta \approx 0.28-0.37$ , and thus both exponents very close to 0.5, implying $N_{\mathrm{opt}}(C) \approx D_{\mathrm{opt}}(C) \propto \sqrt{C}$ (Pearce et al., 2024, Besiroglu et al., 2024, Barkeshli et al., 15 Jan 2026). This leads to the practical prescription that the model should be trained on a number of tokens roughly equal (up to proportionality constants) to its number of parameters—i.e., tokens-per-parameter ratio near 1–20, depending on context (Song et al., 2024).

2. Theoretical and Empirical Basis

The Chinchilla Law stands in contrast to earlier scaling laws, notably Kaplan et al. (2020), which prescribed a much steeper $N_{\rm opt}(C) \propto C^{0.73}$ . Subsequent work has shown that the Kaplan scaling law overestimates the allocation to $N$ primarily because it excluded embedding parameters and fitted the scaling relationship only at small model scales (Pearce et al., 2024, Porian et al., 2024). Correcting for these factors and performing scale-aware hyperparameter tuning recovers the Chinchilla exponent $a \approx 0.5$ (Porian et al., 2024).

Empirically, the Chinchilla Law has been validated across transformer architectures, tokenization strategies, and open web corpora. Advances in regression methodology—such as fitting one-dimensional slices $L(N)_D$ , then using kernel ridge or neural-net regressors for $L(N,D)$ —have yielded improved fits and robust exponents, confirming the law’s stability (Barkeshli et al., 15 Jan 2026).

3. Origin and Interpretation

A resource-theoretic explanation for the Chinchilla Law is provided by modeling neurons as allocatable resources over the network. The key hypotheses are:

Loss per subtask scales as $1/N$ where $N$ is the subtask’s neuron count;
As the model is widened (and deepened), every subtask receives a homogeneously increased budget;
For transformer-style models with $N_p \sim W^3$ (parameters cubic in width), total neurons per subtask scale as $N_p^{1/3}$ , yielding loss $\propto N_p^{-1/3}$ , which matches the empirically observed $\alpha \approx 0.34$ (Song et al., 2024).

Thus, the Chinchilla exponent emerges from both “neurons as resources” and from a convex-quadratic spectral theory of optimization and approximation error, confirming its relevance to both practical and theoretical settings (Volkova et al., 7 Feb 2026).

4. Practical Computation and Regimes of Validity

Best-fit Chinchilla parameters (derived via bootstrapped nonlinear least-squares fits) for large transformer LMs include, for example (Besiroglu et al., 2024): $A = 482.01 \pm 124.58,\quad B = 2085.43 \pm 1293.23,\quad E = 1.8172 \pm 0.03,\ \alpha = 0.3478 \pm 0.02,\quad \beta = 0.3658 \pm 0.02$ so that

$N_{\mathrm{opt}}(C) \approx 0.12\,C^{0.5126},\quad D_{\mathrm{opt}}(C) \approx 8.35\,C^{0.4874}$

The optimal tokens/parameter ratio then weakly decreases as $C^{-0.025}$ and is empirically near 20 at large scale. Confidence intervals on exponents are at the few percent level, confirming robust near-equality of the growth rates for $N$ and $D$ .

The practical regime of validity: $N \in [10^7, 10^{11}]$ , $D$ up to $10^{13}$ , compute budget up to $10^{24}$ FLOPs. For out-of-domain architectures or training at extreme D/N ratios (e.g., $D/N \gg 10^4$ ), the exponents may drift and empirical gains from extra data diminish, as revealed by coefficient ablation (Sardana et al., 2023).

5. Modifications, Extensions, and Limitations

Impact of Inference and Deployment

Standard Chinchilla Law optimizes only for pretraining FLOPs. Incorporating inference cost (responsible for $2N \cdot R$ FLOPs, with $R$ the total lifetime inference tokens) shifts the compute-optimal prescription toward smaller $N$ and larger $D$ as inference demand increases (Sardana et al., 2023). The resulting optimization,

$\min_{N, D_{\text{tr}}} \ \text{Total}_\text{Compute}(N,D_{\text{tr}};R) \text{ subject to } L(N, D_{\text{tr}})=\ell$

requires numerical solution, but in the limit of $R \gg D$ , optimal $N$ falls, and $D$ rises relative to the Chinchilla-only optimum.

Optimizer- and Hyperparameter-Aware Extensions

Traditional Chinchilla Law assumes all non- $(N,D)$ hyperparameters are optimally tuned. Configuration-to-performance laws (e.g., NCPL) explicitly learn $L(\Phi)$ as a function of full training configuration, capturing effects of batch size, learning rate, optimizer type, schedule, etc., and reducing per-run loss prediction error by 20–40% (Zhang et al., 10 Feb 2026).

Optimizer-aware Chinchilla extensions introduce rescaling factors $(\rho_N^{(o)}, \rho_D^{(o)})$ per optimizer $o$ , holding $(A, \alpha, B, \beta, E)$ constant across optimizers and enabling direct cross-optimizer comparison. Empirically, new optimizers (Muon, SOAP) achieve $\rho_D \in [1.5, 2.5]$ , increasing data efficiency beyond vanilla AdamW (Volkova et al., 7 Feb 2026).

Fitting Methodologies and Robustness

The standard fitting procedure (used by Hoffmann et al.) fits Eq. (1) globally to $(N,D,L)$ data. Alternative methods fit 1D slices (robust to exponents’ drift in $D$ and $N$ ), then regress the surface using fully connected nets or RBF ridge regressors, yielding lower validation MSEs and improved compute-optimal predictions (Barkeshli et al., 15 Jan 2026). Replication efforts highlight the importance of correct data extraction, proper initialization, and rigorous bootstrapping for valid uncertainty quantification—overly tight confidence intervals in the original Chinchilla study likely resulted from statistical underestimation (Besiroglu et al., 2024).

6. Practical Implications and Design Guidelines

Chinchilla Law dictates that, under a fixed compute budget, practitioners should allocate parameters and tokens in near-equal (square-root) proportion, with tokens per parameter typically $\sim20$ but weakly declining with scale.
All parameters—including embeddings—must be included when measuring $N$ ; compute must account for all major FLOP contributors.
For model deployment with significant inference load, the pretraining-optimal Chinchilla ratio is suboptimal: models should be trained smaller and longer to save on inference costs.
Hyperparameter, optimizer choice, and hardware constraints interact with (and may violate) Chinchilla predictions; configuration-aware extensions or direct residual learning atop the Chinchilla fit are required for accurate large-scale forecasts (Zhang et al., 10 Feb 2026).
The law is robust across natural language, synthetic graphs, and simplified LLMs, suggesting its applicability to a broad range of transformer-based systems (Barkeshli et al., 15 Jan 2026).

7. Theoretical Foundations and Scope

The Chinchilla exponents are further justified by spectral theory arguments: loss decomposes into approximation and optimization errors, each admitting power-law decay when the data exhibit a polynomial eigenvalue decay in the Hessian. The sum of these contributions yields the empirical scaling law,

$L(N,D) \sim E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$

with $\alpha$ , $\beta$ determined by the spectral “dimension” of the data/model interaction (Volkova et al., 7 Feb 2026).

A plausible implication is that any architectural or optimizer innovation that alters the effective spectral decay or exploits resource allocation more efficiently could yield sharper scaling (either reducing the constants $A$ , $B$ or increasing exponents), but rigorous validation at large scale remains essential.

For further mathematical details and empirical fits, see (Barkeshli et al., 15 Jan 2026, Pearce et al., 2024, Song et al., 2024, Besiroglu et al., 2024, Sardana et al., 2023, Volkova et al., 7 Feb 2026, Zhang et al., 10 Feb 2026, Porian et al., 2024).