Chinchilla Approach 3: Loss Scaling Law

• Chinchilla Approach 3 is a scaling law that jointly models loss as a function of parameters (N) and training tokens (D) using the form L = E + A/N^α + B/D^β.
• It analytically derives the compute-optimal frontier by solving a constrained optimization problem under fixed compute, contrasting with slice-based estimation methods.
• The method has spurred replication debates and influenced extensions to handle inference cost, optimizer choices, and batch-size optimizations in modern LLM deployments.

Chinchilla Approach 3 is the parametric scaling-law procedure that models final pretraining loss as a joint function of model size and training data, then derives the compute-optimal allocation analytically under a compute constraint. In its standard form, the loss surface is written as $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}$, where $N$ is parameter count and $D$ is training-token count. Unlike slice-based procedures that estimate optimality directly from one-dimensional views of the data, Approach 3 fits a global loss surface and infers the compute-optimal frontier from that fitted surface. Subsequent work has made this formulation central both to replication debates about the original Chinchilla estimates and to later extensions that incorporate inference cost, optimizer choice, batch size, training steps, and test-time sampling (Besiroglu et al., 2024).

1. Position within the Chinchilla methodology

Approach 3 is best understood relative to the other two Chinchilla estimation procedures. Approach 1 holds model size fixed and varies training tokens; Approach 2 studies approximately fixed-compute “IsoFLOP profiles”; Approach 3 instead fits a parametric loss law jointly over many runs and then optimizes that law under a compute constraint (Besiroglu et al., 2024).

Approach	Estimation strategy	What is inferred
1	Hold model size fixed and vary training tokens	Where extra data stops being worthwhile for each size
2	Compare runs at approximately fixed compute budgets	Best $N,D$ combinations on IsoFLOP slices
3	Fit a parametric loss law jointly	Compute-optimal frontier from the fitted surface

This distinction is substantive. Approaches 1 and 2 estimate the frontier more directly from empirical slices of the run matrix. Approach 3 estimates it indirectly by assuming a specific functional form for the full loss surface and then solving the constrained optimization problem implied by that form. This makes Approach 3 the most reusable of the three procedures, because once the surface is fitted, compute-optimal allocations at unobserved budgets follow analytically. It also makes it the most sensitive to fitting quality, parameterization, and optimization details.

2. Parametric loss law and derivation of the compute-optimal frontier

The defining equation of Approach 3 is the separable loss model

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$

where $E$ is an irreducible loss floor, $A$ and $B$ are scale coefficients, and $\alpha,\beta$ are the parameter- and data-scaling exponents (Besiroglu et al., 2024). The replication literature describes the fitting procedure in log-space via a log-sum-exp representation,

$$\min_{a,b,e,\alpha,\beta} \sum_{\text{Run } i} \mathrm{Huber}_\delta\!\left(\mathrm{LSE}(a-\alpha\log N_i,\; b-\beta\log D_i,\; e)-\log L_i\right),$$

with $N$0, $N$1, $N$2, and $N$3. The initialization grid reported for this procedure is

$N$4

$N$5

The optimization becomes a scaling law when combined with the standard training-compute approximation $N$6. Under the constraint $N$7, minimizing $N$8 yields the power-law frontier

$N$9

with

$D$0

A particularly important implication is

$D$1

If $D$2, the token/parameter ratio is therefore roughly constant across compute scales. This is the high-level Chinchilla conclusion that model size and data size should scale at approximately the same rate.

3. Replication, numerical pathology, and corrected interpretation

A major later development was a direct replication attempt focused specifically on Approach 3. Because raw logs were unavailable, the reconstruction was performed from the original Figure 1 by downloading the PDF from arXiv, converting it to SVG, extracting point coordinates and colors, mapping those to $D$3, training FLOPs, and loss, and then inferring $D$4 from FLOPs and $D$5. The resulting dataset contained 240 points after excluding five obvious outliers, or 245 with them included. The authors explicitly noted two digitization limitations: a noisy $D$6-axis extraction because the axis lacked tick marks, and color quantization limiting loss precision to about $D$7 (Besiroglu et al., 2024).

The central criticism was not that the functional form itself is unusable, but that the originally published numerical fit and confidence intervals are unreliable. Using the published Approach 3 parameters,

$D$8

the derived exponent is

$D$9

which implies

$N,D$0

That policy suggests about 70 tokens per parameter at the Chinchilla 70B scale, whereas the actual Chinchilla 70B training regime was about 20 tokens per parameter, and Approaches 1 and 2 also support roughly that regime. The replication therefore argued that the original published Approach 3 parameters conflict with both the empirical recommendations and the flagship training recipe.

Their preferred refit, excluding the five outliers, was

$N,D$1

with

$N,D$2

This yields

$N,D$3

which is essentially equal scaling and implies around 20 tokens per parameter, consistent with Chinchilla practice. The replication also reported substantial uncertainty: confidence bands still allow token/parameter ratios from about 4 to 40 at compute scales of $N,D$4 FLOP or more.

Several numerical pathologies were identified. First, the rounded values printed in the original paper materially distort predictions; using $N,D$5 instead of $N,D$6 for $N,D$7 creates an approximately $N,D$8 multiplicative error in the finite-data term at $N,D$9. Second, the reported confidence intervals were implausibly narrow: the replication estimated $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$0, implying an 80% interval width around $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$1, about 50 times wider than the reported 0.001-wide interval. Third, the later explanation attributed those narrow intervals to averaging Huber losses instead of summing them, which altered the scale seen by L-BFGS-B and caused early termination during both the original fit and the bootstrap refits. The resulting consensus is therefore narrow but important: the functional lesson of Approach 3 is retained, whereas the originally published numerical fit is not.

4. Relation to broader scaling-law debates

Approach 3 has also become a reference point in debates about why different scaling-law traditions appear to disagree. One such dispute concerns Kaplan et al. versus Chinchilla. Later work argued that much of the discrepancy can be explained by Kaplan counting non-embedding parameters and fitting in a small-model regime, whereas Chinchilla counts total parameters. Under the Chinchilla-style loss law

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$2

the compute-optimal frontier remains

$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$3

When a Chinchilla-like underlying law is transformed into non-embedding parameter space over Kaplan’s scale range, local effective exponents around $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$4–$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$5 emerge, close to Kaplan’s $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$6. In tiny-model experiments, the same data gave $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$7 in total-parameter space and $L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$8 in non-embedding space, thereby reaffirming the Chinchilla interpretation rather than overturning it (Pearce et al., 2024).

A more ambitious reinterpretation comes from work on neural scaling universality. That position paper proposes fixed exponents for current dense Transformer LLMs—$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta},$9 for training time or tokens, $E$0 for width, and $E$1 for depth—and derives from them an optimal-shape model law with loss improving as $E$2, compute-optimal token/parameter scaling $E$3, and compute-optimal loss scaling $E$4. In that view, the Chinchilla-style conclusion of roughly proportional scaling between parameters and tokens survives, but the key engineering problem shifts from estimating exponents to understanding coefficients such as $E$5, $E$6, $E$7, $E$8, and $E$9 (Liu et al., 23 Jun 2026). This suggests that Approach 3 can be read either as a flexible empirical fit or as an approximation to a more constrained universality class.

5. Deployment-oriented reinterpretations

Later work often retains the Approach 3 loss law but changes the optimization target. The clearest example adds inference cost to the objective while keeping the Chinchilla surface

$A$0

With training FLOPs approximated by $A$1 and inference FLOPs by $A$2, the new problem is to minimize training plus inference compute at fixed target loss. Under sufficiently large inference demand, the optimum moves away from larger, shorter-trained models and toward smaller, longer-trained models. Concrete examples include a 30B-Chinchilla-quality target with $A$3 inference tokens, where total FLOPs can be reduced by 28% by training a 13.6B model on 2.84× the data, and a 30B-quality target at 1.5B requests, where cost can be reduced by 17% by training a 16B model on 3.35T tokens (Sardana et al., 2023).

This deployment-aware reinterpretation is closely related to the design logic of LLaMA. LLaMA states that it is “inspired by the Chinchilla scaling laws,” but it explicitly distinguishes its goal from Chinchilla’s: Hoffmann et al. optimize for a fixed training compute budget, whereas LLaMA optimizes for inference efficiency at a target quality. The resulting family trains comparatively small models for a large number of tokens—6.7B and 13.0B on 1.0T tokens, 32.5B and 65.2B on 1.4T tokens—and reports that a 7B model continues to improve even after 1T tokens, despite the cited Chinchilla recommendation of 200B tokens for a 10B model. LLaMA therefore follows the Chinchilla philosophy of smaller models trained on more data, but adapts it toward inference-efficient deployment rather than strict training-compute optimality (Touvron et al., 2023).

A further extension incorporates repeated sampling at test time. Train-to-Test ($A$4) scaling adds a test-time sample count $A$5 and either augments the loss law with a $A$6-term,

$A$7

or models downstream pass@$A$8 directly. Under fixed train-plus-test budgets, both formulations shift the optimum sharply into the overtraining regime: smaller models, more training tokens per parameter, and more test-time samples. The paper fit its models on 106 checkpoints, then extrapolated into an overtrained region and validated the forecast with 21 additional overtrained checkpoints, reporting relative absolute error of 2.8% for the loss-based formulation and 8.4% for the accuracy-based one (Roberts et al., 1 Apr 2026).

6. Generalizations, replacements, and current methodological status

Approach 3 has also been generalized along axes other than deployment. One line of work studies optimizer choice. It argues that fitting a separate Chinchilla-style law

$A$9

for each optimizer is ill-conditioned, with highly correlated parameter pairs $B$0 and $B$1. The proposed remedy is a shared-exponent law

$B$2

where $B$3 and $B$4 are optimizer-specific rescaling factors relative to a reference optimizer, typically AdamW. In the reported experiments, this shared law reduced extrapolation error by more than $B$5, and most optimizer gains appeared in $B$6 rather than $B$7, suggesting primarily data- or optimization-efficiency gains rather than changes in parameter-efficiency scaling (Volkova et al., 7 Feb 2026).

Another generalization decomposes the token budget itself. Instead of treating $B$8 as a monolithic resource, the three-term law writes

$B$9

with $\alpha,\beta$0, where $\alpha,\beta$1 is token batch size and $\alpha,\beta$2 is the number of optimization steps. At fixed $\alpha,\beta$3, this yields a nontrivial optimal batch size

$\alpha,\beta$4

and on the Li dataset the reported fit was

$\alpha,\beta$5

Because it uses suboptimal batch-size runs instead of only the optimal frontier, this law was reported to recover batch-scaling behavior with far fewer runs; with two batch sizes per sweep, or 28% of the original runs, it still recovered $\alpha,\beta$6 (Schaipp, 1 Jul 2026).

The most direct methodological defense of Approach 3 addresses the common preference for Approach 2. That work argues that the parabolic IsoFLOP approximation in Approach 2 introduces structural bias, especially when $\alpha,\beta$7, the sampling grid is wide, or the sampled points are off-center. It then rehabilitates Approach 3 by exploiting partial linearity. For fixed $\alpha,\beta$8, the model is linear in $\alpha,\beta$9, so variable projection reduces the nonlinear search to two dimensions: $$\min_{a,b,e,\alpha,\beta} \sum_{\text{Run } i} \mathrm{Huber}_\delta\!\left(\mathrm{LSE}(a-\alpha\log N_i,\; b-\beta\log D_i,\; e)-\log L_i\right),$$0 The reported condition number drops from about $$\min_{a,b,e,\alpha,\beta} \sum_{\text{Run } i} \mathrm{Huber}_\delta\!\left(\mathrm{LSE}(a-\alpha\log N_i,\; b-\beta\log D_i,\; e)-\log L_i\right),$$1 in the naive five-dimensional formulation to about $$\min_{a,b,e,\alpha,\beta} \sum_{\text{Run } i} \mathrm{Huber}_\delta\!\left(\mathrm{LSE}(a-\alpha\log N_i,\; b-\beta\log D_i,\; e)-\log L_i\right),$$2 in the reduced two-dimensional problem, and on digitized Llama 3 IsoFLOP data the paper estimates that Approach 2 implied 6.5% deadweight compute loss, or \$1.4M in unnecessary compute at 50% H100 MFU (Czech et al., 21 Mar 2026).

Finally, some work treats Approach 3 not as the endpoint but as a heuristic reduced form to be replaced. The Noisy Quadratic System predicts loss from $$\min_{a,b,e,\alpha,\beta} \sum_{\text{Run } i} \mathrm{Huber}_\delta\!\left(\mathrm{LSE}(a-\alpha\log N_i,\; b-\beta\log D_i,\; e)-\log L_i\right),$$3 rather than $$\min_{a,b,e,\alpha,\beta} \sum_{\text{Run } i} \mathrm{Huber}_\delta\!\left(\mathrm{LSE}(a-\alpha\log N_i,\; b-\beta\log D_i,\; e)-\log L_i\right),$$4, explicitly modeling approximation, optimization bias, and batch-size-controlled stochastic variance. It is presented as a mechanistic alternative that can handle changing batch size and optimize under compound time, memory, and compute constraints. In the reported extrapolation experiments it outperformed the Chinchilla baseline at holdout compute gaps up to 1000 folds, while still containing a Chinchilla-like asymptotic regime when the variance term is ignored (Li et al., 9 May 2026).

Taken together, these developments place Chinchilla Approach 3 in a dual role. Historically, it is the parametric surface-fitting method within the original Chinchilla methodology. Methodologically, it has become the template from which later work either derives corrected equal-scaling conclusions, extends the objective to deployment and test-time regimes, or constructs richer laws over optimizers, steps, batches, and system constraints. Its enduring contribution is the move from frontier estimation by local slices to frontier estimation by fitting and optimizing a global loss surface.