Chinchilla Approach 2: IsoFLOP Profiling

• Chinchilla Approach 2 is the IsoFLOP-profile method that estimates compute-optimal allocation by fitting a parabola to log-size loss curves under a fixed compute budget.
• It involves sampling fixed-compute contours and regressing the log-minima to recover scaling exponents for optimal model size and training tokens.
• Recent critiques address potential biases from the parabolic approximation, prompting alternative methods for more accurate compute-optimal scaling estimates.

Chinchilla Approach 2 is the IsoFLOP-profile method for estimating compute-optimal model and data allocation in the Chinchilla scaling-law program: models of varying sizes are trained at fixed compute budgets, each fixed-budget loss curve is approximated by a parabola in log-size, and the minima across budgets are regressed to recover the optimal scaling exponents. In later literature, the same label is sometimes used more loosely for second-generation Chinchilla-style programs that retain the loss-law perspective while changing the optimization target or the decision variables, but the canonical meaning remains the IsoFLOP parabola-fitting procedure (Besiroglu et al., 2024, Czech et al., 21 Mar 2026).

1. Historical position within the Chinchilla program

In the terminology used by the replication literature, Hoffmann et al.’s three estimation procedures are: Approach 1, “Training models of fixed sizes on varying numbers of tokens”; Approach 2, “Training models of varying sizes targeting fixed compute budgets (IsoFLOP profiles)”; and Approach 3, “Fitting a parametric model of the loss as a function of model size and training tokens” (Besiroglu et al., 2024). Approach 2 is therefore not a separate scaling law in functional form; it is an estimation procedure for the compute-optimal frontier.

Its historical importance is that it supplied direct empirical evidence for the Chinchilla prescription that, for compute-optimal training, model size and the number of training tokens should be scaled equally, with the practical rule of thumb near the Chinchilla operating point being roughly 20 tokens per parameter (Besiroglu et al., 2024). Later reconciliation work argued that this near-equal scaling is the preferred asymptotic description when one uses total parameters and total compute, yielding $N_{\mathrm{optimal}} \propto C^{0.50}$ and $D_{\mathrm{optimal}} \propto C^{0.50}$, whereas Kaplan-like exponents are largely a consequence of non-embedding parameter counting combined with small-scale fitting (Pearce et al., 2024).

2. Formal procedure and mathematical structure

The standard Chinchilla loss surface is written as

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$

with approximate training compute

$C \approx 6ND.$

Under this compute constraint, the compute-optimal allocations satisfy

$$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$

and

$$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$

Approach 2 does not fit all five surface parameters $E, A, B, \alpha, \beta$. Instead, it estimates the actionable allocation exponents $a$ and $b$ from IsoFLOP experiments (Czech et al., 21 Mar 2026).

The procedure has three stages. First, one samples IsoFLOP contours: for each compute budget $C$, models are trained at various $D_{\mathrm{optimal}} \propto C^{0.50}$0 satisfying $D_{\mathrm{optimal}} \propto C^{0.50}$1. Second, for each budget one fits a parabola in log-size,

$D_{\mathrm{optimal}} \propto C^{0.50}$2

and extracts the minimizing $D_{\mathrm{optimal}} \propto C^{0.50}$3. Third, one regresses $D_{\mathrm{optimal}} \propto C^{0.50}$4 against $D_{\mathrm{optimal}} \propto C^{0.50}$5 to recover exponent $D_{\mathrm{optimal}} \propto C^{0.50}$6, and similarly for $D_{\mathrm{optimal}} \propto C^{0.50}$7 to recover $D_{\mathrm{optimal}} \propto C^{0.50}$8 (Czech et al., 21 Mar 2026).

The approximation behind this method is explicitly local. Along a fixed-compute contour, the true loss is approximated near its optimum by a second-order Taylor expansion in a centered log-coordinate $D_{\mathrm{optimal}} \propto C^{0.50}$9,

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$0

This makes the method attractive because it avoids a joint nonlinear fit over the full loss surface, but it also defines the source of its later criticisms (Czech et al., 21 Mar 2026).

3. Relation to Approaches 1 and 3

Approach 2 has often been treated as a benchmark for judging the plausibility of full-surface fits. The replication study of the Chinchilla scaling law did not independently rerun Approach 2, but it repeatedly used Approaches 1 and 2 as comparators and consistency checks for Approach 3. Its central finding was that the published Approach 3 parameters were numerically mis-estimated because of optimizer early stopping and harmful rounding, yet a corrected re-fit of Approach 3 moved toward Approach 2 rather than away from it (Besiroglu et al., 2024).

This point is clearest in the compute-allocation exponent

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$1

Using the published Hoffmann et al. parameters, the replication reports $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$2, which implies much more data per parameter as compute grows. Using the replication’s corrected fit, it reports $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$3, much closer to equal scaling and therefore much closer to the policy implied by Approaches 1 and 2 (Besiroglu et al., 2024). The same paper states that Hoffmann et al.’s published Approach 3 parameters implied approximately 70 tokens per parameter at Chinchilla-relevant scales, whereas Approaches 1 and 2, the actual Chinchilla 70B training choice, and the corrected Approach 3 fit all aligned around approximately 20 tokens per parameter (Besiroglu et al., 2024).

A common misconception is therefore that disagreement between published Approach 3 coefficients and the Chinchilla recipe was evidence against Approach 2. The replication argues the opposite: the discrepancy was likely an artifact of inaccurate parameter estimation in Approach 3, while Approach 2 likely got the practical scaling policy basically right (Besiroglu et al., 2024).

4. Systematic-bias critique and the parabola-fit controversy

A later critique reframed Chinchilla Approach 2 as the now-standard IsoFLOP parabola-fitting method and argued that its parabolic approximation introduces systematic biases in compute-optimal allocation estimates even on noise-free synthetic data (Czech et al., 21 Mar 2026). The critique identifies three sources of error.

The first is IsoFLOP sampling grid width, or Taylor-approximation accuracy. Even with perfectly centered grids and no noise, the fixed-compute loss curve is not actually a parabola. For the Chinchilla surface with $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$4, the reported intercept error grows from 0.3% on a $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$5 grid to 4.1% on a $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$6 grid. At $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$7 FLOPs, extrapolated from fits on $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$8–$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$9 FLOPs, the corresponding token underestimation grows from 0.3% on XS grids to 5.1% on XL grids (Czech et al., 21 Mar 2026).

The second is uncentered IsoFLOP sampling. In practice the grid is centered on a guess rather than the true optimum. If the multiplicative offset is constant across budgets, the intercept is biased while the exponent remains essentially correct. If the offset drifts across budgets, both intercepts and exponents are biased (Czech et al., 21 Mar 2026).

The third is loss-surface asymmetry, meaning $C \approx 6ND.$0. In the idealized centered, noiseless analysis, symmetric surfaces make the quadratic approximation exact enough that the fitted vertex lands correctly, whereas asymmetric surfaces preserve odd-order terms and displace the fitted minimum. On the Chinchilla-like asymmetric surface the paper reports essentially zero exponent error but a -4.1% relative error in the intercept term $C \approx 6ND.$1; on a more asymmetric synthetic surface with $C \approx 6ND.$2, it reports a -8.5% relative error in $C \approx 6ND.$3 (Czech et al., 21 Mar 2026).

The empirical headline of this critique comes from digitized Llama 3 IsoFLOP data. The paper reports that the induced misallocation corresponds to 6.5% of the $C \approx 6ND.$4 FLOP training budget and \$C \approx 6ND.$5412K-\$2.9M (Czech et al., 21 Mar 2026). Its constructive response is to prefer direct surface fitting—its “Approach 3,” especially via Variable Projection with Non-negative Least Squares—because that removes the structural bias introduced by the parabolic approximation (Czech et al., 21 Mar 2026).

5. Broader and later uses of the label

A separate, later usage treats “Approach 2” as a second-generation Chinchilla program rather than the original IsoFLOP protocol. This suggests that the label has become a convenient shorthand for retaining the Chinchilla loss-law spirit while changing the objective or the control variables.

Paper	Retained Chinchilla element	Added objective or variable
(Sardana et al., 2023)	$C \approx 6ND.$6 with Chinchilla constants	Minimize training plus lifetime inference cost at fixed quality
(Roberts et al., 1 Apr 2026)	Chinchilla loss plus pass@$C \approx 6ND.$7 modeling	Jointly optimize $C \approx 6ND.$8 under $C \approx 6ND.$9 and $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$0
(Schaipp, 1 Jul 2026)	Compute-optimal loss law	Split $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$1 into batch size $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$2 and steps $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$3 with $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$4
(Volkova et al., 7 Feb 2026)	Chinchilla-style law in $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$5	Shared exponents with optimizer-specific $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$6
(Li et al., 9 May 2026)	Fit-a-loss-model-and-optimize-it paradigm	Direct loss prediction over $$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$7 under compute, time, memory

The deployment-aware extension “Beyond Chinchilla-Optimal: Accounting for Inference in LLM Scaling Laws” keeps the Chinchilla loss law,

$$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$8

but changes the optimization target to total lifecycle compute,

$$N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}$$9

at fixed quality. Its main claim is that when inference demand is substantial, the optimal frontier shifts toward smaller models trained on more tokens than standard Chinchilla recommends—“smaller and longer” training (Sardana et al., 2023).

The Train-to-Test framework goes further by jointly optimizing model size, training tokens, and number of inference samples under fixed end-to-end budgets. Its accuracy-based formulation, explicitly called Approach 2, models pass@$$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$0 via a Beta distribution over per-question single-pass accuracies and makes the feedback

$$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$1

central to compute-optimality; across eight downstream tasks it forecasts that the optimum shifts radically into the overtraining regime, and the authors describe the practical recommendation as “train a smaller model for longer” (Roberts et al., 1 Apr 2026).

Other extensions preserve the Chinchilla objective while refining the internal variables. One splits the token budget into batch size $$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$2 and number of optimization steps $$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$3, proposing

$$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$4

and derives

$$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$5

under $$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$6 (Schaipp, 1 Jul 2026). Another proposes optimizer-aware scaling with shared exponents and optimizer-specific rescaling factors,

$$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$7

arguing that optimizer differences appear primarily in data efficiency rather than parameter efficiency (Volkova et al., 7 Feb 2026). A further successor-style framework replaces the heuristic $$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$8 law with a direct loss predictor over $$D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.$$9, the Noisy Quadratic System, and reports better extrapolation than Chinchilla when batch size varies and when optimization is performed under compute, time, memory, or compound constraints (Li et al., 9 May 2026).

6. Significance, limits, and current status

In its original sense, Chinchilla Approach 2 became influential because it turns compute-optimal scaling into an experimentally accessible procedure: one needs IsoFLOP sweeps and local quadratic fits rather than a full nonlinear fit of the loss surface. That convenience explains why it became widely used (Czech et al., 21 Mar 2026). It also explains why later work continued to preserve its basic intuition even when the objective shifted from training-only compute to lifetime deployment compute, optimizer-aware efficiency, or internal token-allocation choices (Sardana et al., 2023, Volkova et al., 7 Feb 2026).

Its limits are now comparatively well defined. The parabola-fit version is exact only in a narrow ideal regime: symmetric loss surfaces, perfectly centered IsoFLOP sampling, and sufficiently local grids (Czech et al., 21 Mar 2026). Its practical interpretation also depends on parameter-counting conventions: reconciliation work argues that total parameters and total compute should be used, and that small-scale non-embedding conventions can produce biased local exponents that look Kaplan-like rather than Chinchilla-like (Pearce et al., 2024).

A broader theoretical reinterpretation goes further and argues that current LLMs may belong to a universality class with fixed exponents—time or data exponent $E, A, B, \alpha, \beta$0, width exponent $E, A, B, \alpha, \beta$1, and depth exponent $E, A, B, \alpha, \beta$2—so that the central empirical task is no longer repeated re-estimation of exponents but understanding the coefficients that determine optimal model shape, token-to-parameter ratio, and the compute-optimal frontier (Liu et al., 23 Jun 2026). This suggests that the enduring legacy of Chinchilla Approach 2 is methodological rather than dogmatic: it established fixed-compute allocation as a central object of study, but later work increasingly treats the original IsoFLOP parabola fit as one estimator among several, not as the final form of compute-optimal scaling.