---
title: 'Chinchilla Approach 2: IsoFLOP Profiling'
url: https://www.emergentmind.com/topics/chinchilla-approach-2
type: topic
---

# Chinchilla Approach 2: IsoFLOP Profiling

Chinchilla Approach 2 is the IsoFLOP-profile method for estimating compute-optimal model and data allocation in the Chinchilla scaling-law program: models of varying sizes are trained at fixed compute budgets, each fixed-budget loss curve is approximated by a parabola in log-size, and the minima across budgets are regressed to recover the optimal scaling exponents. In later literature, the same label is sometimes used more loosely for second-generation Chinchilla-style programs that retain the loss-law perspective while changing the optimization target or the decision variables, but the canonical meaning remains the IsoFLOP parabola-fitting procedure [2404.10102] [2603.22339].

## 1. Historical position within the Chinchilla program

In the terminology used by the replication literature, Hoffmann et al.’s three estimation procedures are: **Approach 1**, “Training models of fixed sizes on varying numbers of tokens”; **Approach 2**, “Training models of varying sizes targeting fixed compute budgets (IsoFLOP profiles)”; and **Approach 3**, “Fitting a parametric model of the loss as a function of model size and training tokens” [2404.10102]. Approach 2 is therefore not a separate scaling law in functional form; it is an estimation procedure for the compute-optimal frontier.

Its historical importance is that it supplied direct empirical evidence for the Chinchilla prescription that, for compute-optimal training, model size and the number of training tokens should be scaled equally, with the practical rule of thumb near the Chinchilla operating point being roughly 20 tokens per parameter [2404.10102]. Later reconciliation work argued that this near-equal scaling is the preferred asymptotic description when one uses total parameters and total compute, yielding \(N_{\mathrm{optimal}} \propto C^{0.50}\) and \(D_{\mathrm{optimal}} \propto C^{0.50}\), whereas Kaplan-like exponents are largely a consequence of non-embedding parameter counting combined with small-scale fitting [2406.12907].

## 2. Formal procedure and mathematical structure

The standard Chinchilla loss surface is written as
\[
L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},
\]
with approximate training compute
\[
C \approx 6ND.
\]
Under this compute constraint, the compute-optimal allocations satisfy
\[
N^* \propto C^a \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta}
\]
and
\[
D^* \propto C^b \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta}.
\]
Approach 2 does not fit all five surface parameters \(E, A, B, \alpha, \beta\). Instead, it estimates the actionable allocation exponents \(a\) and \(b\) from IsoFLOP experiments [2603.22339].

The procedure has three stages. First, one samples IsoFLOP contours: for each compute budget \(C\), models are trained at various \((N,D)\) satisfying \(C = 6ND\). Second, for each budget one fits a parabola in log-size,
\[
L = p(\log N)^2 + q(\log N) + r,
\]
and extracts the minimizing \(N^*\). Third, one regresses \(\log N^*\) against \(\log C\) to recover exponent \(a\), and similarly for \(D^*\) to recover \(b\) [2603.22339].

The approximation behind this method is explicitly local. Along a fixed-compute contour, the true loss is approximated near its optimum by a second-order Taylor expansion in a centered log-coordinate \(w\),
\[
L(w) \approx L(0) + \frac{1}{2}L''(0)w^2.
\]
This makes the method attractive because it avoids a joint nonlinear fit over the full loss surface, but it also defines the source of its later criticisms [2603.22339].

## 3. Relation to Approaches 1 and 3

Approach 2 has often been treated as a benchmark for judging the plausibility of full-surface fits. The replication study of the Chinchilla scaling law did not independently rerun Approach 2, but it repeatedly used Approaches 1 and 2 as comparators and consistency checks for Approach 3. Its central finding was that the published Approach 3 parameters were numerically mis-estimated because of optimizer early stopping and harmful rounding, yet a corrected re-fit of Approach 3 moved toward Approach 2 rather than away from it [2404.10102].

This point is clearest in the compute-allocation exponent
\[
a = \frac{\beta}{\alpha+\beta}.
\]
Using the published Hoffmann et al. parameters, the replication reports \(a \approx 0.454\), which implies much more data per parameter as compute grows. Using the replication’s corrected fit, it reports \(a \approx 0.5126\), much closer to equal scaling and therefore much closer to the policy implied by Approaches 1 and 2 [2404.10102]. The same paper states that Hoffmann et al.’s published Approach 3 parameters implied approximately 70 tokens per parameter at Chinchilla-relevant scales, whereas Approaches 1 and 2, the actual Chinchilla 70B training choice, and the corrected Approach 3 fit all aligned around approximately 20 tokens per parameter [2404.10102].

A common misconception is therefore that disagreement between published Approach 3 coefficients and the Chinchilla recipe was evidence against Approach 2. The replication argues the opposite: the discrepancy was likely an artifact of inaccurate parameter estimation in Approach 3, while Approach 2 likely got the practical scaling policy basically right [2404.10102].

## 4. Systematic-bias critique and the parabola-fit controversy

A later critique reframed Chinchilla Approach 2 as the now-standard IsoFLOP parabola-fitting method and argued that its parabolic approximation introduces systematic biases in compute-optimal allocation estimates even on noise-free synthetic data [2603.22339]. The critique identifies three sources of error.

The first is **IsoFLOP sampling grid width**, or Taylor-approximation accuracy. Even with perfectly centered grids and no noise, the fixed-compute loss curve is not actually a parabola. For the Chinchilla surface with \(\alpha=0.34,\beta=0.28\), the reported intercept error grows from **0.3%** on a \(\pm 2\times\) grid to **4.1%** on a \(\pm 16\times\) grid. At \(10^{24}\) FLOPs, extrapolated from fits on \(10^{17}\)–\(10^{21}\) FLOPs, the corresponding token underestimation grows from **0.3%** on XS grids to **5.1%** on XL grids [2603.22339].

The second is **uncentered IsoFLOP sampling**. In practice the grid is centered on a guess rather than the true optimum. If the multiplicative offset is constant across budgets, the intercept is biased while the exponent remains essentially correct. If the offset drifts across budgets, both intercepts and exponents are biased [2603.22339].

The third is **loss-surface asymmetry**, meaning \(\alpha \neq \beta\). In the idealized centered, noiseless analysis, symmetric surfaces make the quadratic approximation exact enough that the fitted vertex lands correctly, whereas asymmetric surfaces preserve odd-order terms and displace the fitted minimum. On the Chinchilla-like asymmetric surface the paper reports essentially zero exponent error but a **-4.1%** relative error in the intercept term \(b_0\); on a more asymmetric synthetic surface with \(\alpha=0.46,\beta=0.15\), it reports a **-8.5%** relative error in \(b_0\) [2603.22339].

The empirical headline of this critique comes from digitized Llama 3 IsoFLOP data. The paper reports that the induced misallocation corresponds to **6.5%** of the \(3.8\times 10^{25}\) FLOP training budget and **\$1.4M** in unnecessary compute at **50% H100 MFU**, with a **90% CI: \$412K-\$2.9M** [2603.22339]. Its constructive response is to prefer direct surface fitting—its “Approach 3,” especially via Variable Projection with Non-negative Least Squares—because that removes the structural bias introduced by the parabolic approximation [2603.22339].

## 5. Broader and later uses of the label

A separate, later usage treats “Approach 2” as a second-generation Chinchilla program rather than the original IsoFLOP protocol. This suggests that the label has become a convenient shorthand for retaining the Chinchilla loss-law spirit while changing the objective or the control variables.

| Paper | Retained Chinchilla element | Added objective or variable |
|---|---|---|
| [2401.00448] | \(L(N,D_{\text{tr}})\) with Chinchilla constants | Minimize training plus lifetime inference cost at fixed quality |
| [2604.01411] | Chinchilla loss plus pass@\(\!k\) modeling | Jointly optimize \(N,D,k\) under \(C_{\text{train}}\) and \(C_{\text{inf}}\) |
| [2607.01487] | Compute-optimal loss law | Split \(D\) into batch size \(M\) and steps \(K\) with \(D=MK\) |
| [2602.07712] | Chinchilla-style law in \(N,D\) | Shared exponents with optimizer-specific \(\rho_N,\rho_D\) |
| [2605.09154] | Fit-a-loss-model-and-optimize-it paradigm | Direct loss prediction over \((N,B,K)\) under compute, time, memory |

The deployment-aware extension “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws” keeps the Chinchilla loss law,
\[
L(N, D_{\text{tr}}) = E + \frac{A}{N^\alpha} + \frac{B}{D_{\text{tr}}^\beta},
\]
but changes the optimization target to total lifecycle compute,
\[
6N D_{\text{tr}} + 2N D_{\text{inf}},
\]
at fixed quality. Its main claim is that when inference demand is substantial, the optimal frontier shifts toward smaller models trained on more tokens than standard Chinchilla recommends—“smaller and longer” training [2401.00448].

The Train-to-Test framework goes further by jointly optimizing model size, training tokens, and number of inference samples under fixed end-to-end budgets. Its accuracy-based formulation, explicitly called **Approach 2**, models pass@\(\!k\) via a Beta distribution over per-question single-pass accuracies and makes the feedback
\[
k = \frac{C_{\text{inf}}}{2N}
\]
central to compute-optimality; across eight downstream tasks it forecasts that the optimum shifts radically into the overtraining regime, and the authors describe the practical recommendation as “train a smaller model for longer” [2604.01411].

Other extensions preserve the Chinchilla objective while refining the internal variables. One splits the token budget into batch size \(M\) and number of optimization steps \(K\), proposing
\[
\mathcal{L}(N,M,K)=E+\frac{A}{N^\alpha}+\frac{B}{M^\beta}+\frac{C}{K^\gamma},
\]
and derives
\[
M^\star \propto D^{\frac{\gamma}{\beta+\gamma}}, \qquad K^\star \propto D^{\frac{\beta}{\beta+\gamma}}
\]
under \(D=MK\) [2607.01487]. Another proposes optimizer-aware scaling with shared exponents and optimizer-specific rescaling factors,
\[
L = \frac{A}{(N \cdot \rho_N)^\alpha} + \frac{B}{(D \cdot \rho_D)^\beta} + E,
\]
arguing that optimizer differences appear primarily in data efficiency rather than parameter efficiency [2602.07712]. A further successor-style framework replaces the heuristic \((N,D)\) law with a direct loss predictor over \((N,B,K)\), the Noisy Quadratic System, and reports better extrapolation than Chinchilla when batch size varies and when optimization is performed under compute, time, memory, or compound constraints [2605.09154].

## 6. Significance, limits, and current status

In its original sense, Chinchilla Approach 2 became influential because it turns compute-optimal scaling into an experimentally accessible procedure: one needs IsoFLOP sweeps and local quadratic fits rather than a full nonlinear fit of the loss surface. That convenience explains why it became widely used [2603.22339]. It also explains why later work continued to preserve its basic intuition even when the objective shifted from training-only compute to lifetime deployment compute, optimizer-aware efficiency, or internal token-allocation choices [2401.00448] [2602.07712].

Its limits are now comparatively well defined. The parabola-fit version is exact only in a narrow ideal regime: symmetric loss surfaces, perfectly centered IsoFLOP sampling, and sufficiently local grids [2603.22339]. Its practical interpretation also depends on parameter-counting conventions: reconciliation work argues that total parameters and total compute should be used, and that small-scale non-embedding conventions can produce biased local exponents that look Kaplan-like rather than Chinchilla-like [2406.12907].

A broader theoretical reinterpretation goes further and argues that current LLMs may belong to a universality class with fixed exponents—time or data exponent \(1/3\), width exponent \(1\), and depth exponent \(1\)—so that the central empirical task is no longer repeated re-estimation of exponents but understanding the coefficients that determine optimal model shape, token-to-parameter ratio, and the compute-optimal frontier [2606.25008]. This suggests that the enduring legacy of Chinchilla Approach 2 is methodological rather than dogmatic: it established fixed-compute allocation as a central object of study, but later work increasingly treats the original IsoFLOP parabola fit as one estimator among several, not as the final form of compute-optimal scaling.

Source: https://www.emergentmind.com/topics/chinchilla-approach-2