IsoFLOP Scaling Laws

Updated 3 June 2026

IsoFLOP scaling laws are empirical and theoretical relationships that define the trade-off between model parameters and training tokens under constant compute budgets.
They utilize grid sampling, loss surface estimation, and power-law fitting to determine compute-optimal allocations and robust scaling exponents.
The approach also addresses fairness and robustness by analyzing subgroup performance disparities and identifying limitations in data-efficient regimes.

IsoFLOP scaling laws refer to the empirical and theoretical relationships that govern neural model performance as a function of fixed, matched-compute budgets—where model size and data scale are traded off under the constraint of constant total training floating-point-operations (FLOPs). The IsoFLOP methodology establishes a rigorous framework for comparing models on an equal-compute footing, isolating the core trade-offs between model capacity, dataset size, and architectural variation, enabling precise statements about scaling trends, optimal allocation, and the implications of scale for robustness and fairness.

1. Definition and Core Formalism

The IsoFLOP paradigm fixes the total number of FLOPs—denoted $C$ —expended during training, irrespective of the specific balance between model parameter count ( $N$ ), number of training tokens ( $D$ ), or other architectural hyperparameters. For any training configuration, $C$ can be expressed as

$C = f(N, D, T, \dots)$

where $f$ accounts for architecture-dependent FLOP accounting, including quadratic context-scaling for attention-based models, linear scaling for efficient variants, and contributions from tokenization or recurrent loops as appropriate (Beck et al., 2 Oct 2025).

Given this constraint, IsoFLOP scaling laws seek to characterize:

The compute-optimal allocation: what choices of $N$ and $D$ (sometimes including further variables such as loop depth $\mu$ for looped architectures) minimize validation loss $L(N, D)$ at fixed compute $N$ 0.
Power-law relationships: whether $N$ 1 and $N$ 2 follow robust scaling exponents in $N$ 3.
Empirical loss surfaces: how the shape of $N$ 4 or its variants informs algorithmic or architectural recommendations.

Formally, IsoFLOP scaling laws typically take the form:

$N$ 5

where $N$ 6, $N$ 7 are architecture- and dataset-dependent exponents determined empirically or via fitted scaling forms such as the Busbridge or Chinchilla function (Beck et al., 2 Oct 2025, Czech et al., 21 Mar 2026).

2. Methodologies for IsoFLOP Analysis

IsoFLOP studies proceed by sweeping across tuples of $N$ 8 that satisfy $N$ 9 for a sequence of budgets $D$ 0 (commonly ranging from $D$ 1 to $D$ 2 FLOPs in LLM research (Held et al., 28 Oct 2025, Czech et al., 21 Mar 2026)). At each budget:

Grid Sampling: Evaluation points $D$ 3 are chosen on or near the IsoFLOP surface.
Loss Surface Estimation: For each sampled configuration, the post-training validation loss $D$ 4 is measured.
Optimum Identification: A parabola (quadratic polynomial) is often fitted to $D$ 5 at fixed $D$ 6, and the minimum of this fit identifies $D$ 7. An analogous procedure applies for $D$ 8 (Beck et al., 2 Oct 2025, Czech et al., 21 Mar 2026, Subramanian et al., 26 Mar 2026).
Power-law Fitting: The exponents $D$ 9, $C$ 0 are extracted via linear regression of $C$ 1 and $C$ 2 versus $C$ 3.

Tables summarizing the procedure:

Step	Description
Budget selection	Choose $C$ 4 from $C$ 5– $C$ 6 FLOPs
IsoFLOP sampling	Sample $C$ 7 with $C$ 8
Loss measurement	Train each config, record $C$ 9
Optimal estimate	Fit $C = f(N, D, T, \dots)$ 0/ $C = f(N, D, T, \dots)$ 1, find parabola minimum
Scaling fit	Regress $C = f(N, D, T, \dots)$ 2, $C = f(N, D, T, \dots)$ 3 vs. $C = f(N, D, T, \dots)$ 4

Advanced analyses use direct surface fitting such as the Chinchilla-style model:

$C = f(N, D, T, \dots)$ 5

Solving for the compute constraint and minimizing $C = f(N, D, T, \dots)$ 6 over $C = f(N, D, T, \dots)$ 7 yields optimal allocation rules with closed-form exponents (e.g., $C = f(N, D, T, \dots)$ 8) (Czech et al., 21 Mar 2026).

3. Key Empirical Findings Across Domains

IsoFLOP scaling laws display consistent power-law structure in a variety of neural and scientific domains, but with exponents and optima sensitive to architecture, learning regime, and dataset characteristics:

LLMs

For decoder-only Transformers (e.g., Qwen 3) trained on public datasets, compute-optimal model size and token count follow:

$C = f(N, D, T, \dots)$ 9

with loss minima robust to $f$ 0, confirmed by $f$ 1 (Held et al., 28 Oct 2025, Beck et al., 2 Oct 2025, Subramanian et al., 26 Mar 2026).

In xLSTM vs Transformer studies, $f$ 2, $f$ 3, $f$ 4 for both—implying xLSTM scales slightly more compute into parameters at fixed FLOP (Beck et al., 2 Oct 2025).
For weather emulation with Swin Transformers, $f$ 5, $f$ 6 mirrors LLM findings, until data limitation causes early saturation (Subramanian et al., 26 Mar 2026).

Looped Architectures (Parcae)

For fixed-parameter looped models, optimal perplexity along the IsoFLOP frontier decays as $f$ 7, with exponents derived from loop and token trade-off:

$f$ 8

Looping and data should be increased jointly under fixed FLOP (Prairie et al., 14 Apr 2026).

Impact of Context Length

In quadratic models (Transformer), IsoFLOP-optimal $f$ 9 decreases steeply as context length $N$ 0 increases ( $N$ 1), whereas for linear models (xLSTM), scaling agnostic to $N$ 2 ( $N$ 3) (Beck et al., 2 Oct 2025).

Systematic Biases in Estimation

Parabolic fitting (Chinchilla Approach 2) introduces systematic bias in the intercept and, when off-centering or grid drift occur, even in structural exponents, causing multi-million-dollar inefficiencies at scale (Czech et al., 21 Mar 2026).
Direct surface fitting (Approach 3 with Variable Projection NNLS) remedies these biases with minimal overhead and higher statistical efficiency.

4. Relative and Fairness-Aware IsoFLOP Scaling

IsoFLOP scaling laws are directly extended to relative scaling laws, which quantify how performance disparities between subpopulations or tasks evolve with compute scale:

Ratio form: $N$ 4
Difference form: $N$ 5

Empirical analysis reveals:

In academic knowledge domains, gaps between STEM and other fields shrink ( $N$ 6), with exponents indicating convergence toward parity (Held et al., 28 Oct 2025).
For regional English dialects, gaps track speaker population and can widen or close with scale.
For clusters of AI risk behaviors, certain risks remain flat or decline, while others increase at scale.

This explicitly demonstrates that simply increasing compute is not a universal equalizer—detailed subgroup analysis via IsoFLOP-relative methods is essential for robust fairness and safety forecasts.

5. Limitations, Extrapolation, and Data Efficiency

Breakdown at Data/Compute Limits: Extrapolation of IsoFLOP laws is robust only within the compute/data regime explored. For fixed-size datasets (e.g., ERA5 in weather emulation), increasing $N$ 7 beyond the data saturation point leads to overfitting and departure from power-law improvement (Subramanian et al., 26 Mar 2026).
Parabola Fitting Biases: Approach 2 is susceptible to (i) window width error, (ii) off-center bias, and (iii) surface asymmetry, markedly affecting optimal allocation recommendations (Czech et al., 21 Mar 2026).
Remedies: Variable Projection Nonnegative Least Squares (VPNLS) offers an analytical, low-dimensional optimization that is stable, scalable, and robust to initialization, with no statistical penalty compared to higher-dimensional direct surface fitting (Czech et al., 21 Mar 2026).

Practical fitting methodology recommendations:

Practice	Recommendation
Fitting Method	Use VPNLS or direct Chinchilla surface fitting
Window Selection (parabola)	Minimize window width, center at near-optimum
Data Sufficiency	Sample densely near loss minimum
Fairness/Robustness diagnostics	Fit relative scaling exponents $N$ 8

6. Practical Implications and Allocation Guidance

IsoFLOP scaling laws provide direct, actionable guidance for allocation of computational and data resources:

For a new compute budget $N$ 9, allocate $D$ 0, $D$ 1 per the estimated exponents for the chosen architecture and data regime (Held et al., 28 Oct 2025, Subramanian et al., 26 Mar 2026, Beck et al., 2 Oct 2025).
For looped models, increase both loop depth and token count proportionally to the observed exponents (Prairie et al., 14 Apr 2026).
Practitioners should monitor for departures from empirical power-laws (e.g., saturation), which signal data limitation or under-optimized architecture.
Relative scaling exponents determine whether additional compute reduces or amplifies disparities, and thus whether “just scaling up” is sufficient for fairness or subgroup robustness (Held et al., 28 Oct 2025).

A cross-paper summary of optimal allocation rules:

Domain/Architecture	$D$ 2 Exponent	$D$ 3 Exponent	Notes
LLM Transformer	0.59 (Held et al., 28 Oct 2025)	0.41 (Held et al., 28 Oct 2025)	Qwen 3 recipe
xLSTM	0.61 (Beck et al., 2 Oct 2025)	0.38 (Beck et al., 2 Oct 2025)	Linear context scaling
Parcae (looped)	0.40 (loops)	0.78 (Prairie et al., 14 Apr 2026)	Loops/data must both increase
Weather Emulation	0.59 (Subramanian et al., 26 Mar 2026)	0.41 (Subramanian et al., 26 Mar 2026)	Data saturation limits at scale

7. Broader Impact and Future Directions

IsoFLOP scaling laws have established themselves as a foundational diagnostic for efficient resource allocation, robust estimation of compute-optimal model settings, and principled forecasting of model performance and subgroup fairness trajectories across deep learning, scientific ML, and foundation model development.

As model scales and training budgets reach regimes where data limitation, overfitting, and architecture-specific nonlinearities emerge, the scope and generality of IsoFLOP scaling laws are expected to further expand to:

Multimodal models and specialized architectures (requiring more complex FLOP accounting and loss surface modeling) (Czech et al., 21 Mar 2026).
Data-limited and generalization-limited settings, where strict power-law scaling may break down (Subramanian et al., 26 Mar 2026).
Cross-task and cross-subgroup analyses to inform both global and targeted fairness/robustness interventions (Held et al., 28 Oct 2025).

A plausible implication is that future standardization of IsoFLOP-based analysis, combined with advanced surface fitting techniques, will be essential for both maximally efficient use of compute and for responsible, robust deployment of state-of-the-art machine learning systems.