---
title: Compute-Optimal Scaling Laws
url: https://www.emergentmind.com/topics/compute-optimal-scaling-laws
type: topic
---

# Compute-Optimal Scaling Laws

Compute-optimal scaling laws quantitatively describe how to allocate a fixed compute budget among neural network model size, training steps (or data), and other architectural and optimization degrees of freedom to minimize generalization error or, equivalently, achieve maximal model performance. These laws constrain the feasible trade-offs on the Pareto-efficient “frontier” of training strategies, providing explicit allocation exponents and prescriptions—usually in closed-form—that specify the optimal division of compute as models and datasets are scaled. Theoretical derivations and empirical validation in recent literature have established a core universal structure for these scaling laws, elucidated their task-/data-dependence, and identified important deviations in pretraining, fine-tuning, and adaptive or multi-task settings.

## 1. Theoretical Foundations: Power-Law Parametrization and Pareto Optimality

At the core of compute-optimal scaling laws is the empirical observation that generalization error $E$ or held-out loss $L$ can be decomposed into leading bottleneck terms, each decaying as a power-law in a limiting resource (model width/parameter count $N$, data size $D$, or training time $T$):

\[
E(N,T) \simeq A\,N^{-\alpha} + B\,T^{-\beta}
\]
or, more generally,
\[
L(N, D) = L_0 + A_N\,N^{-\alpha} + A_D\,D^{-\beta}
\]
where $\alpha$ is the parameter-limited exponent and $\beta$ the data- or time-limited exponent [2402.01092, 2502.12051, 2503.10061]. Setting a fixed compute budget, often $C = N \times T$ or $C = N \times D$, the problem reduces to a constrained minimization yielding allocation exponents for optimal $N^*$ and $T^*$ (or $D^*$):

\[
N^*(C) \propto C^{\gamma_N}, \quad T^*(C) \propto C^{\gamma_T}, \quad \gamma_N = \frac{\beta}{\alpha+\beta}, \quad \gamma_T = \frac{\alpha}{\alpha+\beta}
\]

This yields a compute-optimal error scaling of the form

\[
E^*(C) \propto C^{-\gamma}, \qquad \gamma = \frac{\alpha \beta}{\alpha+\beta}
\]

Analogous derivations hold for continuous architectural parameters and generalization to multiple task inputs [2402.01092, 2502.12051].

## 2. Universal Formulas, Exponents, and Their Task Dependence

Across modern deep learning, this framework is instantiated with problem-specific exponents and fit constants. For language modeling (“Chinchilla”-style), typical empirical values—cross-entropy loss in terms of model parameters $N$ and training tokens $D$—are [2502.12051]:

\[
L(N, D) = 1.0 + 0.44\,N^{-0.34} + 0.39\,D^{-0.28}
\]
The compute-optimal model and data sizes are

\[
N^* \propto C^{0.45}, \quad D^* \propto C^{0.55}
\]
with exponents set by the scaling laws. In some domains (e.g. reinforcement learning, vision, symbolic regression), exponents differ but the form is invariant [2210.00849, 2305.13035, 2510.26064]. Tabulated exponents across modalities:

| Domain            | $\alpha$ (params) | $\beta$ (data) | $N^* \propto C^{\gamma_N}$ | $D^* \propto C^{\gamma_T}$ |
|-------------------|------------------|----------------|-----------------------------|----------------------------|
| LLMs (Chinchilla) | 0.34             | 0.28           | 0.45                        | 0.55                       |
| Vision (ViT)      | ~0.20–0.45       | ~0.22–0.60     | varies per dimension         | varies per dimension        |
| RL (AlphaZero)    | 0.88             | 0.55           | 0.62                        | 0.38 (implied)             |
| SR (Symbolic Reg.)| —                | —              | ~0.20 (loss)                | ~0.36 (solved rate)        |

These exponents determine whether the optimum is “capacity-hungry” (large $\alpha$, more parameters) or “data-hungry” (large $\beta$, more data/steps). Task-specific skill groupings (as in code-generation vs. knowledge QA) yield systematically shifted frontiers [2503.10061].

## 3. Data, Architecture, and Skill-Dependent Deviations

Compute-optimal frontiers are sensitive to task, data distribution, and model architecture:

- **Data complexity**: As training data becomes harder to compress (higher gzip-entropy $H$), both $\alpha, \beta$ decrease, but $\beta$ (data exponent) falls faster, shifting the frontier toward favoring more data rather than parameters [2405.16684].
- **Skill specificity**: Code-generation tasks display larger $\beta$ values than text QA, requiring more data at fixed compute, whereas knowledge QA is more parameter-limited [2503.10061].
- **Mixture-of-experts, retrieval-augmentation, and multitask models**: These architectures may not obey the classical exponents, displaying sublinear expert-count returns or shifting optimal token/parameter ratios [2502.12051].
- **Validation set composition**: Compute-optimal recommendations can shift by up to 50% depending on whether the validation set reflects the mix of downstream skills [2503.10061].

These findings indicate that any “universal” scaling law must be further conditioned on dataset complexity, skill composition, and architecture class [2502.12051, 2405.16684].

## 4. Adaptive and Dynamic Compute-Optimal Schedules

Moving beyond static allocation, compute-optimal laws have been generalized to adaptive schedules where model “shape” parameters (e.g. width, depth, patch size, context length) are increased during training [2311.03233, 2305.13035]:

- At each error threshold, select the architecture yielding the greatest marginal reduction in loss per additional compute, following the *steepest local descent* among all scaling laws.
- By piecewise following the lower envelope of per-shape scaling curves, the composite adaptive path achieves strictly lower loss or target error at a given compute—empirically saving 25–70% of training FLOP in practical scenarios.
- This principle is validated for Transformer width/depth, patch size, and context length.

Shape-adaptive compute-optimality extends to multimodal and multi-domain mixture ratios, where the optimal domain weights (mixture fractions) are solved via the scaling law parameterized loss surface [2507.09404].

## 5. Optimization Procedures and Practical Model Design

The optimization of compute-optimal allocations is algebraically tractable. Consider minimizing

\[
L(N, D) = L_0 + A_N N^{-\alpha} + A_D D^{-\beta}, \quad \text{s.t.} \quad N D = C
\]

This yields

\[
N^* = \left( \frac{\alpha A_N}{\beta A_D} \right)^{1/(\alpha + \beta)} C^{\gamma_N}, \quad D^* = \frac{C}{N^*}, \quad \gamma_N = \frac{\beta}{\alpha + \beta}
\]
with closed-form error at optimum. In the presence of additional constraints (such as time-to-train, batch size limits, or data caps), frontiers are further modified by the empirically fit scaling laws for optimal hyperparameters (batch size, learning rate, weight decay), which also scale as power laws in $N$ and $D$ [2505.13738, 2406.19146]. For high-variance, high-latency, or energy-constrained scenarios, additional optimization over inference compute versus training compute is required [2506.08228].

Key recommendations:

- For language tasks, when $\alpha \simeq \beta$ (as in Chinchilla), set $N \sim D \sim C^{1/2}$; for code tasks, allocate more compute to data.
- In the presence of data scarcity, training for more epochs on existing data (up to $\sim$4) is nearly as beneficial as collecting new data [2502.12051].
- For adaptive architecture schedules, initial pilot runs are essential to robustly fit shape-dependent exponents [2311.03233].
- Fine-tuning and transfer tasks demand modified “rectified” or multi-phase scaling laws, with pre-power (inefficient) and post-power (efficient) regimes [2502.12051].

## 6. Extensions, Limitations, and Phase Behavior

Compute-optimal scaling laws admit generalizations to multidimensional and multiphase regimes:

- The “4+3 phases” in random-feature models distinguish boundaries where scaling transitions from capacity-limited to feature-limited to optimizer-noise-limited, with each phase having distinct compute-optimal exponents [2405.15074].
- In fully information-theoretic settings, optimal allocations depend on data latent complexity and input dimension, with higher complexity favoring greater parameter allocation [2212.01365].
- Deviations from classical behavior arise in very high-complexity tasks, heterogeneous data distributions, and under heavy architectural bottlenecks, necessitating empirical re-fitting or theoretical extension.
- Diminishing returns are quantifiable; past a data/model-size knee, further compute yields negligible improvement [2601.02706].

The compute-optimal framework also forms the analytical foundation for optimal planning in emerging domains such as motion planning, power systems, symbolic regression, and generative reasoning evaluations, with appropriate adjustment of exponents based on observed scaling fits [2506.08228, 2601.02706, 2510.26064, 2509.24012].

## 7. Empirical Procedures and Application Guidelines

Practical implementation of compute-optimal scaling laws involves the following standard steps:

1. Select relevant task, skill, and data regime; fit the two-dimensional or multidimensional power-law scaling law to a sweep of small models/training budgets.
2. Solve the analytic constrained minimization to determine the scaling exponents and closed-form allocations for model and data size.
3. For data-dependent regimes, estimate dataset complexity (e.g., via gzip-compressibility) and adjust the scaling law coefficients and exponents accordingly [2405.16684].
4. For adaptive or multi-domain tasks, fit shape-/mixture-aware scaling laws and optimize either sequentially (per dimension) or jointly.
5. Monitor for phase transitions or saturation points indicating deviation from power-law scaling; validate frontier predictions via pilot full-scale runs [2311.03233, 2305.13035, 2502.12051].
6. Use scaling-law-derived hyperparameter recipes for batch size, learning rate, and weight decay, which themselves scale as power laws in $N$ and $D$ [2505.13738, 2406.19146, 2510.26064].
7. Maintain alignment between the desired downstream skill mix and the composition of validation sets and data—misalignment can shift optimally allocated model size by tens of percent [2503.10061].

A summary table of canonical compute-optimal allocations in language modeling and vision:

| Scaling Law Type         | Model Size ($N^*$)         | Data Size ($D^*$)               | Source              |
|-------------------------|----------------------------|----------------------------------|---------------------|
| Chinchilla (LLM)        | $N^* \sim C^{0.5}$         | $D^* \sim C^{0.5}$               | 2502.12051          |
| Vision Transformer      | $d \sim C^{0.22}$ (width)  | $L \sim C^{0.45}$ (depth), $m\sim C^{0.60}$ (MLP) | 2305.13035          |
| RL (AlphaZero)          | $N^* \sim C^{0.62}$        | —                                | 2210.00849          |
| Symbolic Regression     | $L \sim C^{-0.20}$ (loss)  | $S \sim C^{0.36}$ (solved)       | 2510.26064          |

---
Compute-optimal scaling laws thus encode the quantitative structure underlying all modern large-scale neural network training design, distilling empirical and theoretical understanding into closed-form, universally applicable allocation rules, while accommodating critical deviations driven by task complexity, data heterogeneity, skill mixtures, and architecture. For further implementation details across specific domains and optimization regimes see [2402.01092], [2502.12051], [2406.19146], [2405.16684], [2311.03233], [2305.13035], [2212.01365], [2503.10061].

Source: https://www.emergentmind.com/topics/compute-optimal-scaling-laws