---
title: Compute-Optimal Scaling Law
url: https://www.emergentmind.com/topics/compute-optimal-scaling-law
type: topic
---

# Compute-Optimal Scaling Law

A compute-optimal scaling law describes how to allocate a fixed computational budget between neural network model size and training dataset size to achieve minimal loss in large-scale deep learning. This principle enables the design of models and training regimens that maximize the return on computational investment for a given domain and task. Compute-optimal scaling laws are formulated and empirically tested across modalities including language modeling, vision, reinforcement learning, motion forecasting, and symbolic regression, with distinct but related exponents encoding the efficiency of parameter and data scaling under a fixed compute constraint.

## 1. Mathematical Formulation

Compute-optimal scaling laws are derived from empirical power-law relationships between model performance and the primary scaling axes: parameter count ($N$), dataset size ($D$), and total training compute ($C$). The canonical functional form for loss is:
\[
L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}
\]
where $L(N, D)$ is the loss (e.g., cross-entropy), $E$ is the irreducible minimum, $A, B$ are prefactors, and $\alpha, \beta > 0$ encode how loss improves with $N$ and $D$ respectively.

Under a compute constraint ($C$ measured in FLOPs), typically approximated as $C \propto N D$, the optimization problem becomes finding $(N_{\mathrm{opt}}(C), D_{\mathrm{opt}}(C))$ that minimizes $L(N, D)$ subject to $N D = C$. Analytical minimization yields characteristic exponents:
\[
N_{\mathrm{opt}}(C) \propto C^{p}, \qquad D_{\mathrm{opt}}(C) \propto C^{d}
\]
with
\[
p = \frac{\beta}{\alpha + \beta}, \qquad d = \frac{\alpha}{\alpha + \beta}
\]
and the data-to-model size ratio governed by $\frac{D_{\mathrm{opt}}}{N_{\mathrm{opt}}} \propto \frac{\beta}{\alpha}$. For symmetric exponents ($\alpha \approx \beta$), this ratio is approximately 1, but it can deviate substantially between domains and even between subskills within the same modality [2506.08228][2502.12051][2406.19146][2503.10061].

## 2. Empirical Determination of Exponents

The exponents $\alpha, \beta$ and thereby $p, d$ are empirically estimated via large-scale sweeps over $N$, $D$, and $C$. For example, in autonomous driving agents performing motion forecasting and planning, a detailed fit yields:
- $p = 0.63 \pm 0.08$, $d = 0.44 \pm 0.06$ ($N_{\mathrm{opt}} \propto C^{0.63}$, $D_{\mathrm{opt}} \propto C^{0.44}$)
- $N_{\mathrm{opt}} \propto D_{\mathrm{opt}}^{1.5}$

For general language modeling, the latest robust multi-dataset, multi-hyperparameter synthesis gives $N_{\mathrm{opt}} \approx 0.09 C^{0.497}$ and a nearly fixed $D_{\mathrm{opt}}/N_{\mathrm{opt}} \approx 15$ ($D_{\mathrm{opt}} \propto C^{0.503}$) when all sources of discrepancy, such as omitted decoder head FLOPs, warmup duration, and scale-dependent optimizer tuning, are properly corrected [2406.19146]. In symbolic regression, the optimal exponents are $N_{\mathrm{opt}} \propto C^{0.40}$, $D_{\mathrm{opt}} \propto C^{0.43}$, with a token-to-parameter ratio $r \approx 15 C^{0.03}$ exhibiting only mild drift over a broad compute range [2510.26064].

The following table summarizes representative exponents across domains:

| Domain                  | $p$ (N exponent) | $d$ (D exponent) | Source             |
|-------------------------|------------------|------------------|--------------------|
| Language Modeling       | $0.50$           | $0.50$           | [2406.19146]       |
| Motion Forecasting      | $0.63$           | $0.44$           | [2506.08228]       |
| Symbolic Regression     | $0.40$           | $0.43$           | [2510.26064]       |
| Single-Agent RL         | $0.4$–$0.8$      | $1-\text{(N exponent)}$ | [2301.13442] |
| AlphaZero RL            | $0.62$           | $-$              | [2210.00849]       |
| ViT Image Classification| $0.22$ (width), $0.45$ (depth) | - | [2305.13035] |

## 3. Generalization, Quantization, and Performance Bounds

Compute-optimal scaling also governs the generalization gap and the quantizability of models. Under Chinchilla-level scaling ($N^* \sim D^* \sim C^{1/2}$), the token-wise generalization gap—the excess population loss over the empirical loss—provably shrinks as
\[
\Delta_{\mathrm{gen}}(C) = O(C^{-1/4})
\]
because both the loss variance and quantization gap terms decay with increased parameterization and training set size. Specifically, the Freedman-type martingale bound establishes that larger models, when scaled according to compute-optimal allocation, generalize better and are more amenable to quantization [2504.15208].

## 4. Inference-Time and Task-Specific Trade-Offs

Compute-optimal scaling laws are principally derived for training time; however, inference-time compute efficiency introduces a further axis of optimization. With a fixed inference budget, performance can be traded between model size and the number of samples (e.g., trajectory rollouts in planning). There exists a crossover point such that for constrained FLOPs, exhaustive sampling from a smaller model can outperform a larger model, but beyond this point, increasing model size is more efficient [2506.08228]. Furthermore, different tasks or skill domains may have distinct scaling exponents and optimal trade-offs. For instance, knowledge QA tasks exhibit larger compute-optimal model sizes (higher parameter hunger), while code generation tasks are more data-efficient and benefit from smaller models trained on more tokens. This skill dependence implies that validation set composition directly shifts compute-optimal allocations by as much as 50% for the same compute budget [2503.10061].

## 5. Theoretical Underpinnings and Information-Theoretic Foundations

The structure and exponents of compute-optimal scaling laws are now being linked to underlying information-theoretic and statistical mechanics models. For example, the optimal $N^* \sim D^* \sim C^{1/2}$ "Chinchilla rule" arises in both:
- A graph-based analogy to decoding in LDPC codes, where the learning of concepts from text is mapped to iterative belief propagation, and finite-size scaling arguments dictate the $1/2$ exponent [2410.01243].
- Information-theoretic bounds for shallower networks, where compute-optimal error decreases as $E^*(C) = \Theta(C^{-1/2})$, with optimal allocation maintaining $N^*/n^* = \Theta(1)$ [2212.01365].

These theoretical perspectives align with, and in some cases predict, the empirical exponents observed across domains.

## 6. Domain-Specific and Architectural Extensions

Although the core compute-optimal scaling law holds across a range of tasks, significant deviations arise in certain architectures, modalities, and operational regimes:
- Mixture-of-Experts and sparse models display bifurcated or modified law structure, demanding new optimization axes (e.g., number of experts, routing decisions).
- In vision and multimodal settings, exponents are sensitive to architectural parameters like patch size, depth, and hidden dimension, necessitating joint shape-compute optimization [2305.13035].
- In symbolic regression and motion generation, steeper exponents ($\beta \approx 0.21$ for loss vs. compute) indicate that compute is more rapidly converted into performance improvement compared to language modeling [2510.26064][2412.14559].
- In reinforcement learning, environment horizon and domain structure alter the scaling, with gamma exponents ($\gamma$) spanning a broader range depending on task horizon and intrinsic challenge [2301.13442][2210.00849].

## 7. Practical Guidelines and Limitations

Applying compute-optimal scaling laws in practice involves:
1. Empirically fitting small-scale experiments to extract exponents for a given model, data, and objective.
2. Allocating a fixed compute budget between $N$ and $D$ according to the inferred $p$ and $d$ exponents.
3. Validating predicted loss curves on held-out data to ensure power-law adherence.
4. Accounting for operational constraints (memory, inference cost, data availability) and adapting the allocation accordingly.

Open challenges include modeling inference-aware scaling, data curation effects, adaptation to hybrid architectures, hardware-specific scaling dynamics, and skill-mixed (multi-objective) training [2502.12051]. The laws also assume idealized regimes (large-scale, power-law loss falloff, unlimited data), so extrapolation beyond validated regions or to unreliable domains remains a point of active research.

---

**References:**
- "Scaling Laws of Motion Forecasting and Planning -- A Technical Report" [2506.08228]
- "Compute-Optimal LLMs Provably Generalize Better With Scale" [2504.15208]
- "Towards Scaling Laws for Symbolic Regression" [2510.26064]
- "Resolving Discrepancies in Compute-Optimal Scaling of Language Models" [2406.19146]
- "How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines" [2502.12051]
- "Compute Optimal Scaling of Skills: Knowledge vs Reasoning" [2503.10061]
- "A Dynamical Model of Neural Scaling Laws" [2402.01092]
- "4+3 Phases of Compute-Optimal Neural Scaling Laws" [2405.15074]
- "An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models" [2410.01243]
- "An Information-Theoretic Analysis of Compute-Optimal Neural Scaling Laws" [2212.01365]
- "Scaling laws for single-agent reinforcement learning" [2301.13442]
- "Scaling Laws for a Multi-Agent Reinforcement Learning Model" [2210.00849]
- "Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design" [2305.13035]
- "ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model" [2412.14559]

Source: https://www.emergentmind.com/topics/compute-optimal-scaling-law