---
title: Data Scaling Laws in ML
url: https://www.emergentmind.com/topics/data-scaling-laws
type: topic
---

# Data Scaling Laws in ML

Data scaling laws in machine learning describe systematic, often power-law, relationships between task performance (e.g., test loss) and the quantity of available data, model capacity, or compute budget. These principles, originally observed empirically in large-scale autoregressive generative modeling, have now been theoretically and empirically grounded across modalities such as language, images, audio, multimodal tasks, and mathematical problem solving. Scaling laws encode how performance improves as a resource is increased, the form of irreducible error, and the diminishing returns of data and capacity, and are deeply connected to the statistical structure and intrinsic redundancy of natural data distributions.

## 1. Universal Power-Law Plus Constant Scaling

Across generative image modeling, language, video, multimodal models, and problem solving, observed performance curves are well described by a “power-law plus constant” of the form
\[
L(x) = L_{\infty} + (x_0 / x)^{\alpha_x}
\]
where $L(x)$ is the cross-entropy loss on held-out data, $x$ is a scaling variable (such as dataset size, model size, or compute), $L_{\infty}$ is the irreducible (irreducible) loss—interpreted as the entropy of the true data distribution, and $(x_0/x)^{\alpha_x}$ is the reducible loss (empirically aligning with the KL divergence between the true and model distributions) [2010.14701, 2106.09488].

This scaling describes a regime in which performance smoothly, predictably improves with scale, regardless of bottleneck (model, data, or compute). Even after the reducible loss is a small fraction of $L_{\infty}$, the power-law persists.

**Table: Example Scaling Law Parameters (modality-specific, from [2010.14701])**

| Modality       | Exponent $\alpha_x$ | Irreducible $L_{\infty}$ (nats/image) |
|----------------|--------------------|---------------------------------------|
| Image (8×8)    | ~0.30              | ≈ 2.1                                 |
| Image (32×32)  | ~0.18              | ≈ 2.8                                 |
| Video (lowest) | ~0.21              | Varies                                |

The scaling exponents and constants are domain- and task-dependent but remain stable across scale. This form is robustly validated, including in audio [2106.09488], jet physics [2312.02264], and transfer learning [2504.13219].

## 2. Compute-Optimal Model and Data Allocation

A central consequence is compute–optimality: for a fixed computational budget $C$, the optimal model size increases as a power of compute,
\[
N_{\textrm{opt}} \propto C^{\beta}
\]
with $\beta$ empirically close to $0.7$ in diverse modalities [2010.14701]. This results in sublinear increases in necessary data as models scale (with dataset size $D \propto N^{0.4}$ for $\beta \sim 0.7$), indicating that data needs grow much more slowly than parameter count to maintain scaling efficiency.

Similar relationships appear in acoustic models [2106.09488], where doubling model size only requires a 1.77× data increase, and in theoretical models predicting linear or equiparameterized scaling between number of effective parameters and dataset size as optimal for minimizing error [2407.01456, 2210.16859].

**Scaling Law for Joint Data and Model Limitations:**
\[
L(N, D) = L_\infty + \left[\left(\frac{N_C}{N}\right)^{\alpha_N / \alpha} + \left(\frac{D_C}{D}\right)^{\alpha_D / \alpha}\right]^\alpha
\]
with all irreducible (floor) contributions factored explicitly [2106.09488].

## 3. Information-Theoretic Interpretation and Statistical Structure

The cross-entropy loss is rigorously decomposed as
\[
L = S(\textrm{True}) + D_\textrm{KL}(\textrm{True} \,\Vert\, \textrm{Model})
\]
where $S(\textrm{True})$ is the entropy of the data distribution, and $D_\textrm{KL}$ is the KL divergence between the true and learned distributions [2010.14701, 2203.15556].

The reducible loss strictly tracks $D_\textrm{KL}$, and as model/data increase, the model distribution approaches the true distribution monotonically.

Statistical analysis of natural datasets reveals covariance matrices with power-law spectral decay (eigenvalues $\lambda_i \sim i^{-(1+\alpha)}$), ensuring no sharp cutoff between informative and noise directions [2210.16859, 2509.20721]. Nonlinear feature maps in neural networks extend this regime, so test loss scaling inherits exponents set by this spectral structure, with ultimate scaling determined by the data manifold’s intrinsic dimensionality [2411.06646, 2412.07942].

## 4. Domain-Specific and Data-Dependent Extensions

**Multilingual Scaling:** In multilingual LMs, scaling laws generalize by modeling the test loss for each language family as a function of its sampling ratio in the mixture, model size, and data size:
\[
L_i(N, D, p_i) = \Bigl(E_i + \frac{A_i}{N^{\alpha_i} + B_i/D^{\beta_i}} \Bigr) p_i^{-\gamma_i}
\]
where $p_i$ is the sampling weight for family $i$ [2410.12883]. This decouples analysis from individual languages and allows optimal mixture selection to minimize aggregate loss.

**Data Complexity:** Scaling law constants and exponents depend systematically on quantifiable data complexity measures such as gzip-compressibility [2405.16684]. For less compressible data, the compute–optimal frontier tilts toward prioritizing increased dataset size rather than model size. All constants $(E, A, B, \alpha, \beta)$ in the classic law are modulated by this data-dependent factor:
\[
L(N, D, H) = E(H) + \frac{A(H)}{N^{\alpha(H)}} + \frac{B(H)}{D^{\beta(H)}}
\]
where $H$ is the gzip-normalized compressibility, linking scaling behavior directly to intrinsic data structure.

**Data Mixtures:** For large-scale pretraining, the optimal domain mixture $h$ can be determined via scaling law formulations (either additive or joint) that predict the loss on any target domain or set of domains including the effect of data mixture, model size, and training tokens:
\[
\mathcal{L}(N, D, h) = E + 1 / \left(\sum_{i=1}^k C_i h_i^{\gamma_i} + \frac{A}{N^\alpha} + \frac{B}{D^\beta}\right)
\]
with $h$ a simplex vector over $k$ source domains. This enables principled, compute-budget-aware mixture design [2507.09404].

## 5. Redundancy Laws and Spectral Foundations

The scaling exponent $\alpha$ arises from the spectral-tail of the data covariance: if the spectrum is polynomial, $\lambda_i \asymp i^{-1/\beta}$, the *redundancy index* $\rho_{\mathrm{red}} = 1/\beta$ dictates learning curve slope [2509.20721]. In kernel regression,
\[
\alpha = \frac{2s}{2s + 1/\beta}
\]
where $s$ is the function’s smoothness. Lower redundancy ($\beta \uparrow$) steepens scaling, accelerating returns to data/model scale, a phenomenon robust to data representation, mixtures, finite-width approximations, and even deep architectures (including Transformers in both NTK and feature-learning regimes).

Empirically, reducing redundancy—by learning better representations or data preprocessing—can make the observed scaling exponent larger, improving data efficiency. This provides a rigorous, unifying explanation for the power-law behavior empirically observed in deep models.

## 6. Limitations, Transitions, and Practical Implications

**Criticality and Phase Transitions:** Finite latent dimension or data “task complexity” can cause scaling to plateau once the number of model parameters or data points approaches this intrinsic limit [2210.16859, 2412.07942]. At the percolation threshold of data connectivity, scaling exponents are determined by the power-law size distribution of functional “quanta”; above this threshold, a dominant data manifold controls the scaling exponent, consistent with results from manifold approximation theory [2412.07942].

**Irreducible Loss:** The irreducible loss ($L_\infty$) represents the entropy floor of the data distribution. As models/data increase, returns diminish when approaching this limit, and further resource investment becomes exponentially less effective [2106.09488, 2010.14701].

**Uncertainty Scaling:** Predictive epistemic uncertainties contract with increasing data size, typically as $O(1/N)$ in parametric models; however, even in over-parameterized neural nets, residual epistemic uncertainty remains non-negligible, following power-law decay but rarely vanishing at practical scales [2506.09648]. This underscores the continuing need for uncertainty quantification in large-scale models, even with massive datasets.

**Individual Data Contributions:** The value of individual data points for a specific model shrinks with increasing dataset size in a log-linear (power-law) manner, but with significant heterogeneity between examples. Some data points with slowly-decaying exponents remain highly valuable even in large-data regimes [2405.20456].

**Data Reuse:** In data-constrained settings, reusing samples (multi-pass SGD) improves the effective scaling law; the test error becomes dependent on the total number of effective iterations, amplifying gains over strict one-pass bounds, provided the number of passes does not exceed a certain threshold [2506.08415].

## 7. Domain-Specific Scaling and Methodological Recommendations

- **Classifier Performance:** In high-dimensional nearest-neighbor classification, scaling transitions from fast (polynomial) to slow (exponential) rates depending on favorable geometry (signal alignment) or distributional structure [2308.08247].
- **Transfer Learning:** Scaling laws in visual transfer learning display boundaries where knowledge distillation outperforms standard training only up to a critical data threshold, after which further pretraining data favors direct training [2504.13219].
- **Jet Physics and High-Energy Domains:** Classifier loss in jet physics follows empirical power-law scaling, but exponents differ by method; model selection for a fixed data regime can be misleading, as “fast-scaling” methods will surpass “high baseline, slow-scaling” ones with sufficient data [2312.02264].

**Practical Guidance:**
- Optimal resource allocation demands matching model and data scaling according to empirical or theoretical exponents, often favoring larger models with proportionally less training per parameter at scale [2010.14701, 2407.01456].
- Empirical estimation of constants and exponents should be performed with smaller models and data slices—these exponents generalize to larger settings when experimental setup (context length, tokenization, etc.) is matched [2403.06563].
- Data mixture optimization should be formulated within the scaling law framework; small-scale piloting can predict the optimal mixture for any target tasks and compute regime, avoiding expensive trial-and-error [2507.09404, 2410.12883].
- Data structure complexity (estimated, for example, by gzip-compressibility) should guide budget allocation; “hard” data may necessitate larger effective dataset sizes for the same improvement [2405.16684].

---

Data scaling laws thus provide not only phenomenological descriptions and predictive formulas for error curves, but also a theoretical, information-theoretic, and spectral unification of performance scaling in high-capacity models. They offer actionable guidelines for resource allocation, provide a lens for interpreting empirical performance in novel domains, and reveal fundamental limits set by data complexity, redundancy, and intrinsic task structure.

Source: https://www.emergentmind.com/topics/data-scaling-laws