---
title: Generalizable Scaling Laws in ML
url: https://www.emergentmind.com/topics/generalizable-scaling-laws
type: topic
---

# Generalizable Scaling Laws in ML

Searching arXiv for the cited works to ground the article in current literature.
Generalizable scaling laws are empirical or theoretical relations that retain predictive value when transported across scale regimes, domains, architectures, representations, or problem instances. In machine learning, the term is made explicit by work that asks how to fit a law once on a well-resourced source domain and reliably transport it to new domains where running a full sweep is infeasible, while related work studies invariance under bijective transformations, predictable modification under non-bijective transformations that lower information resolution $\rho$, universality across boundedly invertible transformations and mixtures, and multivariate forms that extrapolate as several axes vary simultaneously [2605.07546][2509.20721][2605.26248]. This suggests that generalizable scaling laws are best viewed not as a single parametric curve, but as transportable relations whose exponents and asymptotes are controlled by identifiable invariants such as intrinsic dimension, spectral decay, source smoothness, representation capacity, or information resolution.

## 1. Formal notion of transportable scaling

A canonical neural-scaling form writes the loss as
$$
L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,
$$
where $N$ is the number of trainable parameters, $D$ the number of training tokens or data points, $\alpha$ the model-size exponent, $\beta$ the data-size exponent, and $E$ the irreducible risk floor [2605.07546]. Generalizable scaling laws begin from this type of relation but ask which quantities remain invariant, and which change in a controlled way, when the data distribution or representation is altered.

A central result is that any bijective, information-preserving transformation of the input leaves $\alpha$, $\beta$, and $E$ unchanged exactly. For non-bijective transformations, the same work introduces the information resolution
$$
\rho(T):=\frac{I(T(X);Y)}{I(X;Y)},\qquad \rho\in(0,1],
$$
and proposes the transported law
$$
L(N,D,\rho)=\frac{A}{N^\alpha}+\left(\frac{B}{D^\beta}\right)\rho^{-\nu}+E+\kappa(1-\rho)^\mu.
$$
The first additional term is a variance-inflation factor, and the second is a Bayes-risk elevation term. Empirically, this framework is validated across language, vision, and speech, including cross-domain prediction for language models trained on electronic health records from laws fit on general text, and for time-series classification under varying levels of noise injection, recovering the data-scaling exponents to within $3\%$ error [2605.07546].

In this formulation, “generalizable” does not mean architecture-agnostic in an unrestricted sense. It means that the law is preserved under a specified class of transformations, or modified along a low-dimensional axis such as $\rho$. The practical consequence is that extrapolation becomes an invariance problem rather than only a regression problem.

## 2. Geometric and data-distribution origins

One route to generalizable scaling laws ties the exponents to the geometry of the data distribution. For transformers on intrinsically low-dimensional data, a rigorous theory considers regression of a target function $f:M\to\mathbb R$, where $M$ is a compact $d$-dimensional Riemannian manifold isometrically embedded in $\mathbb R^D$, and $f$ is $\beta$-Hölder continuous. The resulting approximation and estimation bounds are
$$
\inf_{T\in\mathcal T}\|T-f\|_{L^\infty(M)}^2\lesssim N^{-2\beta/d},
\qquad
\mathbb E_x\|\hat H_n-f\|_{L^2(Q)}^2\le C\cdot D\,d^2\,n^{-2\beta/(2\beta+d)},
$$
which combine into
$$
E(N,n)\le C_1N^{-2\beta/d}+C_2n^{-2\beta/(2\beta+d)}.
$$
In the same theory, the transformer depth need only satisfy $T=O(\log d)$, independent of desired accuracy $\epsilon$, a shallow-in-$d$ property that contrasts with feed-forward ReLU networks requiring $O(\log(\epsilon^{-1}))$ layers [2411.06646].

The empirical side of the same study estimates intrinsic dimension on natural-language datasets using final-layer token embeddings and finds $\hat d_{\text{OpenWebText}}\approx15.6$, $\hat d_{\text{SQL}}\approx17.8$, and $\hat d_{\text{TinyStories}}\approx11.9$. With $\beta=1$, the predicted data-scaling exponent is $\alpha_D=2/(2+\hat d)$, giving $0.114$, $0.101$, and $0.144$, while the observed exponents are $0.11$, $0.13$, and $0.14$, respectively; the agreement is reported as within $\pm0.02$ [2411.06646]. In this account, the transportable quantity is the intrinsic dimension $d$, not the ambient dimension $D$.

A different data-distribution account uses percolation theory. There, two regimes produce distinct neural-scaling exponents. In the “Quantum-dominated” regime, if $c/D\gg\alpha$, then model-size scaling obeys $\mathcal L(N)\sim N^{-\alpha}$ and data-size scaling obeys $\mathcal L(\mathcal D)\sim \mathcal D^{-\alpha/(1+\alpha)}$. In the “Manifold-dominated” regime, if $c/D\ll\alpha$, both model and data scaling obey $\mathcal L\sim N^{-c/D}$ and $\mathcal L\sim\mathcal D^{-c/D}$ [2412.07942]. This suggests that generalizable exponents may be selected by which structural regime of the data distribution dominates.

## 3. Spectral, redundancy, and feature-learning mechanisms

A second family of theories derives scaling exponents from operator spectra, source conditions, and learning dynamics rather than directly from manifold geometry.

| Framework | Controlling quantity | Canonical law |
|---|---|---|
| Transformer theory on manifolds [2411.06646] | intrinsic dimension $d$, Hölder regularity $\beta$ | $E(N,n)\lesssim N^{-2\beta/d}+n^{-2\beta/(2\beta+d)}$ |
| Redundancy laws [2509.20721] | source smoothness $s$, redundancy index $\rho_{\rm red}=1/\beta$ | $\alpha=\frac{2s}{2s+1/\beta}$ |
| QPLR SGD [2502.09106] | covariance decay $\alpha$, source decay $\beta$ | $T^{-(1-1/\beta)}$ or $T^{-(2\beta-2)/(\alpha+\beta)}$ |
| Random-feature/GP dynamics [2602.23039] | kernel spectrum $\lambda_k\sim Ck^{-\alpha}$ | $\varepsilon_{\rm gen}\sim t^{-1/\alpha}$ and $n^{-(\alpha-1)/\alpha}$ |
| Percolation regimes [2412.07942] | cluster exponent $\alpha$, manifold ratio $c/D$ | $N^{-\alpha}$ or $N^{-c/D}$ |

In the kernel-ridge “redundancy law,” the covariance operator eigenvalues satisfy $\lambda_i\asymp i^{-1/\beta}$ and the target obeys the source condition $f^*=\Sigma^s g$. The optimized excess risk then scales as
$$
\mathbb E\,\mathcal E(f_{\lambda^*,n})\asymp n^{-\alpha},
\qquad
\alpha=\frac{2s}{2s+1/\beta}=\frac{2s}{2s+\rho_{\rm red}}.
$$
The same paper proves invariance of the exponent under boundedly invertible transformations, shows that mixtures are dominated by the slowest-decaying tail $\beta=\min_k\beta_k$, and extends the same exponent to random features and transformer NTK settings, with feature-learning kernel drift yielding an interval
$$
\alpha_{\rm drift}\in\left[\frac{2s}{2s+1/\beta_{\max}},\frac{2s}{2s+1/\beta_{\min}}\right]
$$
[2509.20721]. In this formulation, generalizability is spectral-tail invariance.

Algorithmic effects also matter. In quadratically parameterized linear regression, stochastic gradient descent exhibits different asymptotic exponents depending on whether $\beta\ge\alpha$ or $\alpha>\beta$. For $\beta\ge\alpha$,
$$
\mathcal R_M(v^T)-\mathbb E[\xi^2]\lesssim T^{-(1-1/\beta)},
$$
whereas for $\alpha>\beta$,
$$
\mathcal R_M(v^T)-\mathbb E[\xi^2]\lesssim T^{-(2\beta-2)/(\alpha+\beta)}.
$$
The linear-parameterized comparator scales as $T^{-(\beta-1)/\alpha}$ in the $\alpha>\beta$ regime, so the quadratic parameterization yields a strictly larger exponent there [2502.09106]. A plausible implication is that a law can be transportable across scales while still being conditional on parameterization and optimizer-induced implicit regularization.

The dynamical mean-field theory of random-feature regression makes the time dependence explicit. For kernel spectra $\lambda_k\sim Ck^{-\alpha}$, the gradient-flow regime gives
$$
\varepsilon_{\rm gen}(t,n)\sim (nt)^{-1/\alpha},
$$
and the Bayesian equilibrium regime gives
$$
\varepsilon_{\rm gen}(\infty,n)\sim n^{-(\alpha-1)/\alpha}.
$$
Early stopping recovers the same sample exponent $(\alpha-1)/\alpha$ [2602.23039]. This is a dynamic version of a generalizable scaling law: the exponent is transported across training protocols through the shared spectrum.

## 4. Functional forms for multiregime and multiaxis extrapolation

Generalizable scaling laws require functional forms that remain accurate across breaks, saturations, and nonmonotonic regimes. The “Broken Neural Scaling Law” addresses the univariate case with
$$
y(x)=a+b\,x^{-c_0}\prod_{i=1}^{n}\left[1+\left(\frac{x}{d_i}\right)^{f_i}\right]^{-c_i/f_i}.
$$
Here $a$ is the asymptotic floor or ceiling, $c_0$ the initial exponent, $d_i$ the breakpoints, $c_i$ the changes in exponent, and $f_i$ the smoothness parameters. This form is expressly designed to model multiple monotonic power-law regimes, as well as nonmonotonic transitions such as double descent and delayed sharp inflection points. The same work also emphasizes a limit of predictability: one cannot forecast a very sharp break from data only at $x\ll d_{\rm break}$ [2210.14891].

The multivariate extension is the Unified Neural Scaling Law, which is designed to model simultaneous variation in number of model parameters, training dataset size, number of training steps, number of inference steps, amount of compute, and various hyperparameters. Its internal structure combines a multivariate broken neural scaling component $K$, bottleneck and non-bottleneck aggregators $R$, reciprocal “oppositional forces” $Q$, and a final expression for $y$. On held-out extrapolation, it outperforms competing forms such as CF, DC, A1, A2, and A3 across large-scale vision, language, math, and reinforcement-learning settings; across all vision tasks it is best on $60.9\%$ of domains, and across language tasks on $88.9\%$ [2605.26248]. In this line of work, generalizability is a property of the functional family itself.

A contrasting development is “Neural Neural Scaling Laws” (NeuNeu), which treats downstream scaling prediction as time-series extrapolation rather than as a fixed parametric family. It combines temporal context from observed accuracy trajectories with token-level validation losses and predicts future task accuracy by quantile regression. On 66 downstream tasks it achieves $2.04\%$ mean absolute error, compared with $3.29\%$ for logistic scaling laws, and it generalizes zero-shot to unseen model families, parameter counts, and downstream tasks [2601.19831]. This result directly challenges the assumption that a single low-parameter family can generalize across all downstream behaviors.

## 5. Optimization over mixtures, representations, and problem size

Generalizable scaling laws are increasingly used as optimization tools. For data mixtures, one line of work defines loss on a target domain as $\mathcal L(N,D,h)$ for model size $N$, token budget $D$, and domain-weight vector $h\in\Delta_k$, and proposes additive and joint mixture laws that extend the Chinchilla form by making the coefficients depend smoothly on $h$. Parameters are fit from a small set of small-scale runs over $(N^j,D^j,h^j)$, and the resulting law extrapolates to unseen mixtures and larger scales in three settings: large language model, native multimodal model, and large vision model pretraining. The optimal mixture
$$
h^*(N,D)\in \arg\min_{h\in\Delta_k}\mathcal L(N,D,h)
$$
is then obtained by mirror descent [2507.09404]. In this setting, generalizability refers to extrapolation across both scale and domain composition.

A related unification concerns compressed representations. There the proposed law is
$$
\mathrm{Loss}(N,D;R)\simeq A\,[N\,\rho(R)]^{-\alpha}+B\,D^{-\beta}+E,
$$
with effective parameter count $N_{\rm eff}(R)=N\rho(R)$ and representation capacity $\rho(R)\in(0,1]$ derived from the Gaussian MSE of the compression map. Independent compressions compose multiplicatively,
$$
\rho(R_1\circ R_2)=\rho(R_1)\rho(R_2),
$$
and the same law is reported to cover sparsity, scalar quantization, sparse-quantized, and vector-quantized formats [2506.01863]. The paper also proposes RMSE-Banded Backward Masking, which yields up to $20\%$ higher $\rho(s)$ than standard Magnitude Pruning at fixed sparsity.

Generalizability can also be defined across problem size and inference allocation. In AlphaZero experiments on Hex, the performance frontier across board side-lengths $s=3,\dots,9$ is fit by a three-piece change-point model with
$$
\mathrm{Elo}(s,C)=\max\{\mathrm{plateau}(s),\min\{\mathrm{incline}(s,C),0\}\},
$$
where
$$
\mathrm{plateau}(s)=-270\,s+570,\qquad
\mathrm{incline}(s,C)=-430\,s+510\log_{10}C-4400.
$$
Perfect play requires approximately $7\times$ more training compute per unit increase in $s$, and the train-time versus test-time trade-off at fixed Elo obeys
$$
C_{\rm train}^{\beta}\,N_{\rm test}\simeq \mathrm{constant},
\qquad \beta\simeq1.18.
$$
This extends the notion of a scaling law beyond model and dataset size to problem hardness and search allocation [2104.03113].

## 6. Empirical scope, limits, and recurring misconceptions

The empirical literature shows that scaling exponents are not universal constants. In supervised galaxy-image modeling, the loss scales with training dataset size as $\log L(N)=A-b\log N$, with posterior-median $b$ values between approximately $0.84$ and $0.95$ across ResNet, EfficientNet, EfficientNetV2, MaxViT, and ConvNeXt, while parameter scaling is effective only for some tasks and overfitting emerges strongly beyond approximately $100$M parameters [2404.02973]. In deep regression for twisted van der Waals magnets, the observed data-scaling exponents range from about $1$ to $2$, depend on the regressed parameter and architecture, and the model-scaling law is clear only for some targets; the authors further note that theoretical understanding of these deep-regression scaling laws remains undeveloped [2509.10000]. These cases show that transportable laws typically require an explicit conditioning variable—domain, target, representation, or task—rather than a single exponent reused wholesale.

Scaling alone is also not equivalent to transfer. In the galaxy-image study, supervised domain adaptation beyond ImageNet-12k pretraining yields an average relative error rate reduction of $31\%$ across five downstream tasks, and the authors conclude that scaling alone is not sufficient to address the domain gap [2404.02973]. The symbolic-regression study similarly shows that useful laws can exist outside conventional language-model settings, but with their own compute-optimal hyperparameters: validation loss scales as $L(C)\simeq1.1\times10^3\,C^{-0.20}$, solved rate as $S(C)\simeq4.3\times10^{-8}C^{0.37}$, and the optimal token-to-parameter ratio is approximately $15$ in the reported regime [2510.26064].

Several misconceptions recur in the literature. One is that aggregate validation loss is a sufficient statistic for downstream scaling. The NeuNeu results instead report that downstream tasks can improve monotonically, plateau, or even degrade with scale, and that no simple parametric family captures the full spectrum of behaviors [2601.19831]. A second is that sharper functional forms always improve forecastability; the broken-scaling analysis states the opposite for very delayed breaks, because points far below the break do not identify its location [2210.14891]. A third is that “generalizable” means purely machine-learned regularity. Outside machine learning, related work derives scaling laws from invariance under change of units or more general change-of-variables symmetries in biological fluid-elastic systems, and from crossover-defined objective scales in “radical scaling,” indicating that transportable scaling relations are part of a broader scientific program of invariance and regime matching [2502.11398][2507.02631].

Taken together, the current literature presents generalizable scaling laws as a layered concept. At the strongest level, a law is invariant under an explicitly defined transformation. At an intermediate level, it deforms predictably along a small set of latent axes such as intrinsic dimension, redundancy, or information resolution. At the weakest level, it is a flexible extrapolant that remains accurate across breaks and multiple control variables. The main open question is not whether scaling laws exist, but which invariants are sufficient to transport them reliably across the domains in which they are most needed.

Source: https://www.emergentmind.com/topics/generalizable-scaling-laws