Generalizable Scaling Laws in ML

• Generalizable scaling laws are relations that remain invariant under controlled transformations, predicting model performance across diverse scales, domains, and representations.
• They leverage invariants such as intrinsic dimension, spectral decay, and information resolution to adjust scaling predictions under varied data distributions and architectural changes.
• They empower optimization over mixtures, compressed representations, and problem complexity, enabling accurate loss projection and efficient resource allocation.

Searching arXiv for the cited works to ground the article in current literature. Generalizable scaling laws are empirical or theoretical relations that retain predictive value when transported across scale regimes, domains, architectures, representations, or problem instances. In machine learning, the term is made explicit by work that asks how to fit a law once on a well-resourced source domain and reliably transport it to new domains where running a full sweep is infeasible, while related work studies invariance under bijective transformations, predictable modification under non-bijective transformations that lower information resolution $\rho$, universality across boundedly invertible transformations and mixtures, and multivariate forms that extrapolate as several axes vary simultaneously (Han et al., 8 May 2026, Bi et al., 25 Sep 2025, Caballero et al., 25 May 2026). This suggests that generalizable scaling laws are best viewed not as a single parametric curve, but as transportable relations whose exponents and asymptotes are controlled by identifiable invariants such as intrinsic dimension, spectral decay, source smoothness, representation capacity, or information resolution.

1. Formal notion of transportable scaling

A canonical neural-scaling form writes the loss as

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$

where $N$ is the number of trainable parameters, $D$ the number of training tokens or data points, $\alpha$ the model-size exponent, $\beta$ the data-size exponent, and $E$ the irreducible risk floor (Han et al., 8 May 2026). Generalizable scaling laws begin from this type of relation but ask which quantities remain invariant, and which change in a controlled way, when the data distribution or representation is altered.

A central result is that any bijective, information-preserving transformation of the input leaves $\alpha$, $\beta$, and $E$ unchanged exactly. For non-bijective transformations, the same work introduces the information resolution

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$0

and proposes the transported law

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$1

The first additional term is a variance-inflation factor, and the second is a Bayes-risk elevation term. Empirically, this framework is validated across language, vision, and speech, including cross-domain prediction for LLMs trained on electronic health records from laws fit on general text, and for time-series classification under varying levels of noise injection, recovering the data-scaling exponents to within $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$2 error (Han et al., 8 May 2026).

In this formulation, “generalizable” does not mean architecture-agnostic in an unrestricted sense. It means that the law is preserved under a specified class of transformations, or modified along a low-dimensional axis such as $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$3. The practical consequence is that extrapolation becomes an invariance problem rather than only a regression problem.

2. Geometric and data-distribution origins

One route to generalizable scaling laws ties the exponents to the geometry of the data distribution. For transformers on intrinsically low-dimensional data, a rigorous theory considers regression of a target function $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$4, where $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$5 is a compact $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$6-dimensional Riemannian manifold isometrically embedded in $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$7, and $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$8 is $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$9-Hölder continuous. The resulting approximation and estimation bounds are

$N$0

which combine into

$N$1

In the same theory, the transformer depth need only satisfy $N$2, independent of desired accuracy $N$3, a shallow-in-$N$4 property that contrasts with feed-forward ReLU networks requiring $N$5 layers (Havrilla et al., 2024).

The empirical side of the same study estimates intrinsic dimension on natural-language datasets using final-layer token embeddings and finds $N$6, $N$7, and $N$8. With $N$9, the predicted data-scaling exponent is $D$0, giving $D$1, $D$2, and $D$3, while the observed exponents are $D$4, $D$5, and $D$6, respectively; the agreement is reported as within $D$7 (Havrilla et al., 2024). In this account, the transportable quantity is the intrinsic dimension $D$8, not the ambient dimension $D$9.

A different data-distribution account uses percolation theory. There, two regimes produce distinct neural-scaling exponents. In the “Quantum-dominated” regime, if $\alpha$0, then model-size scaling obeys $\alpha$1 and data-size scaling obeys $\alpha$2. In the “Manifold-dominated” regime, if $\alpha$3, both model and data scaling obey $\alpha$4 and $\alpha$5 (Brill, 2024). This suggests that generalizable exponents may be selected by which structural regime of the data distribution dominates.

3. Spectral, redundancy, and feature-learning mechanisms

A second family of theories derives scaling exponents from operator spectra, source conditions, and learning dynamics rather than directly from manifold geometry.

Framework	Controlling quantity	Canonical law
Transformer theory on manifolds (Havrilla et al., 2024)	intrinsic dimension $\alpha$6, Hölder regularity $\alpha$7	$\alpha$8
Redundancy laws (Bi et al., 25 Sep 2025)	source smoothness $\alpha$9, redundancy index $\beta$0	$\beta$1
QPLR SGD (Ding et al., 13 Feb 2025)	covariance decay $\beta$2, source decay $\beta$3	$\beta$4 or $\beta$5
Random-feature/GP dynamics (Kramp et al., 26 Feb 2026)	kernel spectrum $\beta$6	$\beta$7 and $\beta$8
Percolation regimes (Brill, 2024)	cluster exponent $\beta$9, manifold ratio $E$0	$E$1 or $E$2

In the kernel-ridge “redundancy law,” the covariance operator eigenvalues satisfy $E$3 and the target obeys the source condition $E$4. The optimized excess risk then scales as

$E$5

The same paper proves invariance of the exponent under boundedly invertible transformations, shows that mixtures are dominated by the slowest-decaying tail $E$6, and extends the same exponent to random features and transformer NTK settings, with feature-learning kernel drift yielding an interval

$E$7

(Bi et al., 25 Sep 2025). In this formulation, generalizability is spectral-tail invariance.

Algorithmic effects also matter. In quadratically parameterized linear regression, stochastic gradient descent exhibits different asymptotic exponents depending on whether $E$8 or $E$9. For $\alpha$0,

$\alpha$1

whereas for $\alpha$2,

$\alpha$3

The linear-parameterized comparator scales as $\alpha$4 in the $\alpha$5 regime, so the quadratic parameterization yields a strictly larger exponent there (Ding et al., 13 Feb 2025). A plausible implication is that a law can be transportable across scales while still being conditional on parameterization and optimizer-induced implicit regularization.

The dynamical mean-field theory of random-feature regression makes the time dependence explicit. For kernel spectra $\alpha$6, the gradient-flow regime gives

$\alpha$7

and the Bayesian equilibrium regime gives

$\alpha$8

Early stopping recovers the same sample exponent $\alpha$9 (Kramp et al., 26 Feb 2026). This is a dynamic version of a generalizable scaling law: the exponent is transported across training protocols through the shared spectrum.

4. Functional forms for multiregime and multiaxis extrapolation

Generalizable scaling laws require functional forms that remain accurate across breaks, saturations, and nonmonotonic regimes. The “Broken Neural Scaling Law” addresses the univariate case with

$\beta$0

Here $\beta$1 is the asymptotic floor or ceiling, $\beta$2 the initial exponent, $\beta$3 the breakpoints, $\beta$4 the changes in exponent, and $\beta$5 the smoothness parameters. This form is expressly designed to model multiple monotonic power-law regimes, as well as nonmonotonic transitions such as double descent and delayed sharp inflection points. The same work also emphasizes a limit of predictability: one cannot forecast a very sharp break from data only at $\beta$6 (Caballero et al., 2022).

The multivariate extension is the Unified Neural Scaling Law, which is designed to model simultaneous variation in number of model parameters, training dataset size, number of training steps, number of inference steps, amount of compute, and various hyperparameters. Its internal structure combines a multivariate broken neural scaling component $\beta$7, bottleneck and non-bottleneck aggregators $\beta$8, reciprocal “oppositional forces” $\beta$9, and a final expression for $E$0. On held-out extrapolation, it outperforms competing forms such as CF, DC, A1, A2, and A3 across large-scale vision, language, math, and reinforcement-learning settings; across all vision tasks it is best on $E$1 of domains, and across language tasks on $E$2 (Caballero et al., 25 May 2026). In this line of work, generalizability is a property of the functional family itself.

A contrasting development is “Neural Neural Scaling Laws” (NeuNeu), which treats downstream scaling prediction as time-series extrapolation rather than as a fixed parametric family. It combines temporal context from observed accuracy trajectories with token-level validation losses and predicts future task accuracy by quantile regression. On 66 downstream tasks it achieves $E$3 mean absolute error, compared with $E$4 for logistic scaling laws, and it generalizes zero-shot to unseen model families, parameter counts, and downstream tasks (Hu et al., 27 Jan 2026). This result directly challenges the assumption that a single low-parameter family can generalize across all downstream behaviors.

5. Optimization over mixtures, representations, and problem size

Generalizable scaling laws are increasingly used as optimization tools. For data mixtures, one line of work defines loss on a target domain as $E$5 for model size $E$6, token budget $E$7, and domain-weight vector $E$8, and proposes additive and joint mixture laws that extend the Chinchilla form by making the coefficients depend smoothly on $E$9. Parameters are fit from a small set of small-scale runs over $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$00, and the resulting law extrapolates to unseen mixtures and larger scales in three settings: LLM, native multimodal model, and large vision model pretraining. The optimal mixture

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$01

is then obtained by mirror descent (Shukor et al., 12 Jul 2025). In this setting, generalizability refers to extrapolation across both scale and domain composition.

A related unification concerns compressed representations. There the proposed law is

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$02

with effective parameter count $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$03 and representation capacity $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$04 derived from the Gaussian MSE of the compression map. Independent compressions compose multiplicatively,

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$05

and the same law is reported to cover sparsity, scalar quantization, sparse-quantized, and vector-quantized formats (Panferov et al., 2 Jun 2025). The paper also proposes RMSE-Banded Backward Masking, which yields up to $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$06 higher $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$07 than standard Magnitude Pruning at fixed sparsity.

Generalizability can also be defined across problem size and inference allocation. In AlphaZero experiments on Hex, the performance frontier across board side-lengths $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$08 is fit by a three-piece change-point model with

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$09

where

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$10

Perfect play requires approximately $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$11 more training compute per unit increase in $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$12, and the train-time versus test-time trade-off at fixed Elo obeys

$L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$13

This extends the notion of a scaling law beyond model and dataset size to problem hardness and search allocation (Jones, 2021).

6. Empirical scope, limits, and recurring misconceptions

The empirical literature shows that scaling exponents are not universal constants. In supervised galaxy-image modeling, the loss scales with training dataset size as $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$14, with posterior-median $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$15 values between approximately $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$16 and $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$17 across ResNet, EfficientNet, EfficientNetV2, MaxViT, and ConvNeXt, while parameter scaling is effective only for some tasks and overfitting emerges strongly beyond approximately $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$18M parameters (Walmsley et al., 2024). In deep regression for twisted van der Waals magnets, the observed data-scaling exponents range from about $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$19 to $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$20, depend on the regressed parameter and architecture, and the model-scaling law is clear only for some targets; the authors further note that theoretical understanding of these deep-regression scaling laws remains undeveloped (Cadez et al., 12 Sep 2025). These cases show that transportable laws typically require an explicit conditioning variable—domain, target, representation, or task—rather than a single exponent reused wholesale.

Scaling alone is also not equivalent to transfer. In the galaxy-image study, supervised domain adaptation beyond ImageNet-12k pretraining yields an average relative error rate reduction of $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$21 across five downstream tasks, and the authors conclude that scaling alone is not sufficient to address the domain gap (Walmsley et al., 2024). The symbolic-regression study similarly shows that useful laws can exist outside conventional language-model settings, but with their own compute-optimal hyperparameters: validation loss scales as $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$22, solved rate as $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$23, and the optimal token-to-parameter ratio is approximately $L(N,D)=\frac{A}{N^\alpha}+\frac{B}{D^\beta}+E,$24 in the reported regime (Otte et al., 30 Oct 2025).

Several misconceptions recur in the literature. One is that aggregate validation loss is a sufficient statistic for downstream scaling. The NeuNeu results instead report that downstream tasks can improve monotonically, plateau, or even degrade with scale, and that no simple parametric family captures the full spectrum of behaviors (Hu et al., 27 Jan 2026). A second is that sharper functional forms always improve forecastability; the broken-scaling analysis states the opposite for very delayed breaks, because points far below the break do not identify its location (Caballero et al., 2022). A third is that “generalizable” means purely machine-learned regularity. Outside machine learning, related work derives scaling laws from invariance under change of units or more general change-of-variables symmetries in biological fluid-elastic systems, and from crossover-defined objective scales in “radical scaling,” indicating that transportable scaling relations are part of a broader scientific program of invariance and regime matching (Liu et al., 17 Feb 2025, Fardin et al., 3 Jul 2025).

Taken together, the current literature presents generalizable scaling laws as a layered concept. At the strongest level, a law is invariant under an explicitly defined transformation. At an intermediate level, it deforms predictably along a small set of latent axes such as intrinsic dimension, redundancy, or information resolution. At the weakest level, it is a flexible extrapolant that remains accurate across breaks and multiple control variables. The main open question is not whether scaling laws exist, but which invariants are sufficient to transport them reliably across the domains in which they are most needed.