Unified Neural Scaling Law (UNSL)

• UNSL is a scaling law defining the dependence of generalization error on resources like model size, data, training duration, and compute.
• It reveals a two-phase behavior: an initial exponential decay when compute is limited, followed by a slower power-law reduction in error as compute increases.
• The framework recovers classical single-variable scaling laws and guides optimal resource allocation for modern architectures such as transformers.

A unified neural scaling law (UNSL) specifies a quantitative relation describing how the generalization error or evaluation loss of deep neural networks scales with multiple resource variables, including model size, data size, training duration, and compute. UNSL provides a functional form accurate across all resource regimes, predicts phase transitions in error reduction rates, and recovers previously observed single-variable scaling laws as special cases. Recent advances rigorously formalize UNSL for modern architectures, particularly transformers, linking kernel-based ODE dynamics and generalization, and clarifying when and why scaling transitions occur (Yang, 26 Dec 2025).

1. Formal Modeling of Training Dynamics

UNSL is constructed via an explicit ordinary differential equation (ODE) formulation of the training process for multilayer transformers trained on sequence-to-sequence tasks. With stochastic gradient descent (SGD) under infinitesimal step sizes, the parameter trajectory $\theta(t)\in\mathbb{R}^M$ evolves as

$$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$

Layerwise expansion and neural-tangent-kernel (NTK) decomposition yield

$$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$

Here $H^{(v)}(t)$ is the NTK matrix of layer $v$. The weight blocks $U^{(v)}, W^{(v)}$ follow their respective ODEs:

$$\dot{U}^{(v)}(t) = -\frac{\partial L}{\partial U^{(v)}}\,,\qquad \dot{W}^{(v)}(t) = -\frac{\partial L}{\partial W^{(v)}}$$

This yields a continuous-time dynamical system closely paralleling real-world SGD training of large transformers (Yang, 26 Dec 2025).

2. Two-Phase Structure and Phase Transition

The key result is an explicit two-phase upper bound for the excess risk, $\mathrm{AR}(F) := R(F) - R(F^*)$, where $R(F)$ is the population risk and $F^*$ the Bayes predictor. Let $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$0 denote total training compute in terms of model size $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$1, training time $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$2, and effective batch size $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$3.

• Phase I (Optimization-dominated; $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$4): Excess risk exhibits exponential decay:

$$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$5

with noise level $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$6, model depth $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$7, and constants absorbed into $$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$8.

• Critical point:

At compute

$$\frac{d}{dt}\,\theta(t)\;=\;-\nabla_{\theta}L\bigl(t,D\bigr), \quad L(t,D) = \mathbb{E}_{(X,Y)\sim D}\left[\|F(X;\theta(t))-Y\|^2\right]$$9

• Phase II (Statistics-limited; $$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$0): Power-law decay sets in,

$$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$1

More precisely, logarithmic corrections enter through the Lambert W function, but the dominant exponent is $$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$2.

This structure captures the empirically observed transition: initial rapid (nearly exponential) error reduction, followed by a slower, power-law regime as the system saturates optimization limits and enters the statistical regime (Yang, 26 Dec 2025).

3. Recovery of Classical Single-Variable Scaling Laws

Fixing all but one scaling resource enables UNSL to reproduce canonical scaling laws:

• Training time law ($$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$3 fixed, $$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$4, $$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$5 varied):

$$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$6

Indicative of exponential error reduction per additional epoch up to the stochastic noise floor.

• Data law ($$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$7 varied):

$$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$8

Standard data-scaling with a $$\frac{d}{dt}\,\mathrm{vec}(\theta(t)) = -\sum_{v=1}^N (H^{(v)}(t)\otimes I)\,\mathrm{vec}\left(\partial_{p^{(v)}}L(t,D)\right)$$9 exponent.

• Model law (optimize $H^{(v)}(t)$0 for fixed compute; $H^{(v)}(t)$1 up to $H^{(v)}(t)$2):

$H^{(v)}(t)$3

Power-law in $H^{(v)}(t)$4 with sublinear exponent, reflecting the diminishing return from further model growth beyond dataset-aligned regimes.

These reductions provide precise understanding of when each resource dominates and elucidate the respective bottlenecked regime (Yang, 26 Dec 2025).

4. Phase Diagram and Implications

The phase transition at compute $H^{(v)}(t)$5 delineates two fundamental scaling regimes:

Regime	Compute Bound	Error Decay	Return on Compute
Optimization-starved	$H^{(v)}(t)$6	Exponential ($H^{(v)}(t)$7)	Rapid error reduction
Statistical-limited	$H^{(v)}(t)$8	Power law ($H^{(v)}(t)$9)	Diminishing returns

This predicts and explains empirical findings (Kaplan et al. 2020; Hoffmann et al. 2022) showing two-phase error curves: fast early descent, then an extended power-law tail with exponents close to 0.08–0.2. The theory's $v$0 exactly matches observed LLM scaling exponents in the asymptotic regime (Yang, 26 Dec 2025).

5. Comparison to Classical and Empirical Neural Scaling Laws

UNSL derived from ODE/NTK dynamics synthesizes and generalizes empirical laws:

• It matches the two-stage scaling phenomenology: initial exponential error reduction (compute-leveraged), followed by the universal $v$1 power law in the data-limited phase.
• The exponent $v$2 corresponds closely to empirical fits derived from large-scale transformer training.
• The precise phase-boundary quantifies where further compute becomes statistically unproductive, dictating optimal allocation strategies.

This resolves the "mystery" of why observed error curves for LLMs and similar models display sharp regime changes and slow long-run progress despite continued scaling (Yang, 26 Dec 2025).

6. Resource Allocation and Practical Guidance

UNSL provides principled design patterns:

• For $v$3: Allocation to longer training, increased depth/width, or both realizes exponential gains; thus, compute should be concentrated on pushing into the high-yield regime.
• For $v$4: Returns diminish to $v$5. Improvement beyond this requires proportional increases in dataset size or reductions in noise; naively increasing $v$6 or $v$7 is suboptimal.
• Model-scaling saturation: Scaling $v$8 leads to law breakdown; dataset size becomes the limiting constraint.

These prescriptions optimize resource utilization and anchor foundation model design in rigorous theory (Yang, 26 Dec 2025).

7. Theoretical Significance and Outlook

The UNSL represents the first derivation of a phase transition in transformer generalization error using an ODE–NTK approach applicable to multi-layer architectures under arbitrary data distributions. It unifies the statistical and optimization perspectives, bridges the gap between empirical scaling laws and theoretical analysis, and establishes a foundation for principled extrapolation and architecture-aware budgeting in large-scale model development (Yang, 26 Dec 2025).