Mean-Field Residual-Scaling Law

• Mean-Field Residual-Scaling Law is a framework that defines scaling rules for residual branches, initialization variances, and learning rates to maintain non-degenerate activations and gradients.
• It distinguishes between untied networks (using 1/√L scaling) and weight-tied or looped architectures (requiring stronger 1/N scaling) to control accumulation effects.
• The framework extends to continuous-depth models, Neural Feature Dynamics, and AM-μP settings, enabling improved hyperparameter transfer and robust training dynamics.

The mean-field residual-scaling law denotes a family of asymptotic prescriptions for how residual branches, initialization variances, and learning rates must scale with depth, loop count, or both, so that forward activations, backward sensitivities, and one-step parameter updates remain non-degenerate in wide and deep limits. In standard untied residual networks, mean-field analyses typically recover depth scalings of the $1/\sqrt{L}$ type or depth-dependent initializations of the same order; in weight-tied or looped architectures, the same heuristic fails because residual increments are correlated across iterations, and the required within-loop scaling strengthens to $1/N$. Continuous-depth mean-field limits, Neural Feature Dynamics, and Arithmetic-Mean $\mu$P extend the subject from initialization-time signal propagation to full training dynamics and hyperparameter transfer (Yang et al., 2017, Chen et al., 2024, Yao et al., 24 Dec 2025, Wang et al., 16 Jun 2026, Zhang et al., 5 Oct 2025).

1. Early mean-field formulations in residual networks

A foundational mean-field treatment models a residual network through layerwise recurrences for forward variance and backward sensitivity. For the reduced ResNet $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$, the mean-field equations are

$$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$

with backward recursion

$$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$

For the full ResNet, the identity branch itself is parameterized, and the corresponding forward and backward recurrences acquire factors $\sigma_v^2$ and $\sigma_a^2$ (Yang et al., 2017).

The principal conclusion is that skip connections alter the asymptotic depth behavior from the exponential dynamics familiar in classical feedforward networks to subexponential or polynomial laws, depending on the nonlinearity. For $\phi=\tanh$, the forward length satisfies $p^\ell\approx c_1\ell$, while the backward sensitivity grows as $1/N$0. For $1/N$1-ReLU with $1/N$2, the forward norm obeys $1/N$3, and the backward sensitivity is polynomial in depth. By contrast, for ReLU, both $1/N$4 and $1/N$5 remain exponential in depth, although the weight-gradient norms stay $1/N$6 (Yang et al., 2017).

This framework motivated depth-dependent initialization rather than depth-independent Xavier or He schemes. For $1/N$7 ResNets, keeping $1/N$8 requires $1/N$9, and empirically the test-accuracy contours satisfy $\mu$0. For ReLU ResNets, the relevant condition is $\mu$1, so that the multiplicative factor $\mu$2 remains close to $\mu$3 (Yang et al., 2017). In this early sense, the mean-field residual-scaling law was primarily an initialization law that kept residual accumulation near the “edge of chaos.”

2. Scaled deep ResNets and the continuous-depth mean-field limit

A second formulation studies scaled ResNets in the joint limit of large width and large depth, where the residual branch is explicitly normalized by depth. The discrete network is written as

$\mu$4

with predictor

$\mu$5

Taking $\mu$6 with $\mu$7, followed by $\mu$8, yields the mean-field ODE

$\mu$9

and prediction

$x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$0

In this representation, $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$1 is the residual scaling and directly rescales the drift in continuous depth (Chen et al., 2024).

Training is then described by Wasserstein gradient-flow PDEs for the parameter distributions $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$2 and $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$3. The functional derivative with respect to the residual-branch distribution contains both $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$4 and $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$5: $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$6 A time-varying Gram decomposition further gives

$x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$7

with $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$8 the “ResNet-encoder” kernel and $x^{\ell}=\phi(h^{\ell})+x^{\ell-1}$9 the “Output” kernel (Chen et al., 2024).

Under the paper’s continuity and small-movement arguments, $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$0 remains bounded below by $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$1, which yields linear convergence of the empirical loss: $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$2 The same PDE analysis controls the KL divergence of the evolving parameter distributions and then the Rademacher complexity of the KL-constrained hypothesis class, leading to a dominant generalization gap of $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$3 at the trained solution (Chen et al., 2024). In this setting, the residual-scaling law is not a single exponent in depth; rather, it is the per-block factor $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$4 that guarantees a nontrivial continuous-depth mean-field regime.

3. Infinite-depth feature dynamics and depth-wise attenuation

A third line of work studies the joint infinite-width and infinite-depth limit of deep ResNets with single-layer residual blocks. The residual block is attenuated by

$$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$5

so that for each coordinate the forward increment is $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$6. The backward recursion has the same prefactor: $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$7 The stated purpose of the $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$8 factor is that, in the joint limit $$q^{\ell}=\sigma_w^2\,p^{\ell-1}+\sigma_b^2,\qquad p^{\ell}=V_\phi(q^{\ell})+p^{\ell-1},$$9, $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$0, both feature and gradient coordinates remain $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$1 and converge to a nontrivial stochastic limit (Yao et al., 24 Dec 2025).

The limiting object is Neural Feature Dynamics (NFD), a coupled forward-backward SDE system over training iterations $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$2. The forward and backward processes satisfy

$$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$3

$$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$4

Because the forward-backward correlations produced by reusing $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$5 are suppressed by $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$6, the covariances of the driving noises vanish in the infinite-depth limit, and the gradient-independence assumption is restored (Yao et al., 24 Dec 2025).

This depth-wise law also clarifies where it fails structurally. For two-layer residual blocks, the same $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$7 attenuation causes feature-learning collapse in the first internal layer: the first-layer feature update has magnitude $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$8 and vanishes as $$\chi^{\ell-1}=\bigl[\sigma_w^2\,V_{\phi'}(q^{\ell})+1\bigr]\chi^{\ell}.$$9, even though the residual-stream update remains non-degenerate in the diffusion limit. The proposed correction is to amplify the first-layer learning rate by $\sigma_v^2$0,

$\sigma_v^2$1

which “exactly cancels the $\sigma_v^2$2 suppression on the first layer” and empirically restores depth-wise hyperparameter transfer (Yao et al., 24 Dec 2025). This establishes that the mean-field residual-scaling law for depth is architecture-sensitive even within the class of residual networks.

4. Weight sharing and the looped-transformer law

The strongest recent refinement concerns looped, weight-tied Transformers, where a shared residual block is applied $\sigma_v^2$3 times: $\sigma_v^2$4 Prior depth-scaling analyses for ordinary residual networks prescribe $\sigma_v^2$5, but this is insufficient for looped architectures because weight sharing makes residual updates correlated across iterations. Writing

$\sigma_v^2$6

one obtains

$\sigma_v^2$7

where

$\sigma_v^2$8

In an untied deep network, $\sigma_v^2$9 for $\sigma_a^2$0, so $\sigma_a^2$1. With weight sharing, however, the residuals factor through the same $\sigma_a^2$2, and under the paper’s positive-cone and Gaussian-gain assumptions one obtains $\sigma_a^2$3. Theorem 1 therefore states that bounded $\sigma_a^2$4 requires $\sigma_a^2$5 when $\sigma_a^2$6, hence $\sigma_a^2$7 (Wang et al., 16 Jun 2026).

For a multi-layer block with $\sigma_a^2$8 distinct layers reused $\sigma_a^2$9 times,

$\phi=\tanh$0

the law factorizes: $\phi=\tanh$1 Here $\phi=\tanh$2 controls the within-layer loop correlation, and $\phi=\tanh$3 controls the across-layer variance. Theorem 2 states, informally, that this choice keeps the forward residual stream bounded uniformly in $\phi=\tanh$4 (Wang et al., 16 Jun 2026).

The same analysis yields a training-dynamics consequence. Theorem 3 shows that a one-step update of size $\phi=\tanh$5 perturbs the output by

$\phi=\tanh$6

At $\phi=\tanh$7, the stable learning rate is therefore constant in $\phi=\tanh$8. In the multi-layer case,

$\phi=\tanh$9

so the learning rate must scale as $p^\ell\approx c_1\ell$0, independent of $p^\ell\approx c_1\ell$1 (Wang et al., 16 Jun 2026).

The experiments are aligned with the theory. In initialization diagnostics with $p^\ell\approx c_1\ell$2 and $p^\ell\approx c_1\ell$3, $p^\ell\approx c_1\ell$4 remained bounded for $p^\ell\approx c_1\ell$5, while $p^\ell\approx c_1\ell$6 grew rapidly and even exploded for large $p^\ell\approx c_1\ell$7. The cosine similarity of per-step increments $p^\ell\approx c_1\ell$8 displayed many positive alignments in looped stacks but off-diagonal correlations $p^\ell\approx c_1\ell$9 in non-shared deep stacks, directly confirming the $1/N$00 accumulation mechanism. In full training on FineWeb-Edu language modeling, the same base learning rate was near-optimal across loop counts under $1/N$01 scaling, and validation loss improved at large $1/N$02 (Wang et al., 16 Jun 2026).

5. Learning-rate scale, residual-aware initialization, and AM-$1/N$03P

Arithmetic-Mean $1/N$04P addresses a complementary question: not how to scale the residual branch in depth alone, but how to choose a width-robust and depth-robust learning-rate scale for modern architectures. The AM-$1/N$05P rule fixes the network-wide average of the one-step pre-activation second moments,

$1/N$06

For residual architectures, it is paired with residual-aware He fan-in initialization,

$1/N$07

which divides the usual He variance by the number of residual blocks $1/N$08 and is explicitly intended to prevent residual-accumulation blowup (Zhang et al., 5 Oct 2025).

Under the paper’s structural assumptions for one- and two-dimensional convolutional networks, the network-wide average one-step update variance satisfies

$1/N$09

Enforcing the AM-$1/N$10P constraint therefore gives

$1/N$11

with $1/N$12 an $1/N$13 constant. The same $1/N$14 exponent is then established for standard residual networks with general conv+MLP blocks, using a minimal-depth convention in which each residual block counts as one depth unit (Zhang et al., 5 Oct 2025).

Empirically, across CNN families and ResNets with or without BatchNorm or Dropout, the maximal-update learning rate satisfies $1/N$15 with $1/N$16, and segmented fits demonstrate “zero-shot” learning-rate transfer from small to large depths (Zhang et al., 5 Oct 2025). This result does not replace the residual-branch scalings above; rather, it complements them by specifying an optimization-scale law once residual accumulation has already been regularized through depth-aware initialization.

6. Synthesis, scope, and common points of confusion

The literature does not support a single universal exponent called the mean-field residual-scaling law. Instead, it supports a family of architecture-dependent laws whose differences are explained by the correlation structure of residual increments. In untied deep stacks with independent $1/N$17, the classical mean-field picture gives the familiar $1/N$18 behavior for residual magnitude or initialization scale; in looped or recurrent-like residual systems, the same argument breaks because the shared weights induce constructive quadratic accumulation, turning $1/N$19 growth into $1/N$20 growth and forcing the stronger $1/N$21 law within the loop (Yang et al., 2017, Wang et al., 16 Jun 2026).

A related misconception is that residual scaling alone determines trainability. The cited results show a more coupled picture. In scaled mean-field ResNets, the residual factor $1/N$22 enters the drift of the continuous-depth ODE and the functional gradient, while the final-layer factor $1/N$23 controls both the exponential convergence rate and the KL-ball in which the Gram matrix remains well-conditioned (Chen et al., 2024). In the NFD regime, the residual factor $1/N$24 is sufficient for single-layer blocks but induces feature-learning collapse in the first internal layer of two-layer blocks unless the learning rate is corrected by $1/N$25 (Yao et al., 24 Dec 2025). Under AM-$1/N$26P, the learning-rate law $1/N$27 emerges only after imposing a residual-aware initialization that divides the He variance by the number of blocks (Zhang et al., 5 Oct 2025).

A further point of clarification concerns transferability. In looped Transformers, the factorized law

$1/N$28

separates the two sources of growth: the loop axis and the unique-depth axis. The resulting stable learning rate depends only on $1/N$29, not on $1/N$30, which enables direct hyperparameter transfer across loop counts. This is a distinct claim from the AM-$1/N$31P result, where the transferred quantity is the maximal-update learning rate across ordinary depths and the governing exponent is $1/N$32 (Wang et al., 16 Jun 2026, Zhang et al., 5 Oct 2025).

Taken together, these works suggest that “mean-field residual-scaling law” is best understood as a structural principle: residual branches must be attenuated according to the effective correlation pattern of their increments, and optimization scales must then be matched to that attenuation. For independent residual branches, the classical random-walk intuition yields $1/N$33-type laws; for weight-tied loops, positive alignment necessitates $1/N$34; for continuous-depth and feature-learning formulations, the same scaling enters the limiting ODE or SDE; and for practical depth transfer, residual-aware initialization and learning-rate laws such as $1/N$35 become part of the full prescription (Chen et al., 2024, Yao et al., 24 Dec 2025, Wang et al., 16 Jun 2026, Zhang et al., 5 Oct 2025).