Weight Decay–Noise Equilibrium

• Weight decay–noise equilibrium is a steady state in scale-invariant layers where stochastic gradient noise and L2 decay balance, stabilizing weight norms and angular updates.
• Closed-form formulas under SGDM and AdamW reveal how hyperparameters and normalization schemes determine equilibrium norms and angular rotation rates.
• These insights guide hyperparameter tuning and optimizer design, ensuring layer-wise homogeneity and improved training dynamics, as validated in large-scale benchmarks.

Weight decay–noise equilibrium, also described as rotational equilibrium or spherical motion dynamics, denotes the stationary behavior that emerges in the learning dynamics of scale-invariant neural network layers trained with stochastic gradient updates, $L_2$ weight decay, and normalization. In this regime, the interplay between stochastic gradient noise and the shrinking effect of weight decay causes the weight vectors of individual neurons to reach a steady state: their norm and average angular update stabilize, balancing inflating (noise-driven) and shrinking (decay-driven) dynamics. This equilibrium has precise closed-form expressions for the equilibrium norm and angle swept per update, which depend algebraically on hyperparameters, optimizer moments, and normalization scheme. Such equilibria provide geometric and mechanistic insight into the efficacy of methods like AdamW, Weight Standardization, and SGDM when used with normalization (Kosson et al., 2023, Wan et al., 2020).

1. Geometric Structure of Rotational Equilibrium

Consider a neuron’s weight-vector $w_t \in \mathbb{R}^C$ updated at training step $t$ according to

$$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$

where $\Delta_n$ represents the noisy (stochastic) gradient step, $\eta$ is the learning rate, and $\lambda$ is the $L_2$ weight decay coefficient.

In normalized layers (e.g., BatchNorm, Weight Standardization), stochastic gradients tend to be nearly orthogonal to $w_t$ and have magnitude $\|\Delta_n\| \propto \|w_t\|^{-1}$. Weight decay acts purely in the radial direction, shrinking the norm, while the noisy gradient injects energy orthogonal to the weight vector, inducing angular motion. Steady-state—termed rotational (or weight decay–noise) equilibrium—occurs when:

1. The expected norm (radius) $w_t \in \mathbb{R}^C$0 ceases to drift.
2. The expected angular increment $w_t \in \mathbb{R}^C$1 stabilizes to a constant mean.

At this equilibrium, the radial "inflation" from stochastic updates and radial "shrinkage" from weight decay exactly cancel in expectation, while the net angular increment persists (Kosson et al., 2023, Wan et al., 2020).

2. Closed-Form Equilibrium Conditions

Under assumptions of scale invariance and dominant stochastic gradient noise (random-walk regime), explicit closed-form solutions for the equilibrium radius $w_t \in \mathbb{R}^C$2 and equilibrium angular update $w_t \in \mathbb{R}^C$3 can be derived.

For standard SGDM (momentum $w_t \in \mathbb{R}^C$4), the equilibrium is:

• Equilibrium norm:

$w_t \in \mathbb{R}^C$5

where $w_t \in \mathbb{R}^C$6 is the unit-norm gradient.

• Equilibrium angular step:

$w_t \in \mathbb{R}^C$7

For AdamW (with moments $w_t \in \mathbb{R}^C$8), the result for a scale-invariant vector of dimension $w_t \in \mathbb{R}^C$9 is:

• Norm:

$t$0

• Angular increment:

$t$1

For sign-based optimizers like Lion, additional $t$2-based and moment-dependent factors appear. In contrast, for Adam with coupled $t$3 regularization (not decoupled weight decay), both norm and angular step become dependent on per-neuron gradient norms, breaking homogeneity (Kosson et al., 2023).

A mathematical summary:

$t$4

so the angular update stabilizes at $t$5 (Kosson et al., 2023, Wan et al., 2020).

3. Spherical Motion Dynamics and Equilibrium Proof

Within the Spherical Motion Dynamics (SMD) formalism, equilibrium is proved using a recursion for the squared norm:

$t$6

under small-step and bounded-moment assumptions. For plain SGD, the fixed-point is

$t$7

with $t$8 the second moment of the unit-gradient. With heavy-ball momentum $t$9, the denominator acquires a $$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$0 factor.

The angular step per update is likewise:

• Without momentum: $$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$1
• With momentum $$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$2: $$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$3

Both the norm and the mean angular update converge linearly to these values, up to an $$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$4 noise floor (Wan et al., 2020).

4. Empirical Manifestations and Layer-Wise Homogeneity

Empirical studies show rapid and robust convergence to the predicted equilibrium across a wide range of architectures and tasks. For instance, in ResNet-50 on ImageNet and Mask-R-CNN on MSCOCO, the measured norms and angular updates converge closely to theoretical predictions within tens of epochs, with changes in learning rate immediately reflected in predicted transient duration and final equilibrium values.

A notable empirical phenomenon is the "homogeneous equilibrium": in architectures using BatchNorm or Weight Standardization, every scale-invariant layer locks to the same equilibrium angular step, despite variations in gradient variance. This homogenization is disrupted in cases (e.g., Adam+$$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$5 vs AdamW) where per-neuron gradient scales govern the equilibrium, resulting in inter-layer discrepancies in learning speeds and final accuracy (Kosson et al., 2023, Wan et al., 2020).

Optimizer	Equilibrium Norm Formula	Angular Step Scaling
SGDM	$$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$6	$$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$7
AdamW	$$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$8	$$w_{t+1} = w_t + \Delta_n(w_t;\eta) - \eta\lambda w_t$$9
Adam+$\Delta_n$0	Per-neuron, gradient-dependent	Per-neuron, inhomogeneous
Lion	$\Delta_n$1	$\Delta_n$2

Table: Summary of closed-form equilibrium for different optimizers (Kosson et al., 2023)

5. Unified Mechanistic Insights and Optimization Implications

The existence of a weight decay–noise equilibrium provides unified geometric and mechanistic rationale for the observed benefits and tuning rules in training scale-invariant networks:

• AdamW versus Adam+$\Delta_n$3: Decoupled weight decay enforces layer-wise homogeneity in rotation rate, supporting higher accuracy ridges in hyperparameter space. In naïve Adam+$\Delta_n$4, layer speeds diverge according to local gradient norms, distorting learning and optimization difficulty.
• Weight Standardization: Imposes scale-invariance across all neurons, restoring homogeneous equilibrium and enhancing downstream accuracy, even in architectures initially lacking this property.
• Warmup Elimination: By targeting a desired initial angular rate $\Delta_n$5 via analytic solution for $\Delta_n$6 in terms of $\Delta_n$7 and optimizer moments, the need for heuristic learning rate warmup is eliminated—angular rotation immediately begins at its prescribed "safe" value.
• Monitoring and Per-Layer Adjustment: Tracking the running mean of norm and average angular step per layer enables diagnosis and correction; layers deviating from equilibrium can be automatically retuned by adjusting $\Delta_n$8.

These findings suggest that optimization, generalization, and even practical hyperparameter tuning in deep learning can be more systematically governed by geometric and stochastic equilibrium principles, rather than primarily through empirical heuristics (Kosson et al., 2023).

6. Broader Context and Extensions

The weight decay–noise equilibrium paradigm extends the classical notion of effective learning rate to a more geometric, intrinsic metric—average rotation per update—especially salient in normalized networks. This new perspective reveals that the stalling or rapid progress of weights is less about their norm and more about their angular motion, which is readily controlled and measured in equilibrium. Spherical Motion Dynamics and rotational equilibrium provide the conceptual and mathematical underpinnings for analyzing not only SGD and AdamW but also new optimizers such as Lion. These developments have already been empirically validated on large benchmarks such as ImageNet and MS COCO (Kosson et al., 2023, Wan et al., 2020).

A plausible implication is that future algorithms and normalization schemes may further exploit this rotational metric for robust, layer-wise learning control. Additionally, architectures lacking built-in scale invariance can "wrap" their update rules to enforce targeted angular speed explicitly, restoring the equilibrium and its beneficial properties.

• "Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks" (Kosson et al., 2023).
• "Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD" (Wan et al., 2020).