Weight Decay Phase Structure

• Weight decay phase structure is a framework that defines distinct regimes where varying the decay parameter reorganizes training trajectories, curvature, and optimization stability across architectures.
• It identifies regimes such as the edge of stability, grokking transitions, and representation collapse, using diagnostics like Hessian sharpness, NTK spectra, and global alignment metrics.
• The approach leverages both analytical and empirical methods to balance radial versus tangential regularization, influencing convergence speed, generalization, and representation geometry.

to=arxiv_search 天天中彩票nbajson
{"query":"id:2605.16622 OR \"Does Weight Decay Enhance Training Stability?\"","max_results":5}
to=arxiv_search ӡамjson
{"query":"id:2605.20441 OR \"Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics\"","max_results":5}
to=arxiv_search รับเงินบาทjson
{"query":"id:2605.06599 OR \"Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization\"","max_results":5}
to=arxiv_search เงินฟรีjson
{"query":"id:2606.04405 OR \"Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View\"","max_results":5}
to=arxiv_search 彩神争霸怎么json
{"query":"id:2506.14562 OR id:2510.15262 OR id:2309.04644 OR id:2410.04887 OR id:2505.20172","max_results":10}
“Weight decay phase structure” denotes the existence of distinct, qualitatively different regimes induced by varying the weight decay parameter, but the phrase is not used in a single canonical sense across the literature. In current work on deep learning, it has been used to describe architecture-dependent curvature regimes at the Edge of Stability, memorization–grokking–collapse transitions, radial-versus-tangential regularization dynamics in scale-invariant networks, phase-like changes in neural-collapse geometry, module-wise spectral balancing, and a valley-to-bowl transition in the global loss landscape of regularized Transformers [2605.16622], [2605.20441], [2605.06599]. Across these usages, weight decay functions as a control parameter whose variation reorganizes training trajectories, spectral statistics, geometric representations, and asymptotic optimization behavior.

1. Scope, definitions, and recurrent formal structure

A standard starting point is the regularized objective
[
\tilde L(\theta)=L(\theta)+\frac{\gamma}{2}|\theta|^2
]
or, in Transformer language,
[
\mathcal{F}(\theta)=\mathcal{L}(\theta)+\frac{\lambda}{2}|\theta|^2.
]
For full-batch gradient descent this yields
[
\theta_{t+1}=(1-\eta\gamma)\theta_t-\eta\nabla L(\theta_t),
]
while in AdamW-style formulations one writes
[
\theta_{t+1}=\theta_t-\eta\,\hat g_t-\eta\lambda\theta_t.
]
The notation varies by paper: (\gamma) in the Edge-of-Stability analysis, (\lambda) in grokking, Transformer, and neural-collapse work [2605.16622], [2602.18523], [2605.06599].

The term “phase structure” is therefore best understood as a family of regime descriptions indexed by observables. In some papers the observables are Hessian sharpness and oscillations; in others they are grok rate, time-to-grok, commutator defects, heavy-tailed exponents, neural-collapse metrics, or the Villani diagnostic (\Psi_s(\theta)).

Setting	Control parameter and observables	Regimes
Edge of Stability	(\gamma), sharpness (S(\theta)), oscillations, (c_y=\langle \nabla S,\theta\rangle)	CNN damped oscillations; MLP phase transition to lower stabilizing sharpness
Grokking Transformers	(\lambda), grok rate, time-to-grok, (\bar s), (\sigma_H)	memorization; developmental grokking; collapse
Regularized Transformer energy	(\lambda), (\Psi_s(\theta)), (C_{\mathrm{LS}}), (C_{\mathrm{P}})	non-Villani valley phase; weak-decay phase; strong-decay bowl phase

This multiplicity of meanings is not merely terminological. It reflects the fact that weight decay interacts differently with local curvature, global geometry, normalization-induced scale invariance, and task-specific inductive bias.

2. Curvature regimes at the Edge of Stability

In the sharpness-based formulation, the central observable is
[
S(\theta)=\lambda_{\max}(\nabla^2 L(\theta)),
]
with the classical quadratic stability threshold (S(\theta)<2/\eta). Without weight decay, training often exhibits progressive sharpening until the top curvature reaches the Edge of Stability (EoS), followed by persistent oscillatory dynamics. With weight decay, the naive local model predicts a shifted boundary (2/\eta-\gamma), because the Hessian of the regularized loss is (H(\theta)+\gamma I) [2605.16622].

The detailed phase structure is more subtle. Weight decay robustly slows progressive sharpening and delays EoS onset across both MLPs and CNNs. Once EoS is reached, the architecture dependence becomes decisive. In CNNs, increasing (\gamma) damps sharpness oscillations around a level close to (2/\eta) or (2/\eta-\gamma), and larger weight decay delays the later chaotic regime in which multiple Hessian eigenvalues reach the EoS boundary. In MLPs, by contrast, increasing (\gamma) can produce a genuine phase transition: above a critical (\bar\gamma), sharpness stabilizes far below the naive (2/\eta-\gamma) threshold even though per-step loss remains non-monotonic and gradients and sharpness still oscillate. For (\eta=0.02) and (\gamma=0.02), the naive threshold is (99.98), whereas the observed stabilization is around (S\approx 80) [2605.16622].

The mechanistic variable in that transition is the global alignment
[
c_y=\langle \nabla S,\theta^*\rangle,
]
which appears in the reduced dynamics together with the progressive sharpening coefficient
[
\alpha=-\langle \nabla L(\theta^*),\nabla S(\theta^*)\rangle>0.
]
The resulting criterion
[
c_y^{\mathrm{crit}}=\frac{\alpha}{\gamma}+\gamma
]
separates a period-2 EoS limit cycle from a “limit cycle collapse” regime. If (c_y<c_y^{\mathrm{crit}}), training stabilizes near (2/\eta-\gamma); if (c_y\ge c_y^{\mathrm{crit}}), the limit cycle collapses and the system stabilizes at lower sharpness while remaining at the boundary of local instability in the generalized EoS sense [2605.16622].

This same paper links the parameter-space phase structure to function-space stability through the empirical NTK (\Theta=JJ^\top). As residuals shrink, (\lambda_{\max}(\Theta)\approx\lambda_{\max}(H)), so the MLP phase transition also depresses the NTK spectral radius. A plausible implication is that the relevant “stability threshold” under regularization is no longer a single curvature cutoff, but a dynamical balance involving higher-order self-stabilization and global alignment. The paper states this point sharply: curvature thresholds derived from convex or quadratic heuristics are not reliable diagnostics under regularization [2605.16622].

3. Grokking, delayed generalization, and two-timescale phases

A distinct use of the term appears in grokking. In a general gradient-flow framework, adding small weight decay to
[
F_\lambda(w)=F(w)+\frac{\lambda}{2}|w|2^2
]
produces a two-phase behavior as (\lambda\to0). During the initial fast phase, the trajectory follows the unregularized gradient flow and converges to a manifold (\mathcal M) of critical points of (F). At time of order (1/\lambda), the trajectory enters a slow drift phase and follows a Riemannian gradient flow minimizing (\ell_2)-norm on (\mathcal M):
[
\dot{\tilde w}^\circ(t)=-\operatorname{grad}{\mathcal M}\ell_2(\tilde w^\circ(t)).
]
This gives a purely optimization-based account of grokking as interpolation followed by slow norm minimization [2505.20172].

In modular-arithmetic Transformers, the empirical regimes are explicit. One study identifies three regimes along the weight-decay axis: memorization, developmental grokking, and collapse. In a dense sweep on a canonical 4-layer, 8-head Transformer, a near-transition logistic fit localizes the memorization-to-developmental boundary at (\lambda_c=0.0158) with (95\%) confidence interval ([0.0109,0.0200]), and a power-law fit gives an empirical exponent (\nu=0.757) with confidence interval ([0.725,0.799]). Very large (\lambda) produces a collapsed attention state with (\bar s=1.000) and (\sigma_H=0.000) [2605.20441].

The same work introduces two cheap online diagnostics from attention activations alone:
[
\bar s(t)=\mathbb{E}l!\left[\frac{2}{H(H-1)}\sum{i<j}\cos(\mathrm{vec}(A_{li}),\mathrm{vec}(A_{lj}))\right]
]
and
[
\sigma_H(t)=\mathbb{E}l!\left[\mathrm{Std}_h(H[A{lh}])\right].
]
In the developmental grokking regime, training exhibits an early synchronization phase in which (\bar s(t)) rises to a high plateau and a later differentiation phase in which (\bar s(t)) falls and (\sigma_H(t)) rises while test accuracy remains high. That pattern survives a horizon-matched multi-task replication across (\mathrm{mod}+), (\mathrm{mod}-), (\mathrm{mod}\times), and (\mathrm{mod}\div), although the numerical (\lambda_c) is protocol-dependent [2605.20441].

A multi-task geometric analysis sharpens the same picture. In dual-task and tri-task modular arithmetic, AdamW with (\lambda\in{0.0,0.1,0.2,0.3,0.5,1.0}) yields a sharp no-decay failure mode: non-zero (\lambda) always leads to grokking, while (\lambda=0) never does within training budgets up to (250)k steps. As (\lambda) increases, grokking becomes faster, curvature depth becomes more negative, and the reconstruction threshold (k^*(\lambda)) decreases, meaning that the final solution occupies fewer principal trajectory directions. This work explicitly distinguishes a high-decay phase, an intermediate-decay phase, a low-decay phase, and a no-decay failure phase [2602.18523].

Taken together, these results suggest that the two-timescale theory and the empirical memorization–developmental–collapse diagrams are describing the same structural phenomenon from complementary viewpoints: a fast interpolation phase followed by a slower compression or norm-reduction phase, with the weight-decay magnitude setting both the existence and the duration of the second phase.

4. Scale invariance and the geometry of the regularizer

In scale-invariant architectures, the geometry of the regularizer itself becomes the phase-defining object. For Query–Key-normalized and RMSNorm-equipped Transformers, standard Frobenius-norm weight decay acts purely along the radial direction. Writing a scale-invariant block as (W=\rho\Theta) with (\rho=|W|_F) and (|\Theta|_F=1), one has
[
\dot{\rho}=-\lambda\rho
]
and no direct contribution of L2 decay to the angular dynamics of (\Theta). After task gradients vanish, L2 therefore keeps shrinking norms while leaving the represented function essentially unchanged. This defines a post-memorization phase in which standard weight decay is functionally inert [2606.04405].

The same paper introduces Low-Rank Decay (LRD),
[
J_{\mathrm{LRD}}(W)=\mathcal L_{\mathrm{task}}(W)+\lambda|W|_*,
]
whose subgradient is the polar factor (UV^\top). Unlike L2, the nuclear-norm-like penalty has a tangential component even in the scale-invariant setting. Under the decoupled update
[
W\leftarrow W-\eta\lambda\,\operatorname{polar}(W),
]
the singular values undergo subtractive rather than multiplicative shrinkage, promoting effective-rank collapse. On modular addition with (p=97), this expands the data-fraction boundary at which delayed generalization occurs and produces rapid stable-rank collapse in (W_Q) and (W_K) [2606.04405].

A related but older line studies the “disharmony” between the weight-normalization family and standard L2. When (\boldsymbol W'=\boldsymbol W/|\boldsymbol W|) or an analogous standardized form, decay on (\boldsymbol W') becomes a constant, while decay on raw (\boldsymbol W) merely modulates the effective learning rate. In that setting, standard L2 can cause the missing of global minimum and training instability; the proposed (\epsilon)-shifted (L_2) regularizer replaces (\frac{1}{2}\lambda|\boldsymbol W|^2) by a penalty centered at a positive radius (\epsilon), restoring the existence of global minimum and preventing weights from becoming too small [1911.05920].

FixNorm makes the same decomposition explicit in BN-heavy image models. On BN-followed layers, weight decay mainly affects the effective learning rate; on the final fully connected layer, it affects generalization performance by controlling cross-boundary risk. FixNorm therefore discards weight decay and directly controls these two mechanisms by fixing the global norm of convolutional weights and clipping the gain in a weight-normalized classifier [2103.15345]. A plausible implication is that any account of weight decay phase structure in normalized architectures must distinguish radial norm control from functional boundary control; a single scalar “regularization strength” no longer has a uniform interpretation across layers.

5. Representation geometry, spectral balancing, and width-dependent regimes

Weight decay also induces phase structure in representation geometry. In neural-collapse work, the near-optimal loss regime with last-layer batch normalization and weight decay produces explicit (\lambda)-dependent bounds on cosine-based NC metrics. For at least a (1-\delta) fraction of classes and class pairs, larger (\lambda) tightens the bounds toward (\mathit{intra}c\to1), (\mathit{inter}{c,c'}\to -1/(C-1)), and (\cos_\angle(\dot w_c,\tilde h_c)\to1). Empirically, this gives a strong NC phase with BN and sufficiently large WD, a weak NC phase for small WD, and weaker or unstable behavior without BN [2309.04644].

A complementary end-to-end theory shows that wide networks trained with weight decay provably exhibit neural collapse. For networks with a wide first layer and a deep linear head, gradient descent with weight decay yields low training error and balancedness of the linear layers, where balancedness is expressed through quantities such as
[
W_{\ell+1}^\top W_{\ell+1}-W_\ell W_\ell^\top.
]
Under additional bounded-conditioning assumptions, NC1, NC2, and NC3 follow. In this formulation, balancedness and conditioning serve as the structural markers of a neural-collapse phase selected by regularization [2410.04887].

In large language models, the phase structure can be module-wise rather than layer-terminal. AlphaDecay measures heavy-tailedness of module weight-correlation spectra via a power-law exponent (\alpha). Attention modules such as att.q and att.k exhibit strongly heavy-tailed spectra, while mlp.gate, mlp.up, and mlp.down are lighter-tailed. AlphaDecay maps this spectral phase structure to module-wise decay coefficients:
[
f_t(i)=\eta\left(\frac{\alpha_t^i-\alpha_t^{\min}}{\alpha_t^{\max}-\alpha_t^{\min}}(s_2-s_1)+s_1\right),
]
so that low-(\alpha) modules receive weaker decay and high-(\alpha) modules stronger decay. The empirical effect is a more balanced module-wise (\alpha) profile and lower perplexity than uniform decay [2506.14562].

A different width-dependent phase structure appears in wide AdamW-trained Transformers. There the steady-state singular-value spectrum of matrix-like parameters scales in norm as (\sqrt{\eta/\lambda}) with approximately invariant shape, and the top singular value scales approximately as
[
\sqrt{\eta/\lambda}\cdot d^{0.75}.
]
Combining this with the (\mu)P rule (\eta_2\propto d^{-1}) implies the empirical scaling law
[
\lambda_2\propto \sqrt d,
]
which approximately preserves sublayer gain across widths. The resulting good phase is characterized by width-invariant top singular values and sublayer gains; deviations produce exploding-gain or over-regularized phases [2510.15262].

6. Confinement, diagnostics, and unresolved questions

In the most global formulation, weight decay changes the asymptotic type of the loss landscape itself. For the regularized Transformer objective
[
\mathcal F(\theta)=\mathcal L(\theta)+\frac{\lambda}{2}|\theta|^2,
]
one paper proves that (\mathcal F) is a Villani function for every (\lambda>0): it is (C^\infty), coercive, has Gaussian-integrable tails, and satisfies
[
\Psi_s(\theta)=-\Delta\mathcal F(\theta)+\frac{1}{s}|\nabla\mathcal F(\theta)|^2\to\infty
\quad\text{as }|\theta|\to\infty.
]
This yields explicit log-Sobolev and Poincaré bounds,
[
C_{\mathrm{LS}}(\mathcal F)\le \frac{s}{\lambda}\Big(1+\frac{d}{\lambda s}\Big),\qquad
C_{\mathrm P}(\mathcal F)\le C_{\mathrm{LS}}(\mathcal F),
]
and supports a phase interpretation in which (\lambda=0) corresponds to a non-Villani valley phase and (\lambda>0) to a confining bowl phase whose strength increases with (\lambda) [2605.06599].

The literature therefore uses a wide range of diagnostics: Hessian sharpness and alignment terms such as (\alpha(t)=-\langle\nabla L,\nabla S\rangle) and (c_y(t)=\langle\nabla S,\theta(t)\rangle) for EoS dynamics; (\bar s) and (\sigma_H) for cheap online grokking diagnostics; (\Psi_s) for confinement; module-wise heavy-tailed exponents (\alpha) for spectral balancing; and matching top singular values for width-robust hyperparameter transfer [2605.16622], [2605.20441], [2605.06599], [2510.15262].

Several controversies remain explicit. The EoS work argues that curvature thresholds from convex or quadratic heuristics are not reliable under regularization [2605.16622]. The modular-grokking diagnostics paper reports (\nu=0.757) as empirical and defers universality-class identification to denser finite-size-scaling work [2605.20441]. The Villani analysis notes that its constants are likely loose and that exotic architectures are not treated rigorously [2605.06599]. More broadly, the numerical value of a “critical” weight decay is architecture-specific: the transition replicates across Transformer, MLP, LSTM, and Mamba probes, but with different (\lambda_c) values [2605.20441].

The cumulative picture is therefore not that weight decay induces one universal phase diagram. Rather, it induces a family of architecture-dependent, objective-dependent, and observable-dependent phase structures. In sharpness-based work it reorganizes the EoS limit cycle; in grokking it controls memorization, developmental generalization, and collapse; in scale-invariant settings it separates radial from tangential regularization; in representation geometry it strengthens neural collapse; in spectral analyses it balances heavy-tailed modules or preserves width-invariant gains; and in functional-analytic treatments it converts non-coercive valleys into confining bowls.