Incorporate weight decay (e.g., AdamW) into the implicit-bias analysis

Characterize how weight decay or decoupled decay terms—such as the W-term in AdamW—modify the effective sharpness regularizer within the slow SDE/ODE framework for Adam-like methods.

Background

The presented analysis does not cover weight decay or decoupled decay mechanisms commonly used in adaptive methods such as AdamW. These terms can substantially alter optimization trajectories and, potentially, the resulting implicit regularization.

The authors identify the need to extend the framework to include weight-decay effects and to determine how such terms change the effective sharpness objective that Adam implicitly minimizes.

References

Despite these advances, several important avenues remain open. Finally, our approach cannot cover weight‐decay or decoupled decay terms such as the $W$-term in AdamW; characterizing how weight decay alters the effective sharpness regularizer is an important direction for follow-on studies.

— Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold  (2511.02773 - Li et al., 4 Nov 2025) in Conclusions

There exists a critical regularization strength $\lambda* \approx 1.0$ (relative to the AdamW scale used here) such that generalization onset is structurally suppressed for $\lambda < \lambda*$. Below $\lambda*$, the memorizing solution is a stable fixed point of the training dynamics; above $\lambda*$, weight decay destabilizes the memorizing basin, and the network converges to a lower-norm generalizing solution.

— Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking  (2609.10657 - Kataria, 9 Sep 2026) in Conjecture 1 ("Phase boundary"), Section 5, immediately following Figure 4