Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

Extend slow-SDE analysis beyond the 2-scheme scaling of β2

Extend the slow SDE/ODE analysis and implicit-bias characterization for Adam and the broader AGM framework from the 2-scheme regime (where 1−β2=Θ(η^2)) to the intermediate 1.5-scheme and other scalings of 1−β2, deriving the correct continuous-time limit and identifying the associated sharpness regularizer that these methods implicitly minimize.

Background

The paper develops a slow SDE-based analysis for Adam and related adaptive gradient methods under the "2-scheme" scaling, where the second-moment decay parameter satisfies 1−β2=Θ(η2). This scaling is chosen to make the preconditioner evolve on the same slow timescale as the manifold dynamics, enabling a rigorous approximation and an implicit-bias characterization.

The authors explicitly note that extending this analysis beyond the 2-scheme—particularly to the intermediate 1.5-scheme or other scalings of 1−β2—remains unresolved and is left for future work. Such extensions would test the robustness of the current theory and potentially reveal qualitatively different implicit regularizers.

References

Despite these advances, several important avenues remain open. First, we have focused on the “2-scheme” regime (where $1-\beta_2=O(\eta2)$) in order to track Adam’s preconditioner over a long timescale; extending our analysis to the intermediate 1.5-scheme or other scalings of $1-\beta_2$ is left for future work.

— Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold  (2511.02773 - Li et al., 4 Nov 2025) in Conclusions

Still, optimization hyperparameters such as $\beta_2$ of Adam clearly affect the distribution of eigenvalues of Hessian; how to understand its mechanism remains a largely open problem.

— Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models  (2610.13524 - Tian et al., 9 Oct 2026) in Section 4, paragraph “Spectrum Analysis of Hessian and Model Weights”