Understanding training dynamics of deep neural networks

Establish a rigorous, general theory explaining the training dynamics of deep neural networks, characterizing how optimization processes evolve and under what conditions they converge or reach stationary behavior, in order to clarify the mechanisms that govern empirical performance and guide principled choices of training hyperparameters.

Background

The paper frames its contribution within a physics-inspired perspective on optimization, noting that despite strong empirical success, the mechanisms underlying how deep neural networks train and reach stationary behavior are not well understood. This motivates introducing a thermodynamic analogy—interpreting stochastic gradient noise as thermal fluctuations and identifying macroscopic variables (temperature, pressure, volume) linked to learning rate and weight decay.

By focusing on scale-invariant neural networks and deriving stationary distributions from stochastic differential equations, the authors provide partial progress toward this broader goal. However, they explicitly acknowledge that achieving a comprehensive understanding of training dynamics remains an open problem and use their framework to illuminate aspects of stationary behavior and hyperparameter effects.

References

Understanding the training dynamics of deep neural networks remains a major open problem, with physics-inspired approaches offering promising insights.

— Can Training Dynamics of Scale-Invariant Neural Networks Be Explained by the Thermodynamics of an Ideal Gas?  (2511.07308 - Sadrtdinov et al., 10 Nov 2025) in Abstract (page 1)

Another direction is to go beyond the isotropic orthogonal setting, for example to non-isotropic weights or near-orthogonal target directions \citep{oko2024learning,ren2025emergence}.

— Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry  (2609.10879 - Zhou et al., 9 Sep 2026) in Section Conclusion

We do not prove global convergence to a limit cycle, nor extend the result to anisotropic covariance or general deep networks.

— When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay  (2609.09116 - Amin et al., 8 Sep 2026) in Section Further discussion on the scope, limitation, and future directions, paragraph 'Limitations'

However, understanding why such a loss landscape structure appears when training deep learning models is still largely an open problem (see for a first approach).

— The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow  (2609.01034 - Berthier, 1 Sep 2026) in Conclusion, first paragraph