Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks (1710.11029v2)

Abstract: Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such "out-of-equilibrium" behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.

• The paper demonstrates that SGD performs variational inference by implicitly optimizing a potential distinct from the original training loss.
• It shows that SGD avoids classical convergence by following closed-loop trajectories driven by non-isotropic, low-rank gradient noise.
• Empirical results suggest that understanding these non-equilibrium dynamics could inspire new optimization frameworks to enhance deep network training.

Analysis of Stochastic Gradient Descent and its Implicit Behavior in Deep Networks

The paper under discussion provides a comprehensive paper of stochastic gradient descent (SGD), focusing particularly on its behavior when applied to deep neural networks. The authors delve into the precise mechanism through which SGD performs implicit regularization, a concept often discussed but not thoroughly quantified in the literature. The paper reveals surprising insights into how SGD behaves differently from traditional expectations, especially in the context of deep learning.

Key Findings and Methodology

By characterizing SGD as performing variational inference, the paper establishes that SGD minimizes an average potential over the posterior distribution of weights combined with an entropic regularization term. Notably, this potential differs from the original loss function used for training. This observation suggests that SGD implicitly optimizes a different objective than what is conventionally assumed.

The research proposes that standard convergence assumptions for SGD, such as Brownian motion around critical points, do not hold true for deep networks. Instead, SGD displays closed loop trajectories influenced by non-isotropic gradient noise, where the covariance matrix may have a much smaller rank than its dimension, highlighting a critical aspect of the optimization landscape in deep learning.

To substantiate their theoretical assertions, the authors provide extensive empirical validation, demonstrating a consistent deviation of SGD's behavior from traditional models. This reinforces the paper's claims about the deterministic yet non-convergent nature of SGD in deep networks, driven largely by intrinsic gradient noise properties.

Implications and Future Directions

The implications of this research span both practical and theoretical domains in machine learning. From a practical standpoint, recognizing that SGD engages in a form of variational inference with a uniform prior expands the understanding of how various optimization methods might be structured to enhance learning efficiency in deep networks. Additionally, the paper highlights the importance of considering the noise structure in SGD, as variations in learning rate and batch size can significantly impact the model's convergence dynamics and generalization capability.

Theoretically, the paper challenges the conventional views of convergence in optimization, particularly in non-convex settings characteristic of deep learning models. The authors' approach underscores the complexity and richness of the optimization landscape and suggests that exploring beyond equilibrium-based description of SGD can lead to novel insights and techniques, potentially expanding to methods like automated architecture search and noise-induced acceleration in training algorithms.

As the field progresses, incorporating these non-equilibrium aspects into standard practices could enhance our ability to develop and deploy scalable AI systems effectively. The authors also hint at the potential of developing new optimization frameworks that could inherently harness these out-of-equilibrium dynamics for improved learning outcomes.

Conclusion

In conclusion, the paper presents a thorough examination of SGD, providing valuable insights into the intrinsic dynamics of this ubiquitous optimization algorithm when applied to deep neural networks. By moving beyond traditional assumptions about equilibrium and convergence, the research introduces a new perspective on how optimization processes in machine learning can be better understood and potentially leveraged for more effective AI development. This work paves the way for future exploration into optimization techniques that align closely with the complex, high-dimensional spaces encountered in modern AI applications.