An Improved Analysis of Stochastic Gradient Descent with Momentum

Abstract: SGD with momentum (SGDM) has been widely applied in many machine learning tasks, and it is often applied with dynamic stepsizes and momentum weights tuned in a stagewise manner. Despite of its empirical advantage over SGD, the role of momentum is still unclear in general since previous analyses on SGDM either provide worse convergence bounds than those of SGD, or assume Lipschitz or quadratic objectives, which fail to hold in practice. Furthermore, the role of dynamic parameters has not been addressed. In this work, we show that SGDM converges as fast as SGD for smooth objectives under both strongly convex and nonconvex settings. We also establish \textit{the first} convergence guarantee for the multistage setting, and show that the multistage strategy is beneficial for SGDM compared to using fixed parameters. Finally, we verify these theoretical claims by numerical experiments.

• The paper establishes that SGDM converges as fast as SGD on smooth objectives without relying on restrictive assumptions.
• The paper introduces a multistage parameter tuning framework that accelerates early convergence and refines final outcomes.
• The paper devises a novel Lyapunov function to manage momentum-induced variance, aligning theoretical bounds with empirical performance.

An Improved Analysis of Stochastic Gradient Descent with Momentum

The paper "An Improved Analysis of Stochastic Gradient Descent with Momentum" by Yanli Liu, Yuan Gao, and Wotao Yin, addresses the theoretical understanding of Stochastic Gradient Descent with Momentum (SGDM), a popular optimization method in machine learning, particularly for training neural networks. The role of momentum in optimization algorithms, especially under non-convex settings, has been somewhat ambiguous. This paper provides a rigorous convergence analysis for SGDM, comparing its performance to that of standard SGD.

Key Contributions:

1. Convergence Analysis for Smooth Objectives: The authors establish that SGDM converges as fast as SGD for smooth objectives without making restrictive assumptions such as Lipschitz continuity or quadratic objectives. This is significant as it explains why SGDM is empirically competitive with SGD despite theoretical analysis often showing slower convergence under specific assumptions. The convergence is demonstrated for both strongly convex and nonconvex settings, providing a more comprehensive understanding of the algorithm’s capabilities.
2. Multistage SGDM Analysis: The paper is the first to introduce a convergence guarantee for SGDM applied with multistage parameter tuning strategies. Such strategies, popular in training large-scale neural networks, involve adjusting the step size and momentum over different training phases to enhance performance. The analysis shows that incorporating multistage strategies into SGDM can lead to faster convergence in the initial training phases and a more refined final solution.
3. Lyapunov Function Construction: The authors construct a new Lyapunov function to analyze the dynamics of SGDM. This function helps in handling the variance introduced by momentum and the deviation of the update vector from the ideal gradient descent direction. This is crucial for establishing theoretical bounds that match empirical observations.
4. Variance Reduction in Momentum Updates: The momentum term reduces the variance of the update vector, which otherwise leads to discrepancies when analyzing the gradient descent process. By controlling the deviation of the momentum-based update from the true gradient, the analysis presented reconciles the perceived benefits of momentum with the previously established theoretical weaknesses.

Implications and Future Directions:

The results serve both practical and theoretical implications. Practically, the analysis supports the continued use and potential optimization of SGDM in deep learning applications, encouraging the exploration of parameter schedules that can leverage the multistage approach for specific tasks. Theoretically, these results contribute to a richer understanding of momentum methods in stochastic settings, opening avenues for further research in optimizing momentum strategies and exploring their applications across varying machine learning architectures.

Future work could focus on investigating scenarios where SGDM might outperform SGD, particularly in non-convex settings typical of neural network training. There’s also a scope to explore the impacts of different learning rate schedules that could potentially yield even faster convergence rates. Furthermore, the adaptive tuning of momentum parameters depending on the data characteristics and model architecture remains an area ripe for research.

Conclusion:

This paper provides a valuable theoretical foundation for SGDM, clarifying its efficiency and practical advantages over SGD. By resolving ambiguity around momentum’s role and offering insights into multistage strategies, it enhances our understanding of one of deep learning's staple optimization algorithms, paving the way for further advancements in optimization theory and practice.