Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning (2010.05627v2)

Abstract: It is not clear yet why ADAM-alike adaptive gradient algorithms suffer from worse generalization performance than SGD despite their faster training speed. This work aims to provide understandings on this generalization gap by analyzing their local convergence behaviors. Specifically, we observe the heavy tails of gradient noise in these algorithms. This motivates us to analyze these algorithms through their Levy-driven stochastic differential equations (SDEs) because of the similar convergence behaviors of an algorithm and its SDE. Then we establish the escaping time of these SDEs from a local basin. The result shows that (1) the escaping time of both SGD and ADAM~depends on the Radon measure of the basin positively and the heaviness of gradient noise negatively; (2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly because (a) the geometry adaptation in ADAM~via adaptively scaling each gradient coordinate well diminishes the anisotropic structure in gradient noise and results in larger Radon measure of a basin; (b) the exponential gradient average in ADAM~smooths its gradient and leads to lighter gradient noise tails than SGD. So SGD is more locally unstable than ADAM~at sharp minima defined as the minima whose local basins have small Radon measure, and can better escape from them to flatter ones with larger Radon measure. As flat minima here which often refer to the minima at flat or asymmetric basins/valleys often generalize better than sharp ones , our result explains the better generalization performance of SGD over ADAM. Finally, experimental results confirm our heavy-tailed gradient noise assumption and theoretical affirmation.

• The paper shows that SGD’s shorter escaping times from local minima, driven by heavy-tailed noise, contribute to its superior generalization compared to Adam.
• It applies Lévy-driven stochastic differential equations to model the dynamic behavior of both algorithms and their basin transitions.
• Empirical and theoretical findings confirm that SGD’s preference for flatter minima leads to enhanced performance on new data relative to Adam.

Analyzing Generalization in Deep Learning: A Comparative Study of SGD and Adam

The paper addresses the persistent question of why Stochastic Gradient Descent (SGD) often demonstrates superior generalization compared to adaptive gradient methods like Adam in deep learning contexts. Despite Adam's expedited training capabilities, its performance tends to plateau when tested against new data, leading to its perception as having inferior generalization capabilities relative to SGD. This issue is particularly perplexing given Adam's design enhancements, which use coordinate-wise learning rate adaptations to more effectively navigate the optimization landscape.

The authors approach this discrepancy by exploring the local convergence behavior of the algorithms. Central to their analysis is the observation of heavy-tailed gradient noise present in both SGD and Adam. This characteristic motivates the incorporation of Lévy-driven stochastic differential equations (SDEs) to model and analyze these algorithms' dynamic behaviors. The theoretical foundation provided by Lévy SDEs allows the authors to derive the "escaping time" from local minima, which measures the duration required for the algorithm to escape a local optimization basin.

Several significant findings emerge from this analysis:

1. Radon Measure and Generalization: The escaping time — the measure of how quickly the algorithm can move away from a current local minimum — is determined by Radon measures of the basin, linked to geometry adaptation and stochastic properties of the algorithm. The analysis shows that SGD benefits from shorter escaping times in comparison to Adam, allowing it to transition to broader minima (larger Radon measure), which support better generalization.
2. Comparison of Escaping Dynamics: Theoretical results indicate that SGD is more locally unstable at sharper minima compared to Adam. This instability, in the context of optimization, translates to an ability to avoid sharp, poorly generalizing minima in favor of flatter, better-generalizing basins.
3. Influence of Noise Structure: The geometric adaptation inherent in Adam, which scales gradients to mitigate anisotropic noise, along with Adam's exponential averaging, tends to diminish the magnitude of gradient noise fluctuations. This results in lighter noise tails for Adam, contributing to larger escaping times and tethering to sharp minima. In contrast, SGD capitalizes on its heavier-tailed noise for faster basin transition.
4. Experimental and Theoretical Validation: Both theoretical analysis and experimental validations substantiate the findings. Empirical results indicate that SGD maintains a unique advantage in finding flatter minima, which often corresponds to superior generalization on testing datasets. This empirical evidence supports classical hypotheses concerning the preference of SGD for minima that generalize better due to their geometric properties.

The implications of these findings extend beyond understanding existing algorithms. They hint at potential improvements to optimization techniques in deep learning — suggesting avenues where combining adaptive methods with properties of SGD might yield both efficient training and robust generalization. Future research could develop hybrid algorithms that borrow SGD's advantageous noise exploitation while retaining the adaptive nature of algorithms like Adam, culminating in training paradigms that can efficiently scale with deep learning advancements. Such synthesis could enhance optimization strategies, providing a more nuanced toolset for AI practitioners. This careful exploration of noise dynamics and local convergence behaviors significantly contributes to the broader theoretical understanding of generalization in neural network training.