A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks (1901.06053v1)

Abstract: The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion. We argue that the Gaussianity assumption might fail to hold in deep learning settings and hence render the Brownian motion-based analyses inappropriate. Inspired by non-Gaussian natural phenomena, we consider the GN in a more general context and invoke the generalized CLT (GCLT), which suggests that the GN converges to a heavy-tailed $\alpha$-stable random variable. Accordingly, we propose to analyze SGD as an SDE driven by a L\'{e}vy motion. Such SDEs can incur `jumps', which force the SDE transition from narrow minima to wider minima, as proven by existing metastability theory. To validate the $\alpha$-stable assumption, we conduct extensive experiments on common deep learning architectures and show that in all settings, the GN is highly non-Gaussian and admits heavy-tails. We further investigate the tail behavior in varying network architectures and sizes, loss functions, and datasets. Our results open up a different perspective and shed more light on the belief that SGD prefers wide minima.

• The paper presents empirical evidence that stochastic gradient noise in deep neural networks follows a heavy-tailed, α-stable distribution rather than a Gaussian one.
• The study shows that heavy-tailed noise drives SGD to favor wide, flat minima, which may enhance model generalization.
• Robust experiments across various architectures indicate that increasing network size intensifies heavy-tailed behavior, while minibatch size has minimal effect.

A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

This paper explores the characteristics of stochastic gradient noise (GN) in the stochastic gradient descent (SGD) algorithm, specifically challenging the common Gaussianity assumption often applied in such analyses. In traditional settings, especially in the field of deep neural networks (DNNs), SGD is typically modeled as a stochastic process driven by Brownian motion under the assumption that GN adheres to a Gaussian distribution. This assumption is largely for mathematical convenience, allowing the use of the Central Limit Theorem (CLT) to approximate GN's behavior as Gaussian in large-scale data scenarios. However, this paper provides empirical evidence that such an assumption may not be valid in deep learning contexts.

Motivated by phenomena observed in other domains where GN exhibits non-Gaussian behavior, the authors invoke the generalized CLT (GCLT), suggesting that the GN converges instead to a heavy-tailed α-stable distribution. This revelation leads to the proposition that SGD should be analyzed as a process driven by Lévy motion, expanding on the traditional understanding of SDEs in machine learning contexts. Lévy-driven SDEs, characterized by possible 'jumps' in stochastic paths, align better with the observed non-Gaussian heavy-tailed distributions in GN across various DNN architectures.

Key Findings and Contributions

1. Empirical Evidence of Heavy-Tailed GN: Through extensive experiments on common DNN architectures like fully connected networks (FCNs) and convolutional neural networks (CNNs) across standard datasets such as MNIST, CIFAR10, and CIFAR100, the authors demonstrate that GN is consistently non-Gaussian with heavy tails. The data reveals that the tail-index α is notably less than 2, refuting the Gaussian assumption.
2. Implications of Heavy-Tailed Behavior: The authors argue that heavy-tailed behavior in GN implies a natural tendency for SGD to prefer wide minima. Theoretical insights from metastability in Lévy-driven SDEs suggest that these dynamics escape narrow minima more easily, which potentially explains why SGD often yields robust generalizable models.
3. Robustness Across Configurations: The paper shows that while increasing network size influences the tail-index, suggesting stronger non-Gaussian behavior in larger networks, the minibatch size has little impact on the tail-index, challenging the belief that larger minibatches drive GN toward Gaussianity.
4. Dynamic Tail-Index Observations: An analysis over iterations reveals distinct phases where SGD demonstrates a 'jumping' behavior at low α, hinting at underlying complex dynamics and barrier-crossing phenomena.

Theoretical Implications and Future Directions

The findings in this paper encourage a reevaluation of how stochastic processes are modeled in machine learning theory, specifically concerning the behavior of SGD. By leveraging Lévy motion-driven models, there is a clearer framework to understand SGD's ability to gravitate toward wide, flat minima, which are preferred for their generalization properties.

Challenges remain, particularly regarding the discretized nature of SGD processes. The authors highlight a gap in understanding how such processes translate to the continuous-time perspective offered by the current Lévy-driven theories, which need further exploration. Future research should also consider time-variant models and adaptive frameworks that can account for dynamic shifts in GN's behavior over iterations.

In conclusion, this paper provides a significant shift in the understanding of SGD and its theoretical modeling in deep learning frameworks. By establishing a connection with statistical physics and stochastic analysis, it opens new pathways for both analysis and implementation of robust machine learning algorithms.