Gradient Descent Happens in a Tiny Subspace (1812.04754v1)

Abstract: We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

• The paper finds that gradient descent predominantly operates in a tiny subspace aligned with the top eigenvectors of the Hessian matrix.
• The study uses empirical analysis with SGD, showing that gradients converge quickly to a subspace roughly equal in dimension to the dataset’s number of classes.
• The results imply computational efficiency and open avenues for improved second-order optimization methods focusing on the stable top subspace.

Analyzing the Convergence of Gradient Descent to a Subspace

The paper "Gradient Descent Happens in a Tiny Subspace" authored by Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer presents a detailed exploration into the dynamics of gradient descent in deep learning scenarios. Specifically, it investigates the behavior of gradients in large-scale neural networks, revealing that the gradients predominantly converge to a small and specific subspace during training. This fundamental observation provides insights into the optimization process and the efficacy of gradient descent in overcoming the challenges posed by high-dimensional parameter spaces.

Main Observations

The authors identify two primary observations across various model architectures and datasets:

1. Gradient Alignment with the Top Hessian Subspace: The paper observes that the gradient vector quickly becomes restricted to a subspace characterized by the leading eigenvectors of the Hessian matrix. This top subspace is notably small, with a dimension approximately equal to the number of classes in the dataset.
2. Preservation of the Top Subspace: Over long training periods, the eigenvectors spanning this top subspace remain relatively stable even as other aspects of the model parameters evolve. This stability suggests that the subspace retains relevance throughout the training process.

These findings are consistent across different architectures, such as fully connected networks, convolutional networks, and deep residual networks, reaffirming the robustness of the conclusions.

Methodology and Empirical Results

The authors employ extensive empirical measurements to substantiate their claims. They utilize standard stochastic gradient descent (SGD) and analyze the Hessian’s spectrum during training. Notably, they measure overlaps between the gradient and the Hessian-gradient product and track the evolution of eigenvalues and eigenvectors.

The analysis reveals that a significant fraction of the gradient magnitude lies within the top subspace shortly after training begins, suggesting a potentially simpler underlying mechanism than previously recognized. Additionally, the Hessian spectrum typically separates into a bulk of many small eigenvalues and a few significantly larger ones, corresponding to the top subspace.

Toy Model and Analytical Insights

To provide an intuitive understanding, a simplified model involving softmax regression on synthetic data (a mixture of Gaussians) is examined. This toy model, despite its simplicity, exhibits the same qualitative behavior observed in more complex networks. The gradient vector becomes confined to a limited subspace, and the bulk of the Hessian comprises negligible eigenvalues.

This analytic approach shows that even slight perturbations, such as noise addition, do not disrupt the primary insight. The preserved subspace and the concentration of the gradient towards minimal eigenvectors mirror the more complex scenarios, indicating potential universality in this behavior.

Implications and Future Directions

The realization that gradient descent is confined to a minimal and stable subspace bears important implications:

• Computational Efficiency: This behavior can lead to more efficient training algorithms by emphasizing computations within this pivotal subspace, potentially reducing dimensionality concerns without compromising model performance.
• Optimization Dynamics: The findings may elucidate why deep neural networks, despite their nonconvex structures, optimize effectively. With most optimization occurring in a convex-like subspace, traditional intuitions regarding convex optimization could still apply.
• Second-Order Optimization Methods: The insights gained open the possibility of revisiting second-order optimization techniques, like Newton's method, with approximations focusing on the top Hessian subspace, leading potentially to enhanced performance.

Future research could deepen these insights by exploring the nature of this top subspace, including further mathematical characterization of eigenvectors and their evolution. Moreover, the potential modifications to training strategies inspired by these findings may usher in more efficient and robust optimization procedures in deep learning frameworks.

By highlighting the structural efficiencies inherent in the learning process, this paper proposes a crucial advancement in comprehending the dynamics of neural network optimization and offers a foundation for practical improvements in training methodologies.