NGD converges to less degenerate solutions than SGD (2409.04913v2)

Abstract: The number of free parameters, or dimension, of a model is a straightforward way to measure its complexity: a model with more parameters can encode more information. However, this is not an accurate measure of complexity: models capable of memorizing their training data often generalize well despite their high dimension. Effective dimension aims to more directly capture the complexity of a model by counting only the number of parameters required to represent the functionality of the model. Singular learning theory (SLT) proposes the learning coefficient $ \lambda $ as a more accurate measure of effective dimension. By describing the rate of increase of the volume of the region of parameter space around a local minimum with respect to loss, $ \lambda $ incorporates information from higher-order terms. We compare $ \lambda $ of models trained using natural gradient descent (NGD) and stochastic gradient descent (SGD), and find that those trained with NGD consistently have a higher effective dimension for both of our methods: the Hessian trace $ \text{Tr}(\mathbf{H}) $, and the estimate of the local learning coefficient (LLC) $ \hat{\lambda}(w^*) $.

• The paper demonstrates that NGD yields models with higher effective dimensions than SGD, as revealed by LLC estimates and Hessian trace measurements.
• It employs feed-forward and convolutional neural networks on MNIST datasets to compare NGD and SGD, highlighting the role of Fisher matrix smoothing.
• The findings suggest that NGD's ability to escape degenerate regions can enhance model generalization and optimize deep learning performance.

Overview of "NGD converges to less degenerate solutions than SGD"

The paper "NGD converges to less degenerate solutions than SGD," authored by Moosa Saghir, N. R. Raghavendra, Zihe Liu, and Evan Ryan Gunter, investigates the effective complexity of models trained using Natural Gradient Descent (NGD) compared to those trained with Stochastic Gradient Descent (SGD). By leveraging Singular Learning Theory (SLT), the authors propose and empirically validate that NGD results in higher effective dimensions, implying less degenerate solutions, than SGD.

Effective dimension, as encountered in this paper, aims to overcome the limitations of the nominal parameter count by only considering the degrees of freedom essential for the model's functionality. The local learning coefficient (LLC) $\lambda$, derived from SLT, offers an accurate measure of this effective dimension. The authors measure the LLC of models trained using NGD and SGD, providing compelling evidence that NGD achieves a higher effective dimension using both the Hessian trace $\text{Tr}(\mathbf{H})$ and an estimate of $\lambda(w^*)$.

Key Findings

1. Higher Effective Dimension with NGD:
   • Models trained using NGD consistently achieve a higher effective dimension compared to those trained with SGD.
   • Across various model architectures and training setups, NGD-trained models demonstrated higher $\hat{\lambda}$ and $\text{Tr}(\mathbf{H})$ values.
2. Impact of Fisher Matrix Smoothing:
   • Reducing the smoothing in the Fisher Information Matrix $\mathbf{F}$ during NGD increases the LLC. Higher smoothing makes NGD behave more like SGD, decreasing $\lambda$.
3. Escape from Degenerate Minima:
   • NGD shows a tendency to escape high degeneracy regions in the loss landscape. Upon switching from SGD to NGD mid-training, models showed increased $\hat{\lambda}$ signifying less degenerate solutions.

Methodology

The paper extensively employs feed-forward neural networks (FFNN) and convolutional neural networks (CNN) modeled after the Lenet-5 architecture to benchmark both the MNIST and Fashion MNIST datasets. The experiments included training models independently with SGD and NGD, varying the smoothness constants for NGD, and evaluating the impact of switching from SGD to NGD mid-training. The authors tracked training and validation loss, update norms, and estimated LLC $\hat{\lambda}(w^*)$ using a stochastic gradient LLC estimator.

To compute the Hessian trace without explicit matrix construction, a Hessian-vector product method is utilized, leveraging the PyHessian library to facilitate the necessary computations efficiently.

Implications and Future Directions

The findings have significant implications for optimization in deep learning:

• Model Selection: The results suggest that using NGD could result in models with more generalizable features due to the higher effective dimension, potentially reducing the risk of overfitting.
• Theoretical Foundation: These results provide empirical support for SLT's proposition of $\lambda$ as a measure of complexity, reinforcing its utility in practical deep learning scenarios.
• Algorithm Development: Further refinement in NGD algorithms, particularly in controlling the Fisher matrix's properties, could lead to even more efficient and effective optimization techniques.

Future research could explore:

• Extending these experiments to more complex and deeper neural networks.
• Exploring the impact of NGD on other optimization problems, especially those involving highly non-convex loss landscapes.
• Investigating the robustness of NGD across various datasets and real-world applications to understand its broader applicability.

Conclusion

The research presented in "NGD converges to less degenerate solutions than SGD" provides valuable insights into the comparative advantages of NGD over SGD in terms of converging to less degenerate solutions. The authors' rigorous methodology and clear presentation of results underscore the relevance of SLT in modern machine learning contexts, and their contributions pave the way for potential enhancements in training algorithms that prioritize both speed and robustness in finding less singular, more effective model parameters.