Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (1706.04454v3)

Abstract: We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the flatness of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create large connected components at the bottom of the landscape. Second, the dependence of small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light into the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

• The paper characterizes the Hessian's spectrum, showing a central bulk of near-zero eigenvalues alongside distinct outliers.
• The methodology links over-parameterization to bulk scaling while demonstrating how data complexity drives the formation of outlier eigenvalues.
• The study challenges traditional optimization views by revealing that GD and SGD traverse interconnected, flat loss regions, inspiring new exploration strategies.

An Empirical Analysis of the Hessian in Over-Parameterized Neural Networks

The paper "Empirical Analysis of the Hessian of Over-Parametrized Neural Networks" presents a detailed exploration of the geometry of loss surfaces in deep learning models through their Hessian matrices. The authors focus particularly on how the Hessian's spectrum reveals critical insights into the behavior and performance of over-parameterized models.

A primary insight from the paper is the characterization of the Hessian's spectrum, which consistently displays a distinctive structure: a central bulk near zero, coupled with outliers significantly detached from this bulk. This phenomenon appears robust across experiments involving increasing model parameter counts and altering data structures. Such patterns underscore that increasing parameters primarily scales the bulk, while data complexity primarily affects the outliers. Results are substantiated through empirical evidence and theoretical justification, pointing to potential shifts in non-convex optimization discussions.

The analysis proposes two fundamental implications for the understating of neural network loss landscapes. Firstly, the observed flatness is attributed to the abundance of near-zero eigenvalues, suggesting traditional notions of basins of attraction in optimization might be misleadingly simplistic. The authors argue these large, flat regions indicate an interconnectedness at the bottom of the loss landscape, independent of model complexity. Secondly, the dependence of outliers in the spectrum on data distribution opens avenues to reevaluate the interactions in the data-architecture-algorithm framework, specifically highlighting connections with the covariance matrix of model gradients.

The paper's findings suggest notable considerations for optimization strategies. The convergence behaviors of gradient descent methods (GD) versus stochastic gradient descent (SGD), surprisingly, do not diverge towards distinct solution quality or nature; instead, deeper connections exist within the same geometric basin of attraction. This challenges preconceptions about the barriers that supposedly separate solutions found by different methodologies, such as small-batch and large-batch SGD. The numerical results reveal that despite differing in generalization performance, solutions can still be interconnected through these flat regions, implying that exploration of level sets may yield solutions with enhanced generalization.

The intrinsic geometry highlighted by this analysis indicates that within over-parameterized networks, flatness arises from both over-parameterization and the composition of the datasets. The decomposition of the Hessian into the sum of a covariance matrix of gradients and a second term amplifies this notion. It’s further proposed that optimization methods should pivot from seeking narrow basins to traversing and exploring these wide, flat regions to harness potential benefits in generalization.

As networks scale in complexities unprecedentedly, understanding the geometric structures like those investigated in this work becomes invaluable. The paper sparks contemplation on high-dimensional loss landscape navigation, suggesting that future research might focus on pathways and algorithms uniquely optimized for these inherently flat terrains, aligning both practical optimization strategies and theoretical explorations within this modern context.