The Loss Surfaces of Multilayer Networks (1412.0233v3)

Abstract: We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.

• The paper demonstrates that large multilayer networks exhibit a low-loss band where SGD converges to high-quality local minima.
• It establishes a theoretical bridge between neural network loss functions and spherical spin-glass Hamiltonians, validated by empirical experiments.
• The findings imply that scaling network size enhances optimization reliability and generalization by promoting redundant, effective parameter configurations.

This paper (The Loss Surfaces of Multilayer Networks, 2014) investigates the complex, non-convex loss surfaces encountered when training multilayer neural networks. It offers a theoretical explanation for the empirical observation that while these networks have numerous local minima, standard optimization methods like Stochastic Gradient Descent (SGD) consistently find solutions that perform well on unseen data, particularly for large networks. The paper establishes a novel connection between the loss function of a simplified neural network model and the Hamiltonian of a spherical spin-glass model, leveraging results from random matrix theory to analyze the landscape of critical points.

The core idea is that for large networks, the geometry of the loss surface fundamentally changes compared to small networks. While small networks may have "bad" local minima (high loss values) that optimizers can get stuck in, large networks exhibit a different structure. Under several simplifying assumptions, the authors show that the loss function resembles that of a spherical spin-glass model. These assumptions include treating inputs and path activations as independent random variables, imposing a spherical constraint on weights, and assuming redundancy and uniformity in network parameters.

Based on this connection, the paper draws upon theoretical results regarding the critical points of spherical spin glasses [AAC2010]. For large-size networks (characterized by a large parameter $\Lambda$, related to network mass and size), the theoretical analysis predicts the following properties of the loss landscape:

1. Band of Low-Index Critical Points: The vast majority of low-index critical points (local minima and low-index saddle points) are concentrated within a specific band of loss values. Critical points with loss values significantly higher than this band are exponentially likely to be high-index saddle points.
2. Layered Structure: Within this low-loss band, critical points form a layered structure based on their index (the number of negative eigenvalues of the Hessian). The lowest band contains only local minima (index 0). Higher bands contain saddle points of increasing index.
3. Dominance of Local Minima: The number of critical points within the low-loss band grows exponentially with network size, and local minima dominate exponentially over saddle points within this band.

These theoretical findings suggest that for large networks, gradient descent methods like SGD, which are designed to avoid saddle points and descend towards minima, are overwhelmingly likely to converge to one of the many "good" local minima within the low-loss band, rather than getting trapped in high-loss saddle points or poor local minima far from the global optimum.

The paper provides empirical validation for these hypotheses using simulations of a spin-glass model and experiments on a scaled-down MNIST dataset with single-hidden-layer neural networks.

• Spin-Glass and Neural Network Loss Distributions: Experiments show that for both the spin-glass model and the neural network, the distribution of converged loss values changes with size. For small sizes ($\Lambda$ or number of hidden units $n_1$), the distribution is wider, with some solutions converging to higher loss values ("bad" minima). As size increases, the distribution becomes increasingly concentrated around lower loss values, qualitatively matching the theoretical prediction of critical points converging to a low-loss band.
• Index Analysis: The authors compute the Hessian index for converged solutions of the neural network and find that they are overwhelmingly local minima or saddle points with a very low proportion of negative eigenvalues, confirming that SGD finds low-index critical points.
• SGD vs. Simulated Annealing: A comparison between SGD and Simulated Annealing (SA) on a subset of MNIST showed that SGD performed at least as well as SA. Since SA does not rely on gradients and is less prone to getting stuck in high-index saddle points, this further supports the idea that SGD successfully navigates the landscape to find good solutions.
• Redundancy and Generalization: An experiment with simulated annealing where 95% of weights were redundant (quantized to 3 values) showed only a small loss in accuracy, supporting the redundancy assumption used in the theoretical model. Furthermore, the correlation between training loss and test loss was analyzed. As network size increases, this correlation decreases (Table 5.1). This implies that improving training loss further (e.g., by finding the absolute global minimum) is less relevant for improving generalization on the test set for larger networks.

Practical Implications for Implementation:

The paper does not propose a new optimization algorithm but provides a theoretical justification for the effectiveness of existing methods like SGD for training large deep networks.

• Why SGD Works: The benign structure of the loss landscape at low energy levels for large networks explains why simple gradient-based methods can find good solutions without needing complex strategies to escape high-loss local minima or saddle points.
• Scaling Matters: The results strongly suggest that the size of the network is a crucial factor in the ease of optimization and the quality of found solutions. Larger networks are theoretically and empirically easier to train effectively in terms of finding low-loss solutions.
• Global Minimum Irrelevance: The theoretical result that the global minimum becomes increasingly hard to reach with increasing size, combined with the empirical observation that training and test loss decorrelate, supports the practice of stopping training based on validation performance rather than striving for the absolute minimum on the training set. Overfitting is framed as a potential consequence of trying too hard to reach the global minimum training solution.
• Computational Considerations: While the theory explains why large networks are optimizable, it doesn't reduce the computational cost of large networks themselves. Training large networks still requires significant resources for gradient calculation and parameter updates.
• Limitations: The theoretical model makes strong assumptions (independence, spherical constraint) that do not perfectly reflect real neural networks. However, the qualitative agreement between the spin-glass model and empirical neural network results suggests that the core insights regarding the landscape structure for large systems may be robust to some deviations from these ideal conditions.

In summary, the paper provides a theoretical framework, grounded in statistical physics and random matrix theory, to understand the optimization landscape of deep neural networks. It posits that for large networks, the landscape contains a vast number of high-quality local minima, making it easier for methods like SGD to find effective solutions compared to smaller networks where poor local minima might be more prevalent. This work contributes to explaining the empirical success of deep learning in regimes with many parameters.