Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers (1811.04918v6)

Abstract: The fundamental learning theory behind neural networks remains largely open. What classes of functions can neural networks actually learn? Why doesn't the trained network overfit when it is overparameterized? In this work, we prove that overparameterized neural networks can learn some notable concept classes, including two and three-layer networks with fewer parameters and smooth activations. Moreover, the learning can be simply done by SGD (stochastic gradient descent) or its variants in polynomial time using polynomially many samples. The sample complexity can also be almost independent of the number of parameters in the network. On the technique side, our analysis goes beyond the so-called NTK (neural tangent kernel) linearization of neural networks in prior works. We establish a new notion of quadratic approximation of the neural network (that can be viewed as a second-order variant of NTK), and connect it to the SGD theory of escaping saddle points.

• The paper demonstrates that overparameterized two- and three-layer neural networks can efficiently learn smooth concept classes using Stochastic Gradient Descent.
• It reveals that overparameterization allows networks to discover complex functions while maintaining robust generalization, challenging traditional overfitting concerns.
• The study introduces a novel quadratic NTK approximation, offering deeper insights into layer interactions and a benign optimization landscape.

Analyzing Learning in Overparameterized Neural Networks

The paper provides a comprehensive analysis of learning in overparameterized neural networks, focusing on both two-layer and three-layer architectures. It aims to contribute to the theoretical understanding of neural networks, specifically in terms of what can be learned and how overparameterization affects learning and generalization. The work presents rigorous theoretical results, which are germane to the agnostic PAC-learning framework, thereby making these results independent of data distribution characteristics.

The primary theoretical contribution lies in proving that certain concept classes—originating from two-layer and three-layer neural networks with smooth activation functions—can be efficiently learned using overparameterized neural networks with the ReLU activation. The results are achieved through a nuanced application of Stochastic Gradient Descent (SGD) and its variants.

Key Contributions

1. Concept Class Learnability: The paper demonstrates that two-layer neural networks can learn concept classes corresponding to smaller, smooth-activated two-layer networks. Similarly, three-layer neural networks can learn functions representable by three-layer networks with smooth activations. A significant aspect of this result is the efficiency of the learning process, in terms of both computational resources and sample complexity.
2. Explaining Overparameterization: By rigorously analyzing overparameterization, the authors reveal how larger networks can discover more complex functions while still generalizing well on unseen data. This result contradicts traditional views tied to the problem of overfitting, offering a clearer rationale for the observed empirical success of overparameterized networks in practice.
3. Theoretical Framework: The research goes beyond NTK (Neural Tangent Kernel) linearization, introducing a novel quadratic approximation that allows the capture of interactions across different layers in a way NTK does not. This quadratic approximation serves as a second-order NTK variant, pivotal for understanding the deeper interactions within neural networks.
4. Optimization and Generalization: The authors provide a detailed optimization landscape analysis, leveraging the structure of overparameterized networks to argue that there are no spurious local minima or harmful saddle points, resulting in a benign landscape that fosters efficient learning. Moreover, a sophisticated understanding of how implicit and explicit regularization leads to effective generalization is developed.

Theoretical Implications

The results suggest that overparameterized neural networks are capable of representing a wide range of complex functions with robust generalization capabilities, even when the networks have many more parameters than the number of training examples. This insight challenges traditional views of machine learning complexity, specifically the emphasis on VC dimensions. The theory underscores that the landscape induced by overparameterization is fundamental to achieving both trainability and generalization, informing the ongoing discourse on neural network training dynamics.

Practical Implications

On a practical level, the paper offers valuable insights into algorithm design for training deep networks. By uncovering that certain concept classes—often unachievable with simpler learners—are within reach for overparameterized networks, practitioners can have more confidence in deploying larger networks and understanding when and how they may fail. Additionally, this work invites practitioners to consider network size as a lever not just for capacity but for learning efficiency and capability exploration.

Future Directions

In terms of future work, the authors suggest extending these theoretical insights to deeper architectures beyond three layers and considering more sophisticated non-linearities, such as the ReLU. They indicate that the current framework could also be extended to recurrent neural networks and other domain-specific architectures, which may share similar overparameterization characteristics.

This paper sets a solid foundation for further inquiry into the unique capabilities of overparameterized networks, reinforcing the theoretical underpinnings that support their empirical performance and elucidating key factors that contribute to their success in practice.