Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima (1712.00779v2)

Abstract: We consider the problem of learning a one-hidden-layer neural network with non-overlapping convolutional layer and ReLU activation, i.e., $f(\mathbf{Z}, \mathbf{w}, \mathbf{a}) = \sum_j a_j\sigma(\mathbf{w}^T\mathbf{Z}_j)$, in which both the convolutional weights $\mathbf{w}$ and the output weights $\mathbf{a}$ are parameters to be learned. When the labels are the outputs from a teacher network of the same architecture with fixed weights $(\mathbf{w}^*, \mathbf{a}^*)$, we prove that with Gaussian input $\mathbf{Z}$, there is a spurious local minimizer. Surprisingly, in the presence of the spurious local minimizer, gradient descent with weight normalization from randomly initialized weights can still be proven to recover the true parameters with constant probability, which can be boosted to probability $1$ with multiple restarts. We also show that with constant probability, the same procedure could also converge to the spurious local minimum, showing that the local minimum plays a non-trivial role in the dynamics of gradient descent. Furthermore, a quantitative analysis shows that the gradient descent dynamics has two phases: it starts off slow, but converges much faster after several iterations.

• The paper demonstrates that gradient descent can recover true one-hidden-layer CNN parameters even in the presence of spurious local minima.
• It reveals a two-phase convergence process, starting with a slow initial phase followed by accelerated learning.
• The study underscores the role of random initialization and weight normalization in overcoming non-convex optimization challenges.

Gradient Descent Learning in One-hidden-layer CNNs: Challenges and Insights

The paper "Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima" by Simon S. Du et al. explores the theoretical foundations of training a one-hidden-layer convolutional neural network (CNN) using gradient descent, specifically focusing on understanding the implications of spurious local minima in non-convex optimization landscapes inherent to neural networks.

Summary

The research investigates the learnability of neural networks with a single convolutional layer using gradient descent. The authors propose a model where both convolutional weights and output weights are parameters to be learned, assuming labels are generated by a teacher network with fixed weights. The primary contribution of this paper is the demonstration that even in the presence of spurious local minima, gradient descent with random initializations and weight normalization is capable of recovering the true network parameters with a constant probability. This probability can be further increased through multiple restarts.

Numerical Results and Claims

The paper provides a theoretical analysis showing that, for a sufficiently large number of random initializations, the probability of convergence to the true parameters approaches one. Specifically, the learning dynamics exhibit two distinct phases:

1. Initial Slow Phase: The convergence rate is initially slow due to the weak signal of the randomly initialized weights.
2. Accelerated Phase: After a certain number of iterations, convergence accelerates significantly.

One striking claim of the paper is that the dynamics of gradient descent are such that it can converge to a global minimum even in landscapes with spurious local minima, contradicting intuition that such minima would impede convergence.

Implications for Practical and Theoretical Research

The findings of this research have substantial implications for both theoretical and practical aspects of machine learning. Theoretically, it challenges the commonly held belief that the presence of local minima profoundly impacts the efficiency of gradient descent in neural network training. Practically, the insights on initialization and the use of weight normalization could inform better strategies for training deep learning models. Moreover, the paper paves the way for future research into more complex networks, exploring whether these phenomena persist in deeper layers and other architectures.

Future Directions

The authors suggest several avenues for further research. Key among these is extending the analysis to deeper networks, potentially with multiple filters, to see if the observed dynamics of gradient descent and learned initialization schemes hold. Additionally, exploring conditions beyond the Gaussian input assumption could provide broader applicability to real-world scenarios. The insights from this paper may also inspire the development of novel algorithms designed to exploit the nuanced landscape of neural networks effectively.

In conclusion, this paper sheds light on the resilience of gradient descent in the face of non-convexity and offers new perspectives on initialization and layer parameterization in one-hidden-layer CNNs. It also underscores the need for a deeper understanding of neural network optimization landscapes, suggesting that the complexity of such landscapes may not be as daunting as previously thought.