Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs

Abstract: Deep learning models are often successfully trained using gradient descent, despite the worst case hardness of the underlying non-convex optimization problem. The key question is then under what conditions can one prove that optimization will succeed. Here we provide a strong result of this kind. We consider a neural net with one hidden layer and a convolutional structure with no overlap and a ReLU activation function. For this architecture we show that learning is NP-complete in the general case, but that when the input distribution is Gaussian, gradient descent converges to the global optimum in polynomial time. To the best of our knowledge, this is the first global optimality guarantee of gradient descent on a convolutional neural network with ReLU activations.

• The paper shows that for a no-overlap ConvNet with a single hidden layer, gradient descent provably converges to the unique global optimum under IID Gaussian inputs.
• It leverages closed-form loss and gradient expressions to characterize a benign loss landscape that circumvents typical NP-hard challenges in neural network training.
• The study reveals that while overlapping filters introduce suboptimal traps, empirical evidence supports the effectiveness of repeated random restarts to achieve optimal convergence.

Globally Optimal Gradient Descent for ConvNets with Gaussian Inputs

Problem Statement and Motivation

The optimization landscape for deep neural networks, especially those incorporating convolution layers and non-linear activations, remains challenging to analyze theoretically. Empirical success with gradient descent stands in stark contrast to strong negative results, such as NP-hardness in training even shallow neural networks with certain activations or even small architectures. This work addresses the conditions under which global optimality of gradient descent is attainable, focusing on a single-hidden-layer convolutional network with non-overlapping filters and ReLU activations—a class of models closely related to practical ConvNet architectures.

The central question is: Under which input distributions does gradient descent provably avoid suboptimal critical points and reach the global optimum in polynomial time for structured, non-linear architectures?

Network Model and Setting

The considered network is a one-hidden-layer ConvNet with non-overlapping filters, ReLU activation, and average pooling. Specifically, for input $x \in \mathbb{R}^d$, the vector is partitioned into $k$ non-overlapping blocks of size $m = d / k$. A filter $w \in \mathbb{R}^m$ is applied to each block, the responses are passed through a ReLU, and the outputs are averaged. The population objective (infinite data) is the mean squared error between the learned model and the true underlying network, both parameterized by a filter $w^*$.

Two main learning settings are analyzed:

1. Worst-case Instance: Arbitrary distribution over the input–output pairs.
2. Structured Case: IID Gaussian inputs.

Complexity Analysis: Hardness of No-Overlap ConvNets

A fundamental result established is that learning no-overlap ConvNets is NP-complete in the realizable case for unconstrained distributions. The authors provide a reduction from the Set-Splitting-by-$k$-Sets problem, itself shown to be NP-complete, to the learning problem with $k$ hidden units. They define the k-Non-Overlap-Opt problem: finding $w$ achieving population risk within an absolute error tolerance of the global minimum. The reduction demonstrates that, even for networks with tied parameters and simple non-overlapping structure, the learning task remains intractable unless the data distribution is restricted.

Tractability under Gaussian Inputs: Exact Global Optimality

When the input distribution is zero-mean, unit-variance IID Gaussian, the analysis reveals a remarkable transition: gradient descent initialized randomly avoids all non-optimal critical points and converges to the unique global optimum in polynomial time.

Key technical ingredients include:

• Closed-form loss and gradient: For Gaussian inputs, population risk can be expressed in terms of filter norm, alignment with the true filter, and standard kernel integrals from the literature on neural tangent kernels.
• Critical points characterization: The population loss—despite being non-convex and non-differentiable at zero—has only three critical points for $k > 1$: a local maximum at the origin, a unique global minimum at $w = w^*$, and a degenerate saddle at a negative multiple of $w^*$.
• Convergence guarantee: With high probability over random initialization, the iterates of gradient descent stay away from the degenerate points and exploit the geometric structure of the loss to monotonically align with $w^*$. The algorithm reaches $\epsilon$-accurate solutions in $O(1/\epsilon^2)$ steps, matching the best-known results for smooth non-convex problems in terms of rate.

Notably, this is the first global convergence guarantee for gradient-based learning of a ReLU ConvNet architecture without over-parameterization or non-standard activation functions.

Empirical Validation: Tractability Gap

The theoretical tractability gap is verified empirically by comparing gradient descent (Adagrad) optimization on two datasets labeled by the same ConvNet but with different input distributions:

• Set-splitting-based inputs (worst case): Optimization fails, confirming the predicted intractability.
• IID Gaussian inputs: Gradient descent reliably achieves the global optimum, consistent with the theory.

This empirical evidence substantiates the claim that the data distribution is instrumental in determining the optimization landscape's benignity.

Networks With Overlapping Filters

The strict tractability for no-overlap filters does not generalize to ConvNets with overlapping filters (strides smaller than the filter size). The analysis shows that, even under Gaussian inputs in $\mathbb{R}^2$, gradient descent can become trapped in suboptimal regions with probability at least $1/4$ if randomly initialized. Explicit lower bounds are established for the achievable risk in these suboptimal regimes, and detailed loss surface visualizations confirm the complex geometry introduced by overlaps.

However, empirical results suggest that repeated random restarts of gradient descent are effective: the basin of attraction of the global optimum is lower bounded away from zero (at least $1/17$ for parameter ranges studied). This hints at practical tractability for moderate retries, but theoretical guarantees for general architectures and higher overlap remain open.

Theoretical and Practical Implications

Theoretical

• Distribution-dependent tractability: This work provides a concrete instance where the choice of data distribution shifts a fundamentally hard learning problem into the field of global optimization by first-order methods.
• Loss geometry and activation type: The ReLU activation, combined with convolutional structure and data symmetry, leads to loss surfaces with minimal critical points; but small architectural changes (overlap) immediately break this simplicity.
• Bridges to kernel methods: The Gaussian input assumption allows leveraging closed-form kernel analysis, indicating potential for deeper connections between neural network optimization and classical statistical theory.

Practical

• Architectural recommendations: For certain tasks with predominantly Gaussian-like data, no-overlap convolutional structures may be amenable to efficient training without risk of suboptimality.
• Initialization strategies: While overlapping filter architectures can in principle be globally trained, proper randomization and potentially multiple restarts are required to avoid sub-optimal traps.
• Algorithmic design: Results motivate algorithmic techniques (e.g., restart schedules) and call for further research into data-dependent design of neural architectures and their training.

Open Directions

• Extension of global convergence results to overlapping filters, other pooling operations (e.g., max), and non-Gaussian but structured input distributions (e.g., log-concave, sub-Gaussian).
• Population-to-empirical risk transfer: formalization of conditions under which finite-sample empirical optimization inherits the benign geometry observed at population risk.
• Broader characterization of distributional and architectural regimes for globally optimal deep learning.

Conclusion

This work establishes that global optimality of gradient descent for learning well-structured convolutional ReLU networks is possible under specific input distributions—specifically, multivariate Gaussians—highlighting a sharp complexity transition dependent on the data-generating process. The results rigorously delineate the boundary between intractability and efficient optimizability, providing new insights into why and when deep learning works in practice and posing challenging theoretical questions for the extension of these results to more general settings.

Reference:

"Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs" (1702.07966)