Convergence theory for non-convex deep network training

Establish a rigorous theoretical framework describing the convergence behavior of gradient-based optimization methods (including stochastic gradient descent) when minimizing the non-convex empirical risk E(θ) associated with deep neural networks, specifying conditions under which convergence is guaranteed and characterizing the nature of limiting points.

Background

The paper contrasts the well-understood convex case (where fθ depends linearly on θ) with the deep neural network setting, where the empirical risk E(θ) is non-convex. While convex cases admit comprehensive convergence theories, deep networks pose significant analytical challenges due to non-convexity.

The authors explicitly note that the theoretical analysis of deep network optimization remains largely unresolved, motivating the need for a rigorous convergence theory tailored to the non-convex landscape encountered in practice.

References

However, in the case of deep neural networks, $E(\theta)$ is non-convex, making its theoretical analysis challenging and still largely unresolved.

— The Mathematics of Artificial Intelligence  (2501.10465 - Peyré, 15 Jan 2025) in Section “Supervised Learning”, Empirical Risk Minimization paragraph

Finally, characterizing convergence guarantees under realistic settings remains an open problem to study.

— SoftServe: A Scalable Quasi-Newton Method for Deep Learning  (2610.02182 - Ko et al., 1 Oct 2026) in Section 7, Discussion and Limitations, p. 9

However, since the fairness term is non-convex, it is unclear whether the resulting algorithm converges to the underlying optimum, and it also entails complicated hyperparameter tuning for trading off accuracy and fairness.

— FairProp: Fair Node Representation Learning via Differentiable Propagation Layers  (2610.06484 - Kariotakis et al., 5 Oct 2026) in Section 1.1, Related Work, p. 2

We do not prove convergence of the stochastic optimizer or a $\tau\downarrow0$ limit for the trained scheme.

— A Neural JKO Scheme for Hellinger-Kantorovich Gradient Flows via Monge-Growth Pairs  (2610.07602 - Seo et al., 6 Oct 2026) in Section 1, subsection 'Scope and organization'; Section 'Discussion and conclusion'

In d\ge2 dimensions, hidden-weight directions can rotate during training, changing activation boundaries and requiring a geometric characterization of active cones to extend our convergence and max-margin arguments. Training the output layer, increasing width, and adding depth introduce further interactions beyond the current analysis. Although our MNIST experiments with $16$ hidden units and a trainable output layer exhibit similar switching and loss decay, extending the theory to these settings remains open.

— A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability  (2610.23007 - Nguyen et al., 16 Oct 2026) in Section 7, paragraph “Limitations and future work”