Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

Abstract: In the past decade the mathematical theory of machine learning has lagged far behind the triumphs of deep neural networks on practical challenges. However, the gap between theory and practice is gradually starting to close. In this paper I will attempt to assemble some pieces of the remarkable and still incomplete mathematical mosaic emerging from the efforts to understand the foundations of deep learning. The two key themes will be interpolation, and its sibling, over-parameterization. Interpolation corresponds to fitting data, even noisy data, exactly. Over-parameterization enables interpolation and provides flexibility to select a right interpolating model. As we will see, just as a physical prism separates colors mixed within a ray of light, the figurative prism of interpolation helps to disentangle generalization and optimization properties within the complex picture of modern Machine Learning. This article is written with belief and hope that clearer understanding of these issues brings us a step closer toward a general theory of deep learning and machine learning.

• The paper shows that interpolation enabled by over-parameterization allows models to fit data exactly without causing overfitting.
• It contrasts classical under-parameterized regimes with modern over-parameterized methods, highlighting phenomena like double descent.
• Empirical findings reveal that gradient-based optimization in non-convex settings converges to global minima despite exact training fits.

A Mathematical Perspective on Deep Learning: Interpolation and Over-Parameterization

The paper "Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation" by Mikhail Belkin provides a comprehensive discussion on how interpolation and over-parameterization fundamentally shape our understanding of modern deep learning. This paper addresses a crucial gap between the empirical successes of deep learning and the theoretical frameworks developed in earlier decades, proposing a perspective shift from traditional views.

Interpolation and Over-Parameterization

A central thesis of the paper is the need to consider interpolation and over-parameterization as key phenomena that illuminate the workings of deep learning. Interpolation, the practice of fitting data exactly, even when it's noisy, becomes feasible with over-parameterization, where the parameter space is large enough to accommodate complex datasets. Contrary to classical statistical intuition, Belkin argues that interpolation does not inherently lead to overfitting, ushering in a conceptual departure from traditional theories that discourage exact fits due to fears of overfitting.

Classical vs. Modern Regimes

Belkin delineates two regimes in machine learning: classical under-parameterized regimes and modern over-parameterized ones. Classical regimes rely on notions such as Uniform Laws of Large Numbers and capacity control to manage the trade-off between fitting training data and generalizing to new inputs. In contrast, the modern regime leverages the flexibility afforded by over-parameterization to identify interpolating solutions that generalize well, despite fitting the training data exactly. The paper points to a fundamental shift, highlighting how traditional theories based on statistical learning face limitations when applied to understanding modern deep learning architectures.

Empirical Evidence and Theoretical Challenges

The paper reviews empirical evidence that challenges longstanding beliefs, such as zero-error training not equating to poor generalization. It describes experiments showing that neural networks can interpolate noisy data without degradation in test performance, directly conflicting with classical WYSIWYG (What You See Is What You Get) bounds that assume empirical risk closely reflects expected risk.

The Prism of Interpolation

Interpolation serves as a prism that reveals different facets of machine learning's landscape, including how generalization, optimization, and noise robustness interplay in complex models. Belkin introduces important concepts such as the double descent phenomenon, which revises the classical U-shaped generalization curve with a more nuanced understanding that includes a second descent after an interpolation peak due to increased model capacity.

Optimization in Over-Parameterized Systems

The paper discusses the implications of over-parameterization for optimization, moving past the notion of convexity. It highlights the effectiveness of gradient-based optimization methods like stochastic gradient descent (SGD) in non-convex, over-parameterized settings, attributed to properties such as the Polyak-Łojasiewicz condition, which ensures convergence to global minima.

Speculations and Future Directions

Belkin speculates on the potential pathways for a comprehensive theory of machine learning, one that accounts for algorithmic inductive biases and expands beyond current paradigms. He suggests that large neural networks might essentially act as kernel learners, with concepts like the Neural Tangent Kernel (NTK) opening promising paths for future exploration. This perspective urges researchers to reconsider the role of classical structures like norms and smoothness as guiding principles within the manifold of interpolating solutions.

Conclusion

Overall, this paper contributes significantly to the discourse on the theoretical underpinnings of deep learning. By focusing on interpolation and over-parameterization, Belkin invites further inquiry into these mathematical phenomena's roles in shaping the field's future directions. Researchers are encouraged to construct frameworks that reconcile the astounding empirical performance of deep learning models with robust mathematical foundations, fostering a more precise understanding of their capabilities and limits.