A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning (2109.02355v1)

Abstract: The rapid recent progress in ML has raised a number of scientific questions that challenge the longstanding dogma of the field. One of the most important riddles is the good empirical generalization of overparameterized models. Overparameterized models are excessively complex with respect to the size of the training dataset, which results in them perfectly fitting (i.e., interpolating) the training data, which is usually noisy. Such interpolation of noisy data is traditionally associated with detrimental overfitting, and yet a wide range of interpolating models -- from simple linear models to deep neural networks -- have recently been observed to generalize extremely well on fresh test data. Indeed, the recently discovered double descent phenomenon has revealed that highly overparameterized models often improve over the best underparameterized model in test performance. Understanding learning in this overparameterized regime requires new theory and foundational empirical studies, even for the simplest case of the linear model. The underpinnings of this understanding have been laid in very recent analyses of overparameterized linear regression and related statistical learning tasks, which resulted in precise analytic characterizations of double descent. This paper provides a succinct overview of this emerging theory of overparameterized ML (henceforth abbreviated as TOPML) that explains these recent findings through a statistical signal processing perspective. We emphasize the unique aspects that define the TOPML research area as a subfield of modern ML theory and outline interesting open questions that remain.

• The paper provides a comprehensive theory of overparameterized models that achieve benign overfitting and a distinctive double descent behavior.
• It employs analytical frameworks, notably the minimum ℓ2-norm interpolator, to decompose test error into signal and noise components.
• Its insights guide practical model design for deep learning and extend to unsupervised, semi-supervised, and transfer learning applications.

An Overview of the Theory of Overparameterized Machine Learning

The paper "A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning" provides a comprehensive survey of emerging theories in the domain of overparameterized machine learning (TOPML), focusing on understanding why highly complex models often generalize well despite perfectly fitting noisy data. This discussion challenges the classical bias-variance tradeoff, commonly used to explain generalization in underparameterized regimes, by detailing the phenomenon of double descent observed in overparameterized models.

Overparameterized models possess significantly more parameters than training data points and can interpolate noisy training data. Contrary to traditional beliefs, such interpolation does not necessarily result in overfitting, and may lead to improved generalization, exemplified by the double descent curve, which demonstrates a decrease in test error with increased model complexity beyond a critical point. This paper highlights the unique traits and open questions of the emerging TOPML field through a signal processing lens with substantial analytical characterization and experimental verifications.

Key Findings and Theoretical Insights

Recent research has provided analytical frameworks for understanding TOPML behaviors, focusing on the minimum $\ell_2$-norm interpolator, which reveals critical insights through mathematical characterizations of test error decomposition. These analyses suggest that beneficial generalization arises from a tradeoff between two terms:

1. A signal-specific term related to fitting the underlying data structure.
2. A noise-specific term quantifying the impact of interpolating noise.

Minimal harm from noise through interpolation is explained by high effective overparameterization, particularly in high-dimensional spaces where feature covariance demonstrates anisotropic properties. The requirement of low effective dimensionality in data and alignment of data variance with high-energy directions are critical for benign overfitting.

Practical Implications and Future Directions

The revelations concerning benign overfitting and double descent in overparameterized regimes have practical implications for designing and tuning machine learning models. The existence of double descent in classification tasks parallel to regression tasks emphasizes the dense interplay between signal recovery and noise accommodation in achieving good generalization.

While theoretical analyses draw primarily on linear models, implications for complex structures, including deep neural networks, remain imperative. Deep learning practitioners are increasingly considering these findings to optimize architectures and training processes, aligning computational practices with emerging theoretical guidance.

The paper's insights broaden beyond typical supervised learning to encompass unsupervised and semi-supervised frameworks like PCA-based subspace learning, data generation using GANs, transfer learning, pruning, and dictionary learning. These applications illustrate overparameterization's potential benefits in diverse machine learning contexts, suggesting directions for further investigations into TOPML.

Open Challenges

Addressing foundational questions in TOPML theory remains crucial, chiefly regarding defining model complexity in overparameterized settings. Determining effective complexity beyond parameter count—a reducible model of complexity adaptable to dependencies and algorithmic regularities—is a topic of ongoing research.

Lastly, understanding how insights from overparameterized learning translate into applications outside ML is yet to be fully explored. Acknowledging the novel potential for interpolating solutions in fields like signal processing and statistical estimation may lead to further breakthroughs.

This paper acts as a catalyst for dialogue among researchers to refine the evolving theory of overparameterized machine learning, advancing towards a more unified understanding of generalization in the modern machine learning landscape.