Surprises in High-Dimensional Ridgeless Least Squares Interpolation (1903.08560v5)

Abstract: Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum $\ell_2$ norm ("ridgeless") interpolation in high-dimensional least squares regression. We consider two different models for the feature distribution: a linear model, where the feature vectors $x_i \in {\mathbb R}^p$ are obtained by applying a linear transform to a vector of i.i.d. entries, $x_i = \Sigma^{1/2} z_i$ (with $z_i \in {\mathbb R}^p$); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, $x_i = \varphi(W z_i)$ (with $z_i \in {\mathbb R}^d$, $W \in {\mathbb R}^{p \times d}$ a matrix of i.i.d. entries, and $\varphi$ an activation function acting componentwise on $W z_i$). We recover -- in a precise quantitative way -- several phenomena that have been observed in large-scale neural networks and kernel machines, including the "double descent" behavior of the prediction risk, and the potential benefits of overparametrization.

• The paper reveals that ridgeless least squares interpolation in high dimensions exhibits a double descent risk pattern, challenging conventional views on overparametrization.
• It demonstrates that overparametrization can reduce prediction risk by leveraging the minimum ℓ₂ norm solution, despite increased bias in isotropic settings.
• The study extends its theoretical insights to non-isotropic and random neural network models, suggesting potential universality in interpolation behaviors.

Overview of "Surprises in High-Dimensional Ridgeless Least Squares Interpolation"

The paper "Surprises in High-Dimensional Ridgeless Least Squares Interpolation" by Hastie et al. explores the intriguing behavior of interpolators in the context of high-dimensional least squares regression. The central focus is on the minimum $\ell_2$ norm or "ridgeless" interpolation, which achieves zero training error. The motivation arises from modern machine learning models, such as neural networks, which operate in a high-dimensional parameter space and often exhibit similar interpolation behavior.

Key Models and Methodological Approach

The researchers explore two models for the feature distribution:

1. Linear Model: Here, feature vectors are derived by applying a linear transformation to vectors with i.i.d. entries. This model is formulated as $x_i = \Sigma^{1/2} z_i$.
2. Nonlinear Model: In this model, feature vectors pass through a one-layer random neural network: $x_i = \varphi(W z_i)$.

The paper seeks to understand phenomena previously observed in large-scale neural networks such as "double descent" in prediction risk and the impact of overparametrization.

Significant Results

1. Prediction Risk Behaviors:
   • Double Descent: The "double descent" pattern of risk, wherein risk decreases with increasing model complexity after an initial rise, is demonstrated in ridgeless regression. This is particularly significant in the overparametrized regime (where the number of parameters exceeds the number of observations, $p > n$).

• Overparametrization: The paper quantitatively specifies how overparametrization can sometimes reduce prediction risk, challenging conventional beliefs around the negative implications of interpolation.

1. Model-Specific Insights:
   • In isotropic settings where features have independent entries, risk calculations reveal that the bias increases with overparametrization while variance decreases due to the minimum norm solution's inherent regularization effect.
2. Theoretical Contributions:
   • The analysis extends to cases where the feature covariance matrix $\Sigma$ has structure (non-isotropic cases), providing detailed risk approximations and conditions under which interpolation optimally occurs.
   • The paper makes conjectures and provides preliminary evidence towards universality, suggesting that the behaviors are consistent across a variety of distributional settings for the features.

Implications and Future Directions

The results underscore the nuanced nature of interpolation in machine learning models, providing a richer understanding of generalization in high-dimensional settings. Practically, these insights can inform the design of neural networks and feature representations to harness the benefits of overparameterization. Theoretical advancements suggest a potential for universality in results, indicating that observed phenomena may transcend specific model architectures.

Future research could expand on verifying universality rigorously across different architectures and feature generation processes. Moreover, the implications for model selection, particularly regarding the harmonic balance between regularization and interpolation, could revolutionize training paradigms across supervised learning tasks.

In essence, this work catalyzes a deeper investigation into the paradox of interpolation versus generalization, challenging entrenched paradigms of model complexity in machine learning.