Learning in High Dimension Always Amounts to Extrapolation (2110.09485v2)

Abstract: The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample $x$ whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when $x$ falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to correctly interpolate training data. A second (mis)conception is that interpolation happens throughout tasks and datasets, in fact, many intuitions and theories rely on that assumption. We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional ($>$100) dataset, interpolation almost surely never happens. Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances.

• The paper reveals that interpolation is nearly impossible in high-dimensional spaces unless dataset size grows exponentially, indicating that models mostly extrapolate.
• It combines theoretical proofs with empirical evidence from MNIST, CIFAR10, and Imagenet to illustrate that new samples typically lie outside the training convex hull.
• The findings challenge traditional definitions of interpolation and urge the development of more robust, extrapolation-focused methods for high-dimensional learning.

Analysis of "Learning in High Dimension Always Amounts to Extrapolation"

In their paper titled "Learning in High Dimension Always Amounts to Extrapolation," Balestriero, Pesenti, and LeCun critically examine the prevalent assumptions surrounding the notions of interpolation and extrapolation in high-dimensional datasets. Their investigation unveils the underpinnings of these concepts in high-dimensional domains, setting the stage to reconsider the efficacy and limitations of contemporary machine learning models in terms of generalization capabilities.

Core Contributions

The authors challenge two widespread misconceptions: first, the belief that state-of-the-art machine learning algorithms excel primarily due to their interpolation prowess in training datasets; second, the presumption that interpolation is pervasive across various datasets and tasks. They convincingly argue both theoretically and empirically that, in high-dimensional spaces (greater than 100 dimensions), interpolation is an exceedingly rare occurrence. Their findings suggest that existing definitions of interpolation and extrapolation—specifically, based on a sample's relationship to the convex hull of a dataset—are inadequate for assessing generalization performance.

Theoretical Insights

Central to this paper is Theorem \ref{thm:exponential}, which highlights the improbability of interpolation in high dimensions. Specifically, the probability of a sample from a high-dimensional dataset lying within the convex hull of the dataset decreases toward zero as the dimensionality of the data increases unless the dataset's size grows exponentially with dimension. This theorem underscores that most new observations encountered by a model are likely outside the convex hull, thus situating them in the field of extrapolation.

Empirical Evidence

Alongside theoretical discussion, the authors present empirical evidence from synthetic and real-world datasets, including MNIST, CIFAR10, and Imagenet. They show that even when subsets of dimensions are considered, the probability of interpolation diminishes exponentially with the number of dimensions. The exploration extends to various embeddings, demonstrating that standard deep learning models operating in their latent spaces are also primarily functioning in extrapolation regimes.

Practical and Theoretical Implications

The implications of this research are manifold. Practically, it suggests a re-evaluation of the understanding of generalization in machine learning. Given that models are often operating outside the convex hull of training data, extrapolation is more reflective of real-world deployment, urging practitioners to focus on improving robustness in such scenarios. Theoretically, this insight provokes a reconsideration of how interpolation and extrapolation are defined, urging for definitions that align more closely with generalization performance, especially in the context of high-dimensional and manifold-based data representations.

Furthermore, the results challenge the utility of dimensionality reduction techniques in preserving interpolation/extrapolation information and call into question the assumptions held by current practices in embedding data for machine learning tasks. These findings prompt a shift towards research on creating new methodologies and theoretical frameworks that better accommodate and leverage the intrinsic characteristics of high-dimensional data.

Future Directions

This paper sets a precursor for extensive future work in developing refined mathematical frameworks to understand and predict machine learning model behaviors beyond traditional interpolation notions. It also invites further exploration into algorithms that are optimized specifically for extrapolation, potentially leading to more robust predictive models capable of operating efficiently in high dimensions. Continued investigation into the interaction between data geometry and learning algorithms could yield insights that drive more effective generalization strategies in machine learning systems.

In conclusion, "Learning in High Dimension Always Amounts to Extrapolation" dismantles prevalent misconceptions in the field and opens new avenues for recognizing and addressing the challenges presented by high-dimensional learning environments. Through both theoretical and empirical clarity, this work challenges the community to rethink foundational assumptions and reorient research efforts to align with the actual operational context of machine learning models.