How Does Overparameterization Affect Features?

Abstract: Overparameterization, the condition where models have more parameters than necessary to fit their training loss, is a crucial factor for the success of deep learning. However, the characteristics of the features learned by overparameterized networks are not well understood. In this work, we explore this question by comparing models with the same architecture but different widths. We first examine the expressivity of the features of these models, and show that the feature space of overparameterized networks cannot be spanned by concatenating many underparameterized features, and vice versa. This reveals that both overparameterized and underparameterized networks acquire some distinctive features. We then evaluate the performance of these models, and find that overparameterized networks outperform underparameterized networks, even when many of the latter are concatenated. We corroborate these findings using a VGG-16 and ResNet18 on CIFAR-10 and a Transformer on the MNLI classification dataset. Finally, we propose a toy setting to explain how overparameterized networks can learn some important features that the underparamaterized networks cannot learn.

• The paper demonstrates that overparameterization enables networks to learn intrinsically distinct and more expressive features than underparameterized models.
• It introduces a novel feature span error (FSE) metric and employs ridge regression to quantitatively compare feature expressivity across models.
• Residual features unique to overparameterized networks drive superior performance, challenging the notion that success solely stems from increased capacity.

Overparameterization and Learned Features: An Analytical Overview

The paper "How does overparameterization affect features?" by Ahmet Cagri Duzgun, Samy Jelassi, and Yuanzhi Li addresses the nuanced and largely underexplored topic of how overparameterization influences the features learned by deep learning models. This study meticulously compares the characteristics of features in overparameterized versus underparameterized networks, focusing on the implications for model performance.

The foundational premise is the common definition of overparameterization, which refers to cases where a model has more parameters than are strictly necessary to minimize its training loss. Despite the broad acknowledgment of overparameterization as a critical enabler of deep learning success, the specific nature of the features learned and how they contribute to model efficacy remains insufficiently understood. To contribute to this knowledge gap, the authors embark on a methodical examination using models of identical architecture but varying widths, which are indicative of their parameterization level.

The authors introduce the novel "feature span error" (FSE) metric to assess the expressivity of features across differently parameterized models. This involves ridge regression to determine how well overparameterized features can capture those from underparameterized networks and vice versa. They complement this with the "feature performance" (FP) metric, which gauges task performance through linear probing on these features.

Key Findings and Implications

1. Distinctive Features in Overparameterized Networks: The study finds that the feature space of overparameterized networks cannot be effectively spanned by aggregating features from underparameterized counterparts. This indicates that overparameterized and underparameterized networks learn intrinsically different features, with the former exhibiting greater expressivity and distinctiveness.
2. Performance Superiority: Numerical experiments involving VGG-16 and ResNet18 on the CIFAR-10 dataset, and Transformers on MNLI, consistently demonstrate that overparameterized models outperform underparameterized ones. This superiority persists even when many underparameterized networks are concatenated.
3. Residual Features: The authors highlight the critical role of overparameterized feature residuals—features unexplained by concatenated underparameterized networks—in achieving high performance. Conversely, underparameterized feature residuals do not significantly contribute to improved performance.
4. Mechanistic Insights: Through a devised toy setting, the paper illustrates mechanisms by which overparameterized networks capture important features missed by underparameterized networks. This setting involves learning features requiring simultaneous activation that low-width networks struggle to maintain through training.

The results challenge the simplistic notion that models with a high number of parameters owe their success merely to increased capacity. Instead, they reveal that overparameterized networks possess an expressive feature set that underlies their powerful predictive capabilities.

Speculations on Future Directions

The paper opens several avenues for future research. One potential direction is to explore the theoretical underpinnings of feature learning in overparameterized networks beyond the proposed empirical and mechanistic frameworks. Another exciting area is exploring how these findings can inform architecture design, potentially evolving models that can harness the benefits of overparameterization more efficiently.

Further investigation could involve scaling these analyses to larger models and datasets, as well as examining the interplay of overparameterization with other architectural choices, such as depth or network topology. Finally, extending this work into the field of unsupervised or self-supervised learning could provide broader insights applicable across diverse learning paradigms.

In summary, this paper significantly advances the understanding of how overparameterization affects feature learning, providing a robust methodological framework and empirical evidence to support the unique capabilities conferred by overparameterization in neural networks. The insights derived underscore the profound impact of model architecture on the nature and utility of learned features.