Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression (2402.13622v2)

Abstract: We investigate popular resampling methods for estimating the uncertainty of statistical models, such as subsampling, bootstrap and the jackknife, and their performance in high-dimensional supervised regression tasks. We provide a tight asymptotic description of the biases and variances estimated by these methods in the context of generalized linear models, such as ridge and logistic regression, taking the limit where the number of samples $n$ and dimension $d$ of the covariates grow at a comparable fixed rate $\alpha!=! n/d$. Our findings are three-fold: i) resampling methods are fraught with problems in high dimensions and exhibit the double-descent-like behavior typical of these situations; ii) only when $\alpha$ is large enough do they provide consistent and reliable error estimations (we give convergence rates); iii) in the over-parametrized regime $\alpha!<!1$ relevant to modern machine learning practice, their predictions are not consistent, even with optimal regularization.

• The paper introduces a novel mathematical framework that meticulously quantifies bias and variance misestimations of bootstrap and subsampling methods in high-dimensional regression.
• It demonstrates that reliable error estimates emerge only when the sample-to-feature ratio exceeds a critical threshold, particularly failing in over-parametrized regimes.
• The findings highlight significant implications for machine learning, urging future research to develop refined resampling techniques tailored for high-dimensional data.

Insights into Resampling Methods in High-dimensional Regularized Regression

Introduction

In the rapidly evolving field of machine learning, obtaining accurate error estimates in high-dimensional settings remains a challenge. Traditional statistical measures, designed for low-dimensional frameworks, often fail to provide reliable error quantifications when applied to high-dimensional data, particularly in the field of modern machine learning applications. This issue is critically examined in the context of resampling methods, such as bootstrap and subsampling, within high-dimensional regularized regression models.

Setting & Motivation

The paper focuses on generalized linear models, including both ridge and logistic regression, under high-dimensional scenarios where the number of samples $n$ and the number of features $d$ are both large and their ratio $\alpha = n/d$ is fixed. It meticulously investigates how traditional resampling methods perform in such settings, by providing a mathematical analysis of biases and variances estimated through these techniques.

Statistical Framework

The statistical properties of three popular resampling methods are scrutinized:

• Pair bootstrap and residual bootstrap, where the former resamples data points with replacement, and the latter resamples residuals.
• Subsampling, which generates smaller datasets by selecting a fraction of the data without replacement.
• Jackknife, a technique that estimates errors by systematically leaving out one observation at a time.

Key Contributions

A novel mathematical framework is established, offering a detailed asymptotic description of the behavior of resampling methods in high-dimensional spaces. Notably, this analysis reveals:

• In high dimensions, resampling methods, including bootstrap and subsampling, can notably misestimate biases and variances. This misestimation is attributed to the peculiarities of high-dimensional statistical landscapes, such as the 'double descent' phenomenon.
• Reliable estimations emerge only when the ratio $\alpha$ surpasses a certain threshold, indicating a domain where the methods converge and provide consistent error estimates.
• Specifically within the over-parametrized regime ($\alpha < 1$), predictions from these resampling methods become inconsistent, underlining a significant challenge for applying traditional error estimation techniques in modern machine learning paradigms.

Practical and Theoretical Implications

The findings from this paper have profound implications for both theorists and practitioners in the field of machine learning and statistics:

• The demonstrated inconsistency of traditional resampling methods in high dimensions critically impacts their reliability for error estimation in many contemporary machine learning applications.
• The analysis propels further research into developing or adapting error estimation methods that are robust in high-dimensional settings.
• Practically, the results serve as a cautionary tale for relying on bootstrap and subsampling methods for error estimates in high-dimensional supervised learning tasks without accounting for dimensionality effects.

Future Directions

The paper suggests several avenues for future research, including the exploration of resampling methods that could circumvent the identified high-dimensional limitations, and the investigation into how data structure influences the performance of existing resampling methods. There's a compelling need for novel methodologies or significant adaptations of current techniques to ensure reliable error estimation in the high-dimensional regime pivotal to cutting-edge machine learning models.

Conclusion

This comprehensive analysis underscores a critical gap in the reliability of conventional resampling methods when applied to high-dimensional regularized regression tasks. By mathematically highlighting the conditions under which these methods fail or succeed, the paper lays a foundation for future investigations aimed at overcoming these limitations, ultimately guiding the development of more robust error estimation techniques for high-dimensional data analysis.