Random features models: a way to study the success of naive imputation
Abstract: Constant (naive) imputation is still widely used in practice as this is a first easy-to-use technique to deal with missing data. Yet, this simple method could be expected to induce a large bias for prediction purposes, as the imputed input may strongly differ from the true underlying data. However, recent works suggest that this bias is low in the context of high-dimensional linear predictors when data is supposed to be missing completely at random (MCAR). This paper completes the picture for linear predictors by confirming the intuition that the bias is negligible and that surprisingly naive imputation also remains relevant in very low dimension.To this aim, we consider a unique underlying random features model, which offers a rigorous framework for studying predictive performances, whilst the dimension of the observed features varies.Building on these theoretical results, we establish finite-sample bounds on stochastic gradient (SGD) predictors applied to zero-imputed data, a strategy particularly well suited for large-scale learning.If the MCAR assumption appears to be strong, we show that similar favorable behaviors occur for more complex missing data scenarios.
- On robustness of principal component regression. Advances in Neural Information Processing Systems, 32, 2019.
- Near-optimal rate of consistency for linear models with missing values. In International Conference on Machine Learning, pages 1211–1243. PMLR, 2022.
- Naive imputation implicitly regularizes high-dimensional linear models. In International Conference on Machine Learning, 2023.
- Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629–681, 2017.
- Non-strongly-convex smooth stochastic approximation with convergence rate o (1/n). Advances in neural information processing systems, 26, 2013.
- From predictive methods to missing data imputation: an optimization approach. Journal of Machine Learning Research, 18(196):1–39, 2018.
- Beyond impute-then-regress: Adapting prediction to missing data. arXiv preprint arXiv:2104.03158, 2021.
- Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368, 2007.
- Eric Carlen. Trace inequalities and quantum entropy: an introductory course. Entropy and the quantum, 529:73–140, 2010.
- Learning with sgd and random features. Advances in neural information processing systems, 31, 2018.
- Missing covariates in logistic regression, estimation and distribution selection. Statistical Modelling, 11(2):159–183, 2011.
- On the mean and variance of the generalized inverse of a singular wishart matrix. 2011.
- Nonparametric stochastic approximation with large step-sizes. 2016.
- Harder, better, faster, stronger convergence rates for least-squares regression. The Journal of Machine Learning Research, 18(1):3520–3570, 2017.
- Christophe Giraud. Introduction to high-dimensional statistics. CRC Press, 2021.
- Statistical learning with sparsity: the lasso and generalizations. CRC press, 2015.
- Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949–986, 2022.
- Random design analysis of ridge regression. In Conference on learning theory, pages 9–1. JMLR Workshop and Conference Proceedings, 2012.
- Logistic regression with missing covariates—parameter estimation, model selection and prediction within a joint-modeling framework. Computational Statistics & Data Analysis, 145:106907, 2020.
- Iain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of Statistics, 29(2):295 – 327, 2001. doi: 10.1214/aos/1009210544. URL https://doi.org/10.1214/aos/1009210544.
- Michael P Jones. Indicator and stratification methods for missing explanatory variables in multiple linear regression. Journal of the American statistical association, 91(433):222–230, 1996.
- On the consistency of supervised learning with missing values. arXiv preprint arXiv:1902.06931, 2019.
- Linear predictor on linearly-generated data with missing values: non consistency and solutions. In International Conference on Artificial Intelligence and Statistics, pages 3165–3174. PMLR, 2020.
- What’sa good imputation to predict with missing values? Advances in Neural Information Processing Systems, 34:11530–11540, 2021.
- Roderick JA Little. Regression with missing x’s: a review. Journal of the American statistical association, 87(420):1227–1237, 1992.
- An elementary analysis of ridge regression with random design. arXiv preprint arXiv:2203.08564, 2022.
- Thomas J Page Jr. Multivariate statistics: A vector space approach. Journal of Marketing Research, 21(2):236–236, 1984.
- Benchmarking missing-values approaches for predictive models on health databases. GigaScience, 11:giac013, 2022.
- Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
- Early stopping and non-parametric regression: an optimal data-dependent stopping rule. The Journal of Machine Learning Research, 15(1):335–366, 2014.
- Generalization properties of learning with random features. Advances in neural information processing systems, 30, 2017.
- Missforest—non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1):112–118, 2012.
- Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144–160, 2019.
- Dietrich Von Rosen. Moments for the inverted wishart distribution. Scandinavian Journal of Statistics, pages 97–109, 1988.
- Does imputation matter? benchmark for predictive models. arXiv preprint arXiv:2007.02837, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
