Unbiased Estimation of Structured Prediction Error
Abstract: Many modern datasets, such as those in ecology and geology, are composed of samples with spatial structure and dependence. With such data violating the usual independent and identically distributed (IID) assumption in machine learning and classical statistics, it is unclear a priori how one should measure the performance and generalization of models. Several authors have empirically investigated cross-validation (CV) methods in this setting, reaching mixed conclusions. We provide a class of unbiased estimation methods for general quadratic errors, correlated Gaussian response, and arbitrary prediction function $g$, for a noise-elevated version of the error. Our approach generalizes the coupled bootstrap (CB) from the normal means problem to general normal data, allowing correlation both within and between the training and test sets. CB relies on creating bootstrap samples that are intelligently decoupled, in the sense of being statistically independent. Specifically, the key to CB lies in generating two independent "views" of our data and using them as stand-ins for the usual independent training and test samples. Beginning with Mallows' $C_p$, we generalize the estimator to develop our generalized $C_p$ estimators (GC). We show at under only a moment condition on $g$, this noise-elevated error estimate converges smoothly to the noiseless error estimate. We show that when Stein's unbiased risk estimator (SURE) applies, GC converges to SURE as in the normal means problem. Further, we use these same tools to analyze CV and provide some theoretical analysis to help understand when CV will provide good estimates of error. Simulations align with our theoretical results, demonstrating the effectiveness of GC and illustrating the behavior of CV methods. Lastly, we apply our estimator to a model selection task on geothermal data in Nevada.
- Akaike, H. (1998). Information Theory and an Extension of the Maximum Likelihood Principle, pages 199–213. Springer New York, New York, NY.
- Cross-validation: what does it estimate and how well does it do it?
- Breiman, L. (1992). The little bootstrap and other methods for dimensionality selection in regression: X-fixed prediction error. Journal of the American Statistical Association, 87(419):738–754.
- Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2):123–140.
- Smoothing effects of bagging: Von mises expansions of bagged statistical functionals.
- Efron, B. (1986). How biased is the apparent error rate of a prediction rule? Journal of the American Statistical Association, 81(394):461–470.
- Efron, B. (2004). The estimation of prediction error. Journal of the American Statistical Association, 99(467):619–632.
- Efron, B. (2014). Estimation and accuracy after model selection. Journal of the American Statistical Association, 109(507):991–1007. 25346558[pmid], PMC4207812[pmcid].
- Comparing spatial regression to random forests for large environmental data sets. PLOS ONE, 15(3):1–22.
- Friedman, J. H. (1991). Multivariate Adaptive Regression Splines. The Annals of Statistics, 19(1):1 – 67.
- The elements of statistical learning: data mining, inference and prediction. Springer, 2 edition.
- Random forest as a generic framework for predictive modeling of spatial and spatio-temporal variables. PeerJ.
- Data blurring: sample splitting a single sample.
- Levenberg, K. (1944). A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2(2):164–168.
- Mallows, C. L. (1973). Some comments on Cpsubscript𝐶𝑝{C}_{p}italic_C start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT. Technometrics, 15(4):661–675.
- Marquardt, D. W. (1963). An algorithm for least-squares estimation of nonlinear parameters. Journal of the Society for Industrial and Applied Mathematics, 11(2):431–441.
- The matérn function as a general model for soil variograms. Geoderma, 128(3):192–207. Pedometrics 2003.
- Scikit-gstat: A scipy flavoured geostatistical variogram analysis toolbox.
- Unbiased risk estimation in the normal means problem via coupled bootstrap techniques.
- Coupled bootstrap test error estimation for Poisson variables.
- Predicting alpha diversity of african rain forests: models based on climate and satellite-derived data do not perform better than a purely spatial model. Journal of Biogeography, 38(6):1164–1176.
- Spatial validation reveals poor predictive performance of large-scale ecological mapping models. Nature Communications, 11(1):4540.
- Adaptive nonparametric confidence sets. The Annals of Statistics, 34(1):229 – 253.
- From fixed-x to random-x regression: Bias-variance decompositions, covariance penalties, and prediction error estimation. Journal of the American Statistical Association, 115(529):138–151.
- Random forests for dependent data.
- Stein, C. M. (1981). Estimation of the Mean of a Multivariate Normal Distribution. The Annals of Statistics, 9(6):1135 – 1151.
- Tian, X. (2020). Prediction error after model search. The Annals of Statistics, 48(2):763 – 784.
- Selective inference with a randomized response.
- Tibshirani, R. J. (2015). Degrees of freedom and Model Search. Statistica Sinica, 25(3):1265 – 1296.
- Degrees of freedom in lasso problems. The Annals of Statistics, 40(2):1198 – 1232.
- Spatial cross-validation is not the right way to evaluate map accuracy. Ecological Modelling, 457:109692.
- Ye, J. (1998). On measuring and correcting the effects of data mining and model selection. Journal of the American Statistical Association, 93(441):120–131.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.