Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Orthogonal Bootstrap: Efficient Simulation of Input Uncertainty (2404.19145v2)

Published 29 Apr 2024 in stat.ME, cs.LG, econ.EM, math.ST, stat.ML, and stat.TH

Abstract: Bootstrap is a popular methodology for simulating input uncertainty. However, it can be computationally expensive when the number of samples is large. We propose a new approach called \textbf{Orthogonal Bootstrap} that reduces the number of required Monte Carlo replications. We decomposes the target being simulated into two parts: the \textit{non-orthogonal part} which has a closed-form result known as Infinitesimal Jackknife and the \textit{orthogonal part} which is easier to be simulated. We theoretically and numerically show that Orthogonal Bootstrap significantly reduces the computational cost of Bootstrap while improving empirical accuracy and maintaining the same width of the constructed interval.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Approximating full conformal prediction at scale via influence functions. arXiv preprint arXiv:2202.01315, 2022.
  2. Abadi, M. Tensorflow: learning functions at scale. In Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming, pp.  1–1, 2016.
  3. An asymptotic characterization of bias reduction by jackknifing. The Annals of Mathematical Statistics, pp.  1606–1612, 1971.
  4. Discriminative jackknife: Quantifying uncertainty in deep learning via higher-order influence functions. In International Conference on Machine Learning, pp.  165–174. PMLR, 2020.
  5. Stochastic simulation: algorithms and analysis, volume 57. Springer, 2007.
  6. Chang, K.-C. Methods in nonlinear analysis, volume 10. Springer, 2005.
  7. Debiasing samples from online learning using bootstrap. In International Conference on Artificial Intelligence and Statistics, pp.  8514–8533. PMLR, 2022.
  8. Double/debiased machine learning for treatment and structural parameters, 2018.
  9. Locally robust semiparametric estimation. Econometrica, 90(4):1501–1535, 2022a.
  10. Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3):967–1027, 2022b.
  11. Constructing prediction intervals with neural networks: An empirical evaluation of bootstrapping and conformal inference methods. arXiv preprint arXiv:2210.05354, 2022.
  12. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495–508, 1980.
  13. Residuals and influence in regression. New York: Chapman and Hall, 1982.
  14. An introduction to Bartlett correction and bias reduction. Springer, 2014.
  15. Dayal, S. A converse of taylor’s theorem for functions on banach spaces. Proceedings of the American Mathematical Society, 65(2):265–273, 1977.
  16. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
  17. Efron, B. The jackknife, the bootstrap and other resampling plans. SIAM, 1982.
  18. Efron, B. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics, pp.  569–593. Springer, 1992a.
  19. Efron, B. Jackknife-after-bootstrap standard errors and influence functions. Journal of the Royal Statistical Society: Series B (Methodological), 54(1):83–111, 1992b.
  20. The jackknife estimate of variance. The Annals of Statistics, pp.  586–596, 1981.
  21. An introduction to the bootstrap. CRC press, 1994.
  22. Operator augmentation for general noisy matrix systems. arXiv preprint arXiv:2104.11294, 2021.
  23. Operator augmentation for noisy elliptic systems. arXiv preprint arXiv:2010.09656, 2020.
  24. Fernholz, L. T. Von Mises calculus for statistical functionals, volume 19. Springer Science & Business Media, 2012.
  25. Filippova, A. Mises’ theorem on the asymptotic behavior of functionals of empirical distribution functions and its statistical applications. Theory of Probability & Its Applications, 7(1):24–57, 1962.
  26. Orthogonal statistical learning. arXiv preprint arXiv:1901.09036, 2019.
  27. A higher-order swiss army infinitesimal jackknife. arXiv preprint arXiv:1907.12116, 2019a.
  28. A swiss army infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.  1139–1147. PMLR, 2019b.
  29. Hall, P. On the bootstrap and confidence intervals. The Annals of Statistics, pp.  1431–1452, 1986.
  30. Hall, P. The bootstrap and Edgeworth expansion. Springer Science & Business Media, 2013.
  31. On bootstrap resampling and iteration. Biometrika, 75(4):661–671, 1988.
  32. Jaeckel, L. A. The infinitesimal jackknife. Bell Telephone Laboratories, 1972.
  33. Bias correction with jackknife, bootstrap, and taylor series. IEEE Transactions on Information Theory, 66(7):4392–4418, 2020.
  34. Influence functions for machine learning: Nonparametric estimators for entropies, divergences and mutual informations. arXiv preprint arXiv:1411.4342, 2014.
  35. Comprehensive review of neural network-based prediction intervals and new advances. IEEE Transactions on Neural Networks, 22(9):1341–1356, 2011. doi: 10.1109/TNN.2011.2162110.
  36. A score based approach to wild bootstrap inference. Journal of Econometric Methods, 1(1):23–41, 2012.
  37. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.  1885–1894. PMLR, 2017.
  38. Koltchinskii, V. Estimation of smooth functionals in high-dimensional models: bootstrap chains and gaussian approximation. The Annals of Statistics, 50(4):2386–2415, 2022.
  39. Estimation of smooth functionals in normal models: bias reduction and asymptotic efficiency. The Annals of Statistics, 49(5):2577–2610, 2021.
  40. Lam, H. A cheap bootstrap method for fast inference. arXiv preprint arXiv:2202.00090, 2022.
  41. Subsampling variance for input uncertainty quantification. In 2018 Winter Simulation Conference (WSC), pp.  1611–1622. IEEE, 2018.
  42. Subsampling to enhance efficiency in input uncertainty quantification. Operations Research, 70(3):1891–1913, 2022.
  43. Li, S. Debiasing the debiased lasso with bootstrap. Electronic Journal of Statistics, 14(1):2298–2337, 2020.
  44. Liu, R. Y. Bootstrap procedures under some non-iid models. The annals of statistics, 16(4):1696–1708, 1988.
  45. Uncertainty estimation with infinitesimal jackknife, its distribution and mean-field approximation. arXiv preprint arXiv:2006.07584, 2020.
  46. Correcting convexity bias in function and functional estimate. arXiv preprint arXiv:2208.07996, 2022.
  47. Martens, J. Deep learning via hessian-free optimization. In ICML, volume 27, pp.  735–742, 2010.
  48. Mises, R. v. On the asymptotic distribution of differentiable statistical functions. The annals of mathematical statistics, 18(3):309–348, 1947.
  49. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning, 10(1-2):1–141, 2017.
  50. Improved worst-group robustness via classifier retraining on independent splits. arXiv preprint arXiv:2204.09583, 2022.
  51. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  52. Quenouille, M. H. Approximate tests of correlation in time-series 3. In Mathematical Proceedings of the Cambridge Philosophical Society, volume 45, pp.  483–484. Cambridge University Press, 1949.
  53. REEDS, J. I. On the definition of von mises functionals. Ph. D thesis, Harvard University, 1976.
  54. Robust statistics: the approach based on influence functions. John Wiley & Sons, 2011.
  55. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp.  8346–8356. PMLR, 2020.
  56. A subsampled double bootstrap for massive data. Journal of the American Statistical Association, 111(515):1222–1232, 2016.
  57. Serfling, R. J. Approximation theorems of mathematical statistics. John Wiley & Sons, 2009.
  58. Advanced tutorial: Input uncertainty quantification. In Proceedings of the Winter Simulation Conference 2014, pp.  162–176. IEEE, 2014.
  59. Stine, R. A. Bootstrap prediction intervals for regression. Journal of the American Statistical Association, 80(392):1026–1031, 1985.
  60. Tukey, J. Bias and confidence in not quite large samples. Ann. Math. Statist., 29:614, 1958.
  61. Van der Vaart, A. W. Asymptotic statistics, volume 3. Cambridge university press, 2000.
  62. Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. The Journal of Machine Learning Research, 15(1):1625–1651, 2014.
  63. Wu, C.-F. J. Jackknife, bootstrap and other resampling methods in regression analysis. the Annals of Statistics, 14(4):1261–1295, 1986.
  64. High-order statistical functional expansion and its application to some nonsmooth problems. arXiv preprint arXiv:2112.15591, 2021a.
  65. Unbiased optimal stopping via the muse. arXiv preprint arXiv:2106.02263, 2021b.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now