Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 83 tok/s
Gemini 2.5 Pro 42 tok/s Pro
GPT-5 Medium 30 tok/s Pro
GPT-5 High 36 tok/s Pro
GPT-4o 108 tok/s Pro
Kimi K2 220 tok/s Pro
GPT OSS 120B 473 tok/s Pro
Claude Sonnet 4 40 tok/s Pro
2000 character limit reached

Progression: an extrapolation principle for regression (2410.23246v1)

Published 30 Oct 2024 in stat.ME and stat.ML

Abstract: The problem of regression extrapolation, or out-of-distribution generalization, arises when predictions are required at test points outside the range of the training data. In such cases, the non-parametric guarantees for regression methods from both statistics and machine learning typically fail. Based on the theory of tail dependence, we propose a novel statistical extrapolation principle. After a suitable, data-adaptive marginal transformation, it assumes a simple relationship between predictors and the response at the boundary of the training predictor samples. This assumption holds for a wide range of models, including non-parametric regression functions with additive noise. Our semi-parametric method, progression, leverages this extrapolation principle and offers guarantees on the approximation error beyond the training data range. We demonstrate how this principle can be effectively integrated with existing approaches, such as random forests and additive models, to improve extrapolation performance on out-of-distribution samples.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. A note on second order conditions in extreme value theory: linking general and heavy tail conditions. REVSTAT-Statistical Journal 5(3), 285–304.
  2. Tail asymptotics of light-tailed weibull-like sums. Probability and Mathematical Statistics 37, 235–256.
  3. Generalized random forests. Ann. Statist. 47, 1148–1178.
  4. Residual life time at great age. The Annals of Probability 2, 792–804.
  5. Densities with Gaussian tails. Proc. Lond. Math. Soc. 66, 568–588.
  6. Statistics of Extremes: Theory and Applications. West Sussex: John Wiley & Sons.
  7. A modeler’s guide to extreme value software. Extremes 26(4), 595–638.
  8. Modeling and simulating spatial extremes by combining extreme value theory with generative adversarial networks. Environmental Data Science 1, e5.
  9. Box, G. E. P. and D. R. Cox (1964). An analysis of transformations. Journal of the Royal Statistical Society. Series B (Methodological) 26(2), 211–252.
  10. Breiman, L. (2001). Random forests. Machine Learning 45, 5–32.
  11. Bühlmann, P. (2020). Invariance, causality and robustness. Statistical Science 35, 404–426.
  12. Caeiro, F. and M. I. Gomes (2016). Threshold selection in extreme value analysis. In Extreme value modeling and risk analysis, pp.  69–86. CRC Press, Boca Raton, FL.
  13. Chernozhukov, V. (2005). Extremal quantile regression. The Annals of Statistics 33(2), 806 – 839.
  14. BART: Bayesian additive regression trees. Ann. Appl. Stat. 4(1), 266–298.
  15. A causal framework for distribution generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 6614–6630.
  16. Cline, D. B. (1986). Convolution tails, product tails and domain of attraction. Probability Theory and Related Fields 72, 529–557.
  17. Coles, S. (2001). An Introduction to Statistical Modeling of Extreme Values. London: Springer Series in Statistics.
  18. Learning bounds for importance weighting. In Advances in Neural Information Processing Systems, Volume 23, pp.  442–450.
  19. Extreme Value Theory: and introduction. New York: Springer.
  20. A probabilistic theory of pattern recognition, Volume 31 of Applications of Mathematics (New York). Springer-Verlag, New York.
  21. First steps toward understanding the extrapolation of nonlinear models to unseen domains. In The Eleventh International Conference on Learning Representations.
  22. Modelling extremal events: for insurance and finance, Volume 13. Zurich: Springer.
  23. Extremal dependence of random scale constructions. Extremes 22(4), 623–666.
  24. Neural network extrapolation to distant regions of the protein fitness landscape. Nature Communications 15, 6405.
  25. Local linear forest. Journal of computational and graphical statistics 30(2), 503–517.
  26. Friedman, J. H. (1991). Multivariate adaptive regression splines. Ann. Statist. 19(1), 1–141.
  27. Galton, F. (1886). Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland 15, 246–263.
  28. Domain-adversarial training of neural networks. Journal of Machine Learning Research 17(59), 1–35.
  29. Second-order regular variation, convolution and the central limit theorem. Stochastic Process. Appl. 69(2), 139–159.
  30. Convolutions of heavy-tailed random variables and applications to portfolio diversification and MA⁢(1)MA1{\rm MA}(1)roman_MA ( 1 ) time series. Adv. in Appl. Probab. 32(4), 1011–1026.
  31. Boosted control function. arXiv 2310.05805.
  32. Extremal random forests. Journal of the American Statistical Association. To appear.
  33. Extreme Value Theory: an introduction. New York: Springer.
  34. Generalized additive models: Some applications. Journal of the American Statistical Association 82(398), 371–386.
  35. The elements of statistical learning (Second ed.). Springer Series in Statistics. Springer, New York.
  36. Heffernan, J. E. and J. A. Tawn (2004). A conditional approach for multivariate extreme values (with discussion). Journal of the Royal Statistical Society: Series B (Statistical Methodology) 66, 497–546.
  37. Assessing present and future risk of water damage using building attributes, meteorology, and topography. Journal of the Royal Statistical Society Series C: Applied Statistics 72(4), 809–828.
  38. On regression in extreme regions. https://arxiv.org/abs/2303.03084.
  39. On binary classification in extreme regions. Advances in Neural Information Processing Systems 31 31, 3092–3100.
  40. Koh, J. (2023). Gradient boosting with extreme-value theory for wildfire prediction. Extremes 26, 273–299.
  41. Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning, Chapter 6, pp.  130–166.
  42. Learning skillful medium-range global weather forecasting. Science 382(6677), 1416–1421.
  43. Optimally tackling covariate shift in rkhs-based nonparametric regression. Ann. Statist. 51, 738–761.
  44. Meinshausen, N. (2007). Quantile regression forests. JMLR 7, 983–999.
  45. Mikosch, T. (2014). Modeling heavy-tailed time series.
  46. Namkoong, H. and J. C. Duchi (2016). Stochastic gradient methods for distributionally robust optimization with f-divergences. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Advances in Neural Information Processing Systems, Volume 29. Curran Associates, Inc.
  47. Neural networks for extreme quantile regression with an application to forecasting of flood risk. Ann. Appl. Stat.. To appear.
  48. Validating deep-learning weather forecast models on recent high-impact extreme events. https://arxiv.org/abs/2404.17652.
  49. Extrapolation-aware nonparametric statistical inference. arXiv:2402.09758.
  50. Pickands, J. (1975). Statistical inference using extreme order statistics. The Annals of Statistics 3, 119–131.
  51. Prediction of regional wildfire activity in the probabilistic bayesian framework of firelihood. Ecological Applications 31(5), e02316.
  52. Dataset shift in machine learning. London: MIT Press.
  53. Resnick, S. I. (2007). Heavy-tail Phenomena: Probabilistic and Statistical Modeling. New York: Springer.
  54. Anchor regression: Heterogeneous data meet causality. Journal of Royal Statistical Society. Series B. 83, 215–246.
  55. Engression: Extrapolation for nonlinear regression? arXiv:2307.00835.
  56. Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90, 227–244.
  57. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research 8(35), 985–1005.
  58. Gradient boosting for extreme quantile regression. Extremes 26, 639–667.
  59. Extreme value theory for anomaly detection — the gpd classifier. Extremes 23, 501–520.
  60. How neural networks extrapolate: From feedforward to graph neural networks. In International Conference on Learning Representations.
  61. Youngman, B. D. (2019). Generalized Additive Models for Exceedances of High Thresholds With an Application to Return Level Estimation for U.S. Wind Gusts. J. Am. Stat. Assoc. 114(528), 1865–1879.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube