Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Conformal Inference for Online Prediction with Arbitrary Distribution Shifts (2208.08401v3)

Published 17 Aug 2022 in stat.ME and cs.LG

Abstract: We consider the problem of forming prediction sets in an online setting where the distribution generating the data is allowed to vary over time. Previous approaches to this problem suffer from over-weighting historical data and thus may fail to quickly react to the underlying dynamics. Here we correct this issue and develop a novel procedure with provably small regret over all local time intervals of a given width. We achieve this by modifying the adaptive conformal inference (ACI) algorithm of Gibbs and Cand`{e}s (2021) to contain an additional step in which the step-size parameter of ACI's gradient descent update is tuned over time. Crucially, this means that unlike ACI, which requires knowledge of the rate of change of the data-generating mechanism, our new procedure is adaptive to both the size and type of the distribution shift. Our methods are highly flexible and can be used in combination with any baseline predictive algorithm that produces point estimates or estimated quantiles of the target without the need for distributional assumptions. We test our techniques on two real-world datasets aimed at predicting stock market volatility and COVID-19 case counts and find that they are robust and adaptive to real-world distribution shifts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486 – 507, 2021. doi: 10.1214/20-AOS1965. URL https://doi.org/10.1214/20-AOS1965.
  2. Conformal prediction beyond exchangeability. arXiv preprint, 2022. arXiv:2202.13415.
  3. Practical adversarial multivalid conformal prediction. arXiv preprint, 2022. arXiv:2206.01067.
  4. Exact and robust conformal inference methods for predictive machine learning with dependent data. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 732–749. PMLR, 06–09 Jul 2018. URL http://proceedings.mlr.press/v75/chernozhukov18a.html.
  5. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 08 2020. ISSN 2049-8772. doi: 10.1093/imaiai/iaaa017. URL https://doi.org/10.1093/imaiai/iaaa017. iaaa017.
  6. Hedging predictions in machine learning. The Computer Journal, 50(2):151–163, 2007. doi: 10.1093/comjnl/bxl065.
  7. Adaptive conformal inference under distribution shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1660–1672. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/0d441de75945e5acbc865406fc9a2559-Paper.pdf.
  8. Adaptive regret for control of time-varying dynamics. arXiv preprint, 2020. arXiv:2007.04393.
  9. Elad Hazan. Introduction to online convex optimization. arXiv preprint, 2019. arXiv:1909.05207.
  10. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637–5664. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/koh21a.html.
  11. Distribution‐free prediction bands for non‐parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76, 01 2014. doi: 10.1111/rssb.12021.
  12. Conformal prediction with temporal quantile adjustments. arXiv preprint, 2022. arXiv:2205.09940.
  13. Harris Papadopoulos. Inductive conformal prediction: Theory and application to neural networks. In Tools in Artificial Intelligence,, pages 315–330, 2008.
  14. Inductive confidence machines for regression. In Tapio Elomaa, Heikki Mannila, and Hannu Toivonen, editors, Machine Learning: ECML 2002, pages 345–356, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg. ISBN 978-3-540-36755-0.
  15. Distribution-free uncertainty quantification for classification under label shift. arXiv preprint, 2021. arXiv:2103.03323.
  16. An open repository of real-time covid-19 indicators. Proceedings of the National Academy of Sciences, 118(51):e2111452118, 2021. doi: 10.1073/pnas.2111452118. URL https://www.pnas.org/doi/abs/10.1073/pnas.2111452118.
  17. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/5103c3584b063c431bd1268e9b5e76fb-Paper.pdf.
  18. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223–234, 2019. doi: 10.1080/01621459.2017.1395341. URL https://doi.org/10.1080/01621459.2017.1395341.
  19. Transduction with confidence and credibility. In Sixteenth International Joint Conference on Artificial Intelligence (IJCAI ’99) (01/01/99), pages 722–726, 1999. URL https://eprints.soton.ac.uk/258961/.
  20. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(12):371–421, 2008. URL http://jmlr.org/papers/v9/shafer08a.html.
  21. R. J. Tibshirani. Can symptoms surveys improve covid-19 forecasts? url https://delphi.cmu.edu/blog/2020/09/21/. http://web.archive.org/web/20080207010024/http://www.808multimedia.com/winnt/kernel.htm, 2020. Accessed: 2022-06-17.
  22. Conformal prediction under covariate shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/8fb21ee7a2207526da55a679f0332de2-Paper.pdf.
  23. Machine-learning applications of algorithmic randomness. In Sixteenth International Conference on Machine Learning (ICML-1999) (01/01/99), pages 444–453, 1999. URL https://eprints.soton.ac.uk/258960/.
  24. Algorithmic Learning in a Random World. Springer-Verlag, Berlin, Heidelberg, 2005. ISBN 0387001522.
  25. Volodimir G. Vovk. Aggregating strategies. In Proceedings of the Third Annual Workshop on Computational Learning Theory, COLT ’90, page 371–386, San Francisco, CA, USA, 1990. Morgan Kaufmann Publishers Inc. ISBN 1558601465.
  26. Olivier Wintenberger. Optimal learning with bernstein online aggregation. Mach. Learn., 106(1):119–141, jan 2017. ISSN 0885-6125. doi: 10.1007/s10994-016-5592-6. URL https://doi.org/10.1007/s10994-016-5592-6.
  27. Doubly robust calibration of prediction sets under covariate shift. arXiv preprint, 2022. arXiv:2203.01761.
  28. Adaptive conformal predictions for time series. arXiv preprint, 2022. arXiv:2202.07282.
Citations (69)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now