Conformal online model aggregation (2403.15527v2)
Abstract: Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification without making strong distributional assumptions. It wraps around any black-box prediction model and converts point predictions into set predictions that have a predefined marginal coverage guarantee. However, conformal prediction only works if we fix the underlying machine learning model in advance. A relatively unaddressed issue in conformal prediction is that of model selection and/or aggregation: for a given problem, which of the plethora of prediction methods (random forests, neural nets, regularized linear models, etc.) should we conformalize? This paper proposes a new approach towards conformal model aggregation in online settings that is based on combining the prediction sets from several algorithms by voting, where weights on the models are adapted over time based on past performance.
- Conformal PID control for time series prediction. Advances in Neural Information Processing Systems, 36.
- Online conformal prediction with decaying step sizes. arXiv preprint arXiv:2402.01139.
- Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845.
- A paradox in decision-theoretic interval estimation. Statistica Sinica, 3(1):141–155.
- Follow the leader if you can, hedge if you must. Journal of Machine Learning Research, 15(37):1281–1316.
- A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119–139.
- Merging uncertainty sets via majority vote. arXiv preprint arXiv:2401.09379.
- Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660–1672.
- Conformal inference for online prediction with arbitrary distribution shifts. arXiv preprint arXiv:2208.08401.
- Harries, M. (2003). Splice-2 comparative evaluation: Electricity pricing.
- Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111.
- Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13, pages 345–356. Springer.
- Randomized and Exchangeable Improvements of Markov’s, Chebyshev’s and Chernoff’s inequalities. arXiv preprint arXiv:2304.02611.
- Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- A Tutorial on Conformal Prediction. Journal of Machine Learning Research, 9(3).
- Conformal prediction under covariate shift. Advances in neural information processing systems, 32.
- Algorithmic Learning in a Random World, volume 29. Springer.
- Adaptive conformal predictions for time series. In International Conference on Machine Learning, pages 25834–25866. PMLR.
- Matteo Gasparin (5 papers)
- Aaditya Ramdas (180 papers)