Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Selecting informative conformal prediction sets with false coverage rate control (2403.12295v3)

Published 18 Mar 2024 in math.ST, stat.ML, and stat.TH

Abstract: In supervised learning, including regression and classification, conformal methods provide prediction sets for the outcome/label with finite sample coverage for any machine learning predictor. We consider here the case where such prediction sets come after a selection process. The selection process requires that the selected prediction sets be informative' in a well defined sense. We consider both the classification and regression settings where the analyst may consider as informative only the sample with prediction sets small enough, excluding null values, or obeying other appropriatemonotone' constraints. We develop a unified framework for building such informative conformal prediction sets while controlling the false coverage rate (FCR) on the selected sample. While conformal prediction sets after selection have been the focus of much recent literature in the field, the new introduced procedures, called InfoSP and InfoSCOP, are to our knowledge the first ones providing FCR control for informative prediction sets. We show the usefulness of our resulting procedures on real and simulated data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Selective conformal inference with false coverage-statement rate control. Biometrika, page asae010.
  2. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486–507.
  3. Testing for outliers with conformal p-values.
  4. Selective Inference on Multiple Families of Hypotheses. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):297–318.
  5. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B, 57(1):289–300.
  6. Adaptive linear step-up procedures that control the false discovery rate. Biometrika, 93(3):491–507.
  7. False discovery rate–adjusted multiple confidence intervals for selected parameters. Journal of the American Statistical Association, 100(469):71–81.
  8. Two simple sufficient conditions for FDR control. Electron. J. Stat., 2:963–992.
  9. Joint distribution optimal transportation for domain adaptation. In Advances in neural information processing systems 30 (NIPS 2017), volume 30.
  10. Class-conditional conformal prediction with many classes. arXiv preprint arXiv:2306.09335.
  11. Empirical Bayes analysis of a microarray experiment. J. Amer. Statist. Assoc., 96(456):1151–1160.
  12. On the Benjamini-Hochberg method. Ann. Statist., 34(4):1827–1849.
  13. A constructive approach to selective risk control. arXiv preprint arXiv:2401.16651.
  14. Transductive conformal inference with adaptive scores. arXiv preprint arXiv:2310.18108.
  15. On stepwise control of directional errors under independence and some dependence. Journal of Statistical Planning and Inference, 163:21–33.
  16. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496.
  17. Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24(244):1–41.
  18. Confidence on the focal: Conformal prediction with selection-conditional coverage.
  19. Distribution-free predictive inference for regression. J. Amer. Stat. Assoc., 113(523):1094–1111.
  20. A conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence, 74:29–43.
  21. Adaptive novelty detection with false discovery rate guarantee. The Annals of Statistics, 52(1):157–183.
  22. False clustering rate control in mixture models. arXiv preprint arXiv:2203.02597.
  23. Error rate control for classification rules in multiclass mixture models. The international journal of biostatistics, 18(2):381–396.
  24. e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. R package version 1.7-13.
  25. Inductive confidence machines for regression. In 13th European Conference on Machine Learning (ECML 2002), pages 345–356. Springer.
  26. A unified treatment of multiple testing with prior knowledge using the p-filter. The Annals of Statistics, 47(5):2790–2821.
  27. A burden shared is a burden halved: A fairness-adjusted approach to classification. arXiv preprint arXiv:2110.05720.
  28. Exact and approximate stepdown methods for multiple hypothesis testing. J. Amer. Statist. Assoc., 100(469):94–108.
  29. Conformalized quantile regression. Advances in neural information processing systems, 32.
  30. Exact calculations for false discovery proportion with application to least favorable configurations. Ann. Statist., 39(1):584–612.
  31. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223–234.
  32. Sarkar, S. K. (2008). On methods controlling the false discovery rate. Sankhya, Ser. A, 70:135–168.
  33. Conformal prediction using conditional histograms. Advances in Neural Information Processing Systems, 34:6304–6315.
  34. Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol., 64(3):479–498.
  35. Algorithmic learning in a random world. Springer.
  36. Controlling fsr in selective classification. arXiv preprint arXiv:2311.03811.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets