Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
96 tokens/sec
GPT-4o
11 tokens/sec
Gemini 2.5 Pro Pro
48 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Conformal Prediction for Deep Classifier via Label Ranking (2310.06430v2)

Published 10 Oct 2023 in cs.LG, cs.CV, math.ST, and stat.TH

Abstract: Conformal prediction is a statistical framework that generates prediction sets containing ground-truth labels with a desired coverage guarantee. The predicted probabilities produced by machine learning models are generally miscalibrated, leading to large prediction sets in conformal prediction. To address this issue, we propose a novel algorithm named $\textit{Sorted Adaptive Prediction Sets}$ (SAPS), which discards all the probability values except for the maximum softmax probability. The key idea behind SAPS is to minimize the dependence of the non-conformity score on the probability values while retaining the uncertainty information. In this manner, SAPS can produce compact prediction sets and communicate instance-wise uncertainty. Extensive experiments validate that SAPS not only lessens the prediction sets but also broadly enhances the conditional coverage rate of prediction sets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
  2. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
  3. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b.
  4. Efficient and differentiable conformal prediction with general function classes. arXiv preprint arXiv:2202.11091, 2022.
  5. Conformal prediction for reliable machine learning: theory, adaptations and applications. Newnes, 2014.
  6. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1–34, 2021.
  7. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149–178, 2023.
  8. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613–1622. PMLR, 2015.
  9. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016.
  10. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.  1721–1730, 2015.
  11. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. arXiv preprint arXiv:2004.10181, 2020.
  12. Learning prediction intervals for regression: Generalization and calibration. In International Conference on Artificial Intelligence and Statistics, pp.  820–828. PMLR, 2021.
  13. Batch-ensemble stochastic neural networks for out-of-distribution detection. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  1–5. IEEE, 2023.
  14. Training conformal predictors. In Conformal and Probabilistic Prediction and Applications, pp.  55–64. PMLR, 2020.
  15. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.  248–255. Ieee, 2009.
  16. Class-conditional conformal prediction with many classes. arXiv preprint arXiv:2306.09335, 2023.
  17. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  18. Conformal prediction is robust to label noise. arXiv preprint arXiv:2209.14295, 2022a.
  19. Training uncertainty-aware classifiers with conformalized deep learning. Advances in Neural Information Processing Systems, 35:22380–22395, 2022b.
  20. Few-shot conformal prediction with auxiliary tasks. In International Conference on Machine Learning, pp. 3329–3339. PMLR, 2021.
  21. Conformal prediction sets with limited false positives. In International Conference on Machine Learning, pp. 6514–6532. PMLR, 2022.
  22. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050–1059. PMLR, 2016.
  23. Adversarially robust conformal prediction. In International Conference on Learning Representations, 2021.
  24. Improving uncertainty quantification of deep classifiers via neighborhood conformal prediction: Novel algorithm and theoretical analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.  7722–7730. AAAI Press, 2023.
  25. Prediction and outlier detection in classification problems. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(2):524–546, 2022.
  26. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321–1330. PMLR, 2017.
  27. Learning multiple layers of features from tiny images. 2009.
  28. Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404, 2023.
  29. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71–96, 2014.
  30. Bias reduction through conditional conformal prediction. Intelligent Data Analysis, 19(6):1355–1375, 2015.
  31. Federated conformal predictors for distributed uncertainty quantification. In International Conference on Machine Learning, pp. 22942–22964. PMLR, 2023.
  32. Harris Papadopoulos. A cross-conformal predictor for multi-label classification. In Artificial Intelligence Applications and Innovations: AIAI 2014 Workshops: CoPA, MHDW, IIVC, and MT4BD, Rhodes, Greece, September 19-21, 2014. Proceedings 10, pp.  241–250. Springer, 2014.
  33. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence, pp.  844–853. PMLR, 2021.
  34. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.
  35. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389–5400. PMLR, 2019.
  36. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928, 2023.
  37. Conformalized quantile regression. Advances in Neural Information Processing Systems, 32, 2019.
  38. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581–3591, 2020.
  39. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223–234, 2019.
  40. Adaptive conformal classification with noisy labels. arXiv preprint arXiv:2309.05092, 2023.
  41. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
  42. Applications of class-conditional conformal predictor in multi-class classification. In 2013 12th International Conference on Machine Learning and Applications, volume 1, pp.  235–239, 2013. doi: 10.1109/ICMLA.2013.48.
  43. Ralph C Smith. Uncertainty quantification: theory, implementation, and applications, volume 12. Siam, 2013.
  44. Learning optimal conformal classifiers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  45. Conformal prediction under covariate shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  46. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347–10357. PMLR, 2021.
  47. Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian Conference on Machine Learning, pp.  475–490. PMLR, 2012.
  48. Algorithmic learning in a random world, volume 29. Springer, 2005.
Citations (16)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now