Papers
Topics
Authors
Recent
2000 character limit reached

Conformal Classification with Equalized Coverage for Adaptively Selected Groups (2405.15106v2)

Published 23 May 2024 in stat.ML and cs.LG

Abstract: This paper introduces a conformal inference method to evaluate uncertainty in classification by generating prediction sets with valid coverage conditional on adaptively chosen features. These features are carefully selected to reflect potential model limitations or biases. This can be useful to find a practical compromise between efficiency -- by providing informative predictions -- and algorithmic fairness -- by ensuring equalized coverage for the most sensitive groups. We demonstrate the validity and effectiveness of this method on simulated and real data sets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Deep learning models for medical diagnosis. Journal of Medical Imaging, 7(2):021008, 2020.
  2. J. K. Brown et al. Using machine learning for job application screening: A comparative analysis. In International Conference on Artificial Intelligence in HR, pages 12–21, 2021.
  3. Interpretable classification models for recidivism prediction. J. R. Stat. Soc. (A), 180(3):689–722, 2017.
  4. Nested conformal prediction sets for classification with applications to probation data. Ann. Appl. Stat., 17(1):761–785, 2023.
  5. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition, pages 427–436, 2015.
  6. On calibration of modern neural networks. In Int. Conf. Mach. Learn., pages 1321–1330. PMLR, 2017.
  7. Fairness through awareness. In Innovations in Theoretical Computer Science Conference, page 214–226. Association for Computing Machinery, 2012.
  8. Learning fair representations. In Int. Conf. Mach. Learn., 2013.
  9. Equality of opportunity in supervised learning. In Adv. Neural. Inf. Process. Syst., volume 29, 2016.
  10. Improving fairness in criminal justice algorithmic risk assessments using optimal transport and conformal prediction sets. Sociological Methods & Research, 2021.
  11. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Adv. Neural Inf. Process. Syst., 32, 2019.
  12. Inductive confidence machines for regression. In Eur. Conf. Mach. Learn., pages 345–356, 2002.
  13. Algorithmic learning in a random world. Springer, 2005.
  14. Distribution-free predictive inference for regression. J. Am. Stat. Assoc., 113(523):1094–1111, 2018.
  15. Deborah Hellman. Measuring algorithmic fairness. Virginia Law Review, 106(4):811–866, 2020.
  16. Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2):153–163, 2017.
  17. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review, 2020a.
  18. Neyman-Pearson and equal opportunity: when efficiency meets fairness in classification. arXiv preprint arXiv:2310.01009, 2023.
  19. The limits of distribution-free conditional predictive inference. Information and Inference, 10(2):455–482, 2021.
  20. Distribution-free inference for regression: discrete, continuous, and in between. Adv. Neural Inf. Process. Syst., 34:7448–7459, 2021.
  21. Testing for outliers with conformal p-values. Ann. Stat., 51(1):149 – 178, 2023.
  22. Adaptive novelty detection with false discovery rate guarantee. Ann. Stat., 52(1):157–183, 2024.
  23. Integrative conformal p-values for out-of-distribution testing with labelled outliers. J. R. Stat. Soc. (B), page qkad138, 2024a.
  24. Distribution-free prediction sets. J. Am. Stat. Assoc., 108(501):278–287, 2013.
  25. Least ambiguous set-valued classifiers with bounded error levels. J. Am. Stat. Assoc., 114(525):223–234, 2019.
  26. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence, pages 844–853. PMLR, 2021.
  27. Classification with valid and adaptive coverage. Adv. Neural Inf. Process. Syst., 33, 2020b.
  28. Uncertainty sets for image classifiers using conformal prediction. In Int. Conf. Learn. Represent., 2021.
  29. Distribution-free prediction bands for non-parametric regression. J. R. Stat. Soc. (B), 76(1):71–96, 2014.
  30. Conformalized quantile regression. In Adv. Neural Inf. Process. Syst., pages 3538–3548, 2019.
  31. Conformal prediction using conditional histograms. Adv. Neural Inf. Process. Syst., 34, 2021.
  32. Training uncertainty-aware classifiers with conformalized deep learning. In Adv. Neural Inf. Process. Syst., volume 35, 2022.
  33. Conformal inference is (almost) free for neural networks trained with early stopping. In Int. Conf. Mach. Learn., 2023.
  34. Conformal prediction beyond exchangeability. Ann. Stat., 51(2):816–845, 2023.
  35. Conformal prediction under covariate shift. Adv. Neural Inf. Process. Syst., 32, 2019.
  36. Conformal inference for online prediction with arbitrary distribution shifts. arXiv preprint arXiv:2208.08401, 2022.
  37. Doubly robust calibration of prediction sets under covariate shift. J. R. Stat. Soc. (B), page qkae009, 2024.
  38. Structured conformal inference for matrix completion with applications to group recommender systems. arXiv preprint arXiv:2404.17561, 2024b.
  39. Fair conformal predictors for applications in medical imaging. AAAI Conference on Artificial Intelligence, 36:12008–12016, 06 2022.
  40. Equal opportunity of coverage in fair regression. In Adv. Neural Inf. Process. Syst, 2023.
  41. Conformalized fairness via quantile regression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Adv. Neural Inf. Process. Syst., 2022.
  42. Mondrian confidence machine. Technical report, Royal Holloway, University of London, 2003. On-line Compression Modelling project.
  43. Class-conditional conformal prediction with many classes. Adv. Neural Inf. Process. Syst., 36, 2024.
  44. Bias reduction through conditional conformal prediction. Intell. Data Anal., 19(6):1355–1375, nov 2015a. ISSN 1088-467X.
  45. Batch multivalid conformal prediction. In Int. Conf. Learn. Represent., 2023.
  46. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.
  47. Statistical inference for fairness auditing. arXiv preprint arXiv:2305.03712, 2023.
  48. Confidence on the focal: Conformal prediction with selection-conditional coverage. arXiv preprint arXiv:2403.03868, 2024.
  49. Vladislav Rajkovic. Nursery. UCI Machine Learning Repository, 1997. DOI: https://doi.org/10.24432/C5P88W, License: CC-BY 4.0.
  50. Adult. UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5XW20, License: CC-BY 4.0.
  51. A tutorial on conformal inference. J. Mach. Learn. Res., 9(3), 2008.
  52. Bias reduction through conditional conformal prediction. Intell. Data Anal., 19(6):1355–1375, 11 2015b.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Video Overview

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 3 tweets with 7 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free