Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Label Noise Robustness of Conformal Prediction (2209.14295v3)

Published 28 Sep 2022 in cs.LG, cs.AI, math.ST, stat.ME, stat.ML, and stat.TH

Abstract: We study the robustness of conformal prediction, a powerful tool for uncertainty quantification, to label noise. Our analysis tackles both regression and classification problems, characterizing when and how it is possible to construct uncertainty sets that correctly cover the unobserved noiseless ground truth labels. We further extend our theory and formulate the requirements for correctly controlling a general loss function, such as the false negative proportion, with noisy labels. Our theory and experiments suggest that conformal prediction and risk-controlling techniques with noisy labels attain conservative risk over the clean ground truth labels whenever the noise is dispersive and increases variability. In other adversarial cases, we can also correct for noise of bounded size in the conformal prediction algorithm in order to ensure achieving the correct risk of the ground truth labels without score or data regularity.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Label noise types and their effects on deep learning. arXiv preprint arXiv:2003.10471, 2020.
  2. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
  3. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint, 2021. arXiv:2110.01052.
  4. Conformal risk control. arXiv preprint arXiv:2208.02814, 2022.
  5. Learning from noisy examples. Machine Learning, 2(4):343–370, 1988.
  6. On the sample complexity of noise-tolerant learning. Information Processing Letters, 57(4):189–195, 1996.
  7. Rina Foygel Barber. Is distribution-free inference possible for binary regression? arXiv:2004.09477, 2020.
  8. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486 – 507, 2021.
  9. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415, 2022.
  10. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6), September 2021. ISSN 0004-5411.
  11. Capturing human categorization of natural images by combining deep networks and cognitive models. Nature communications, 11(1):1–14, 2020.
  12. bio. Physicochemical properties of protein tertiary structure data set. https://archive.ics.uci.edu/ml/datasets/Physicochemical+Properties+of+Protein+Tertiary+Structure. Accessed: January, 2019.
  13. Predictive inference with weak supervision. arXiv preprint arXiv:2201.08315, 2022.
  14. How many labelers do you have? a closer look at gold-standard labels. arXiv preprint arXiv:2206.12041, 2022.
  15. Pranet: Parallel reverse attention network for polyp segmentation. In International conference on medical image computing and computer-assisted intervention, pages 263–273. Springer, 2020.
  16. Achieving risk control in online learning settings. arXiv preprint arXiv:2205.09095, 2022.
  17. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845–869, 2013.
  18. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
  19. Adversarially robust conformal prediction. In International Conference on Learning Representations, 2021.
  20. Adaptive conformal inference under distribution shift. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  21. Deep bilevel learning. In Proceedings of the European conference on computer vision (ECCV), pages 618–633, 2018.
  22. Learning deep networks from noisy labels with dropout regularization. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 967–972. IEEE, 2016.
  23. Visual aesthetic quality assessment with a regression model. In 2015 IEEE International Conference on Image Processing (ICIP), pages 1583–1587. IEEE, 2015.
  24. Robust learning of multi-label classifiers under label noise. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pages 90–97. 2020.
  25. Binary classification with corrupted labels. Electronic Journal of Statistics, 16(1):1367 – 1392, 2022.
  26. Distribution-free prediction sets. Journal of the American Statistical Association, 108(501):278–287, 2013.
  27. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018.
  28. Microsoft COCO: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. doi: 10.1007/978-3-319-10602-1˙48.
  29. Dimensionality-driven learning with noisy labels. In International Conference on Machine Learning, pages 3355–3364. PMLR, 2018.
  30. meps_19. Medical expenditure panel survey, panel 19. https://meps.ahrq.gov/mepsweb/data_stats/download_data_files_detail.jsp?cboPufNumber=HC-181. Accessed: January, 2019.
  31. Ava: A large-scale database for aesthetic visual analysis. In 2012 IEEE conference on computer vision and pattern recognition, pages 2408–2415. IEEE, 2012.
  32. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617–9626, 2019.
  33. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence, pages 844–853. PMLR, 2021.
  34. Tresnet: High performance gpu-dedicated architecture. In proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1400–1409, 2021.
  35. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32, pages 3543–3553. 2019.
  36. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, pages 3581–3591, 2020.
  37. End-to-end deep prototype and exemplar models for predicting human behavior. arXiv preprint arXiv:2007.08723, 2020.
  38. NIMA: Neural image assessment. IEEE transactions on image processing, 27(8):3998–4011, 2018.
  39. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11244–11253, 2019.
  40. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32, pages 2530–2540. 2019.
  41. Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74(1-2):9–28, 2015.
  42. Machine-learning applications of algorithmic randomness. In International Conference on Machine Learning, pages 444–453, 1999.
  43. Algorithmic Learning in a Random World. Springer, New York, NY, USA, 2005.
  44. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TBWA6PLJZQm.
  45. L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise. Advances in neural information processing systems, 32, 2019.
  46. Learning to recover 3d scene shape from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 204–213, 2021.
  47. Iterative cross learning on noisy labels. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 757–765. IEEE, 2018.
  48. Evaluating multi-label classifiers with noisy labels. arXiv preprint arXiv:2102.08427, 2021.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets