Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty? (2306.09586v1)

Published 16 Jun 2023 in cs.LG, cs.AI, and stat.ML

Abstract: Adequate uncertainty representation and quantification have become imperative in various scientific disciplines, especially in machine learning and artificial intelligence. As an alternative to representing uncertainty via one single probability measure, we consider credal sets (convex sets of probability measures). The geometric representation of credal sets as $d$-dimensional polytopes implies a geometric intuition about (epistemic) uncertainty. In this paper, we show that the volume of the geometric representation of a credal set is a meaningful measure of epistemic uncertainty in the case of binary classification, but less so for multi-class classification. Our theoretical findings highlight the crucial role of specifying and employing uncertainty measures in machine learning in an appropriate way, and for being aware of possible pitfalls.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. A non-specificity measure for convex sets of probability distributions. International journal of uncertainty, fuzziness and knowledge-based systems, 8(03):357–367, 2000.
  2. Building classification trees using the total uncertainty criterion. International Journal of Intelligent Systems, 18(12):1215–1225, 2003.
  3. Maximum of entropy for credal sets. International journal of uncertainty, fuzziness and knowledge-based systems, 11(05):587–597, 2003.
  4. Additivity of uncertainty measures on credal sets. International Journal of General Systems, 34(6):691–713, 2005.
  5. Mathieu Anel. The Geometry of Ambiguity: An Introduction to the Ideas of Derived Geometry, volume 1, pages 505–553. Cambridge University Press, 2021.
  6. Introduction to imprecise probabilities. John Wiley & Sons, 2014.
  7. Convex bodies with few faces. Proceedings of the American Mathematical Society, 110(1):225–231, 1990.
  8. Pitfalls of epistemic uncertainty quantification through loss minimisation. In Advances in Neural Information Processing Systems, 2022.
  9. Isabelle Bloch. Some aspects of Dempster-Shafer evidence theory for classification of multi-modality medical images taking partial volume effect into account. Pattern Recognition Letters, 17(8):905–919, 1996.
  10. Axioms for uncertainty measures on belief functions and credal sets. In NAFIPS 2008-2008 Annual Meeting of the North American Fuzzy Information Processing Society, pages 1–6. IEEE, 2008.
  11. Measures of uncertainty for imprecise probabilities: an axiomatic approach. International journal of approximate reasoning, 51(4):365–390, 2010.
  12. Distances between probability distributions of different dimensions. IEEE Transactions on Information Theory, 2022.
  13. Imprecise Bayesian Neural Networks. arXiv preprint arXiv:2302.09656, 2023a.
  14. EpiC INN: Epistemic Curiosity Imprecise Neural Network. Technical report, University of Pennsylvania, Department of Computer and Information Science, 01 2023b.
  15. The sphere packing problem in dimension 24. Annals of Mathematics, 185(3):1017–1033, 2017.
  16. Learning reliable classifiers from small or incomplete data sets: The naive credal classifier 2. Journal of Machine Learning Research, 9(4), 2008.
  17. Bayesian networks with imprecise probabilities: Theory and application to classification. In Data Mining: Foundations and Intelligent Paradigms, pages 49–93. Springer, 2012.
  18. Examples of independence for imprecise probabilities. In Proceedings of the First International Symposium on Imprecise Probabilities and Their Applications (ISIPTA 1999), pages 121–130, 1999.
  19. Fabio Cuzzolin. The Geometry of Uncertainty. Artificial Intelligence: Foundations, Theory, and Algorithms. Springer Nature Switzerland, 2021.
  20. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184–1193. PMLR, 2018.
  21. A local search-based method for sphere packing problems. European Journal of Operational Research, 247:482–500, 2019.
  22. Stephen C Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 54(2-3):217–223, 1996.
  23. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457–506, 2021.
  24. Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison. In Uncertainty in Artificial Intelligence, pages 548–557. PMLR, 2022.
  25. Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? Available at arxiv:2209.03302, 2022.
  26. A new definition of entropy of belief functions in the Dempster–Shafer theory. International Journal of Approximate Reasoning, 92:49–65, 2018.
  27. On uncertainty, tempering, and data augmentation in bayesian classification. arXiv preprint arXiv:2203.16481, 2022.
  28. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
  29. Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE Transactions on Information Technology in Biomedicine, 15(1):93–99, 2010.
  30. Isaac Levi. The Enterprise of Knowledge. London : MIT Press, 1980.
  31. Karl Mosler. Zonoids and lift zonoids. In Multivariate Dispersion, Central Regions, and Depth: The Lift Zonoid Approach, volume 165 of Lecture Notes in Statistics, pages 25–78. New York : Springer, 2002.
  32. Uncertainty measures for evidential reasoning i: A review. International Journal of Approximate Reasoning, 7(3-4):165–183, 1992.
  33. Uncertainty measures for evidential reasoning ii: A new measure of total uncertainty. International Journal of Approximate Reasoning, 8(1):1–16, 1993.
  34. Forecasting with imprecise probabilities. International Journal of Approximate Reasoning, 53(8):1248–1261, 2012. Imprecise Probability: Theories and Applications (ISIPTA’11).
  35. Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16–29, 2014.
  36. Burr Settles. Active Learning Literature Survey. Technical report, University of Wisconsin-Madison, Department of Computer Sciences, 2009.
  37. Aleatoric and epistemic uncertainty with random forests. In Advances in Intelligent Data Analysis XVIII: 18th International Symposium on Intelligent Data Analysis, IDA 2020, Konstanz, Germany, April 27–29, 2020, Proceedings 18, pages 444–456. Springer, 2020.
  38. Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
  39. Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533, 2018.
  40. Kush R Varshney. Engineering safety in machine learning. In 2016 Information Theory and Applications Workshop (ITA), pages 1–5. IEEE, 2016.
  41. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big data, 5(3):246–255, 2017.
  42. Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
  43. Maryna S. Viazovska. The sphere packing problem in dimension 8. Annals of Mathematics, 185(3):991–1015, 2017.
  44. Peter Walley. Statistical Reasoning with Imprecise Probabilities, volume 42 of Monographs on Statistics and Applied Probability. London : Chapman and Hall, 1991.
  45. Peter Walley. Inferences from multinomial data: learning about a bag of marbles. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):3–34, 1996.
  46. Using random forest for reliable classification and cost-sensitive learning for medical diagnosis. BMC bioinformatics, 10(1):1–14, 2009.
  47. Marco Zaffalon. The naive credal classifier. Journal of statistical planning and inference, 105(1):5–21, 2002.
Citations (16)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now