2000 character limit reached
Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules (2404.12215v2)
Published 18 Apr 2024 in cs.LG and stat.ML
Abstract: Uncertainty representation and quantification are paramount in machine learning and constitute an important prerequisite for safety-critical applications. In this paper, we propose novel measures for the quantification of aleatoric and epistemic uncertainty based on proper scoring rules, which are loss functions with the meaningful property that they incentivize the learner to predict ground-truth (conditional) probabilities. We assume two common representations of (epistemic) uncertainty, namely, in terms of a credal set, i.e. a set of probability distributions, or a second-order distribution, i.e., a distribution over probability distributions. Our framework establishes a natural bridge between these representations. We provide a formal justification of our approach and introduce new measures of epistemic and aleatoric uncertainty as concrete instantiations.
- Disaggregated total uncertainty measure for credal sets. Int. Journal of General Systems, 35(1).
- A non-specificity measure for convex sets of probability distributions. Int. Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 8:357–367.
- Pitfalls of epistemic uncertainty quantification through loss minimisation. In Advances in Neural Information Processing Systems.
- Csiszár, I. (2008). Axiomatic characterizations of information measures. Entropy, 10:261–273.
- De Finetti, B. (1980). Foresight: It’s logical laws, it’s subjective sources. In Kyburg, H. and Smokler, H., editors, Studies in Subjective Probability. R.E. Krieger, New York.
- Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184–1193. PMLR.
- Representing partial ignorance. IEEE Transactions on Systems, Man and Cybernetics, Series A, 26(3):361–377.
- Strictly proper scoring rules, prediction, and estimation. Technical Report 463R, Department of Statistics, University of Washington.
- Hartley, R. (1928). Transmission of information. Bell Syst. Tech. Journal, 7(3):535–563.
- Hora, S. C. (1996). Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 54(2-3):217–223.
- Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745.
- Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457–506.
- On uncertainty, tempering, and data augmentation in bayesian classification. arXiv preprint arXiv:2203.16481.
- What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30.
- On the uniqueness of possibilistic measure of uncertainty and information. Fuzzy Sets and Systems, 24(2):197–219.
- Predictive uncertainty quantification via risk decompositions for strictly proper scoring rules. CoRR, abs/2402.10727.
- Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Appice, A., Rodrigues, P. P., Costa, V. S., Soares, C., Gama, J., and Jorge, A., editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I, volume 9284 of Lecture Notes in Computer Science, pages 68–85. Springer.
- Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE Transactions on Information Technology in Biomedicine, 15(1):93–99.
- Levi, I. (1974). On indeterminate probabilities. Journal of Philosophy, 71:391–418.
- Levi, I. (1980). The Enterprise of Knowledge. MIT Press, Cambridge.
- Mitchell, T. M. (1977). Version spaces: A candidate elimination approach to rule learning. In Proceedings of the 5th international joint conference on Artificial intelligence-Volume 1, pages 305–310.
- DropConnect is effective in modeling uncertainty of Bayesian networks. CoRR, abs/1906.04569.
- Rényi, A. (1970). Probability Theory. North-Holland, Amsterdam.
- Second-order uncertainty quantification: Variance-based measures. arXiv preprint arXiv:2401.00276.
- Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783–801.
- Introducing an improved information-theoretic measure of predictive uncertainty. CoRR, abs/2311.08309.
- Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16–29.
- Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533.
- Walley, P. (1991). Statistical Reasoning with Imprecise Probabilities. Chapman and Hall.
- Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence, pages 2282–2292. PMLR.
- Yager, R. (1983). Entropy and specificity in a mathematical theory of evidence. International Journal of General Systems, 9:249–260.
- Using random forest for reliable classification and cost-sensitive learning for medical diagnosis. BMC bioinformatics, 10(1):1–14.