Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures? (2209.03302v2)

Published 7 Sep 2022 in cs.LG

Abstract: The quantification of aleatoric and epistemic uncertainty in terms of conditional entropy and mutual information, respectively, has recently become quite common in machine learning. While the properties of these measures, which are rooted in information theory, seem appealing at first glance, we identify various incoherencies that call their appropriateness into question. In addition to the measures themselves, we critically discuss the idea of an additive decomposition of total uncertainty into its aleatoric and epistemic constituents. Experiments across different computer vision tasks support our theoretical findings and raise concerns about current practice in uncertainty quantification.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Deep Ensembles Work, But Are They Necessary?, 2022.
  2. An Introduction to MCMC for Machine Learning. Machine Learning, 50:5–43, 2003.
  3. R. B. Ash. Information Theory. Dover Publications, 1965.
  4. Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning. In ICLR, 2020.
  5. The Power of Ensembles for Active Learning in Image Classification. In CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.
  6. L. Breiman. Random Forests. Machine Learning, 45(1):5–32, 2001.
  7. A. Bronevich and G. J. Klir. Axioms for uncertainty measures on belief functions and credal sets. In NAFIPS, pages 1–6. IEEE, 2008.
  8. A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis. Medical Image Analysis, 71, 2021.
  9. Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts. In NeurIPS, pages 1103–1130, 2020.
  10. Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning, 2022.
  11. Elements of Information Theory. Wiley, 2 edition, 2006.
  12. Laplace Redux – Effortless Bayesian Deep Learning, 2021.
  13. S. Depeweg. Modeling Epistemic and Aleatoric Uncertainty with Bayesian Neural Networks and Latent Variable. PhD thesis, TU München, 2019.
  14. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning, 2018.
  15. Representing partial ignorance. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 26(3):361–377, 1996.
  16. Y. Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
  17. Deep Bayesian Active Learning with Image Data, 2017.
  18. Bayesian Data Analysis. CRC Press, 2021.
  19. Bayesian Active Learning for Classification and Preference Learning, 2011.
  20. Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks. In ICPR, 2020.
  21. E. Hüllermeier and W. Waegeman. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods. Machine Learning, 2021.
  22. Quantification of Credal Uncertainty in Machine Learning: A Critical Analysis and Empirical Comparison. In UAI, 2022.
  23. A. Kendall and Y. Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In NeurIPS, 2017.
  24. Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Sampling, 2022.
  25. Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable? In ICML, pages 5707–5718, 2021.
  26. Being a Bit Frequentist Improves Bayesian Neural Networks. In AISTATS, 2022.
  27. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  28. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In NeurIPS, 2017.
  29. Gradient-Based Learning Applied to Document Recognition. In Proceedings of the IEEE, 1998.
  30. SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning. In ICML, 2021.
  31. Lightning AI. PyTorch Lightning: Train and Deploy PyTorch at Scale, 2023.
  32. A. Malinin. Uncertainty Estimation in Deep Learning with Application to Spoken Language Assessment. PhD thesis, University of Cambridge, 2019.
  33. A. Malinin and M. Gales. Predictive Uncertainty Estimation via Prior Networks. In NeurIPS, 2018.
  34. Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control, 2018.
  35. DropConnect is effective in modeling uncertainty of Bayesian deep networks. Scientific Reports, 11, 2021.
  36. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  37. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019.
  38. Uncertainty measures for evidential reasoning I: A review. International Journal of Approximate Reasoning, 7(3-4):165–183, 1992.
  39. Uncertainty measures for evidential reasoning ii: A new measure of total uncertainty. International Journal of Approximate Reasoning, 8(1):1–16, 1993.
  40. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019.
  41. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
  42. Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons, 2022.
  43. Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16–29, 2014.
  44. C. E. Shannon. A Mathematical Theory of Communication. The Bell System Technical Journal, 27:379–423, 1948.
  45. Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates. In Proceedings of the 16th Conference of the EACL, pages 1698–1712. Association for Computational Linguistics, 2021.
  46. Second-moment loss: A novel regression objective for improved uncertainties. CoRR, abs/2012.12687, 2020.
  47. L. Smith and Y. Gal. Understanding Measures of Uncertainty for Adversarial Example Detection, 2018.
  48. M. Tan and Q. V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In ICML, 2019.
  49. FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation. Accepted at the Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022.
  50. A. G. Wilson. The Case for Bayesian Deep Learning, 2020.
  51. Stochastic Control for Bayesian Neural Network Training. Entropy, 24, 2022.
  52. J. O. Woo. Analytic Mutual Information in Bayesian Neural Networks, 2022.
  53. Ensemble Approaches for Uncertainty in Spoken Language Assessment. In Interspeech 2020, pages 3860–3864. ISCA, 2020.
Citations (49)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets