Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Credal Wrapper of Model Averaging for Uncertainty Estimation in Classification (2405.15047v2)

Published 23 May 2024 in cs.LG and cs.AI

Abstract: This paper presents an innovative approach, called credal wrapper, to formulating a credal set representation of model averaging for Bayesian neural networks (BNNs) and deep ensembles (DEs), capable of improving uncertainty estimation in classification tasks. Given a finite collection of single predictive distributions derived from BNNs or DEs, the proposed credal wrapper approach extracts an upper and a lower probability bound per class, acknowledging the epistemic uncertainty due to the availability of a limited amount of distributions. Such probability intervals over classes can be mapped on a convex set of probabilities (a credal set) from which, in turn, a unique prediction can be obtained using a transformation called intersection probability transformation. In this article, we conduct extensive experiments on several out-of-distribution (OOD) detection benchmarks, encompassing various dataset pairs (CIFAR10/100 vs SVHN/Tiny-ImageNet, CIFAR10 vs CIFAR10-C, CIFAR100 vs CIFAR100-C and ImageNet vs ImageNet-O) and using different network architectures (such as VGG16, ResNet-18/50, EfficientNet B2, and ViT Base). Compared to the BNN and DE baselines, the proposed credal wrapper method exhibits superior performance in uncertainty estimation and achieves a lower expected calibration error on corrupted data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35:33646–33660, 2022.
  2. Disaggregated total uncertainty measure for credal sets. International Journal of General Systems, 35(1):29–44, 2006.
  3. A non-specificity measure for convex sets of probability distributions. International journal of uncertainty, fuzziness and knowledge-based systems, 8(03):357–367, 2000.
  4. Benchmarking Bayesian deep learning on diabetic retinopathy detection tasks. In Proceedings of Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  5. Weight uncertainty in neural network. In Proceedings of the International Conference on Machine Learning, pages 1613–1622. PMLR, 2015.
  6. Using probability trees to compute marginals with imprecise probabilities. International Journal of Approximate Reasoning, 29(1):1–46, 2002.
  7. Imprecise Bayesian neural networks. arXiv preprint arXiv:2302.09656, 2023.
  8. Conservative uncertainty estimation by fitting prior networks. In International Conference on Learning Representations, 2019.
  9. Bayesian networks with imprecise probabilities: Theory and application to classification. Data Mining: Foundations and Intelligent Paradigms: Volume 1: Clustering, Association and Classification, pages 49–93, 2012.
  10. Learning reliable classifiers from small or incomplete data sets: The naive credal classifier 2. Journal of Machine Learning Research, 9(4), 2008.
  11. Fabio Cuzzolin. Credal semantics of bayesian transformations in terms of probability intervals. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 40(2):421–432, 2009.
  12. Fabio Cuzzolin. The geometry of uncertainty: The geometry of imprecise probabilities. Springer Nature, 2020.
  13. Fabio Cuzzolin. The intersection probability: betting with probability intervals. arXiv preprint arXiv:2201.01729, 2022.
  14. Laplace Redux - Effortless Bayesian Deep Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 20089–20103. Curran Associates, Inc., 2021.
  15. Probability intervals: A tool for uncertain reasoning. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 02:167–196, June 1994.
  16. Bruno De Finetti. Theory of probability: A critical introductory treatment, volume 6. John Wiley & Sons, 2017.
  17. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  18. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184–1193. PMLR, 2018.
  19. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  20. Consonant approximations of belief functions. International Journal of Approximate Reasoning, 4(5-6):419–449, 1990.
  21. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning, pages 1050–1059. PMLR, 2016.
  22. A survey of uncertainty in deep neural networks. ArXiv Preprint ArXiv:2107.03342, 2021.
  23. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017.
  24. Decision making with interval probabilities. European Journal of Operational Research, 203(2):444–454, 2010.
  25. Evaluating scalable Bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 318–319, 2020.
  26. Bayesian deep ensembles via the neural tangent kernel. Advances in neural information processing systems, 33:1010–1022, 2020.
  27. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  28. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
  29. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
  30. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262–15271, 2021.
  31. Fast predictive uncertainty for classification with Bayesian deep networks. In James Cussens and Kun Zhang, editors, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pages 822–832. PMLR, 01–05 Aug 2022.
  32. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1):1593–1623, 2014.
  33. Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison. In Proceedings of the Uncertainty in Artificial Intelligence, pages 548–557. PMLR, 2022.
  34. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457–506, 2021.
  35. Conformalized credal set predictors. arXiv preprint arXiv:2402.10723, 2024.
  36. Hands-on Bayesian neural networks—A tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29–48, 2022.
  37. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 2017.
  38. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.
  39. Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
  40. CIFAR-10 (Canadian Institute For Advanced Research). Technical report, CIFAR, 2009.
  41. Bayesian hypernetworks. ArXiv Preprint ArXiv:1710.04759, 2017.
  42. Henry E Kyburg Jr. Bayesian and non-bayesian evidential updating. Artificial intelligence, 31(3):271–293, 1987.
  43. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
  44. Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE Transactions on Information Technology in Biomedicine, 15(1):93–99, 2010.
  45. Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  46. Isaac Levi. The enterprise of knowledge: An essay on knowledge, credal probability, and chance. MIT press, 1980.
  47. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498–7512, 2020.
  48. Credal sum-product networks. In Proceedings of the Tenth International Symposium on Imprecise Probability: Theories and Applications, pages 205–216. PMLR, 2017.
  49. Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise. Medical Image Analysis, 89:102914, 2023.
  50. Dropconnect is effective in modeling uncertainty of Bayesian deep networks. Scientific Reports, 11(1):1–14, 2021.
  51. Variational dropout sparsifies deep neural networks. In Proceedings of the International Conference on Machine Learning, pages 2498–2507. PMLR, 2017.
  52. Credal sets representable by reachable probability intervals and belief functions. International Journal of Approximate Reasoning, 129:84–102, 2021.
  53. Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24384–24394, 2023.
  54. Radford M Neal et al. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2(11):2, 2011.
  55. Hung T Nguyen. On random sets and belief functions. Journal of Mathematical Analysis and Applications, 65(3):531–542, 1978.
  56. Measuring calibration in deep learning. In CVPR workshops, volume 2, 2019.
  57. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
  58. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), pages 372–387. IEEE, 2016.
  59. Tractable function-space variational inference in bayesian neural networks. Advances in Neural Information Processing Systems, 35:22686–22698, 2022.
  60. Is the volume of a credal set a good measure for epistemic uncertainty? In Uncertainty in Artificial Intelligence, pages 1795–1804. PMLR, 2023.
  61. Is the volume of a credal set a good measure for epistemic uncertainty? In Accepted for Conference on Uncertainty in Artificial Intelligence (UAI 2023), pages 0–12, 2023.
  62. Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16–29, 2014.
  63. Ensemble-based uncertainty quantification: Bayesian versus credal inference. In PROCEEDINGS 31. WORKSHOP COMPUTATIONAL INTELLIGENCE, volume 25, page 63, 2021.
  64. K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
  65. Efficientnetv2: Smaller models and faster training. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10096–10106. PMLR, 18–24 Jul 2021.
  66. Bjørnar Tessem. Interval probability propagation. International Journal of Approximate Reasoning, 7(3-4):95–120, 1992.
  67. All you need is a good functional prior for Bayesian deep learning. ArXiv Preprint ArXiv:2011.12829, 2020.
  68. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020.
  69. Peter Walley. Statistical reasoning with imprecise probabilities. 1991.
  70. Slope hybrid reliability analysis considering the uncertainty of probability-interval using three-parameter weibull distribution. Natural Hazards, 105:565–586, 2021.
  71. Flipout: Efficient pseudo-independent weight perturbations on mini-batches. In Proceedings of the International Conference on Learning Representations, 2018.
  72. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697–4708, 2020.
  73. Decision making under interval probabilities. International Journal of Approximate Reasoning, 22(3):195–215, 1999.
  74. Lotfi Asker Zadeh. Fuzzy sets as a basis for a theory of possibility. Fuzzy sets and systems, 1(1):3–28, 1978.
  75. Marco Zaffalon. The naive credal classifier. Journal of statistical planning and inference, 105(1):5–21, 2002.
  76. Evaluating adversarial attacks on driving safety in vision-based autonomous vehicles. IEEE Internet of Things Journal, 9(5):3443–3456, 2021.
  77. Learning from the wisdom of crowds by minimax entropy. Advances in Neural Information Processing Systems, 25, 2012.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets