Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
91 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
o3 Pro
5 tokens/sec
GPT-4.1 Pro
15 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
Gemini 2.5 Flash Deprecated
12 tokens/sec
2000 character limit reached

Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models (2312.15297v1)

Published 23 Dec 2023 in cs.LG, cs.CV, and stat.ML

Abstract: Deep Neural Networks (DNNs) are powerful tools for various computer vision tasks, yet they often struggle with reliable uncertainty quantification - a critical requirement for real-world applications. Bayesian Neural Networks (BNN) are equipped for uncertainty estimation but cannot scale to large DNNs that are highly unstable to train. To address this challenge, we introduce the Adaptable Bayesian Neural Network (ABNN), a simple and scalable strategy to seamlessly transform DNNs into BNNs in a post-hoc manner with minimal computational and training overheads. ABNN preserves the main predictive properties of DNNs while enhancing their uncertainty quantification abilities through simple BNN adaptation layers (attached to normalization layers) and a few fine-tuning steps on pre-trained models. We conduct extensive experiments across multiple datasets for image classification and semantic segmentation tasks, and our results demonstrate that ABNN achieves state-of-the-art performance without the computational budget typically associated with ensemble methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (96)
  1. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 2020.
  2. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In ICLR, 2019.
  3. Layer normalization. In NeurIPSW, 2016.
  4. Variational inference: A review for statisticians. JASA, 2017.
  5. Weight uncertainty in neural network. In ICML, 2015.
  6. On last-layer algorithms for classification: Decoupling representation from uncertainty estimation. arXiv preprint arXiv:2001.08049, 2020.
  7. Segmentmeifyoucan: A benchmark for anomaly segmentation. In NeurIPS, 2021.
  8. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
  9. Beyond first-order uncertainty estimation with evidential models for open-world recognition. In ICMLW, 2021.
  10. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
  11. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR, 2020.
  12. Laplace redux–effortless Bayesian deep learning. In NeurIPS, 2021a.
  13. Bayesian deep learning via subnetwork inference. In ICML, 2021b.
  14. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
  15. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  16. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In ICML, 2018.
  17. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  18. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  19. Masksembles for uncertainty estimation. In CVPR, 2021.
  20. Efficient and scalable bayesian neural nets with rank-1 factors. In ICML, 2020.
  21. William Feller. An introduction to probability theory and its applications, Volume 2. John Wiley & Sons, 1991.
  22. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
  23. Tradi: Tracking deep neural network weight distributions. In ECCV, 2020.
  24. Muad: Multiple uncertainties for autonomous driving, a benchmark for multiple uncertainty types and tasks. In BMVC, 2022.
  25. Encoding the latent posterior of bayesian neural networks for uncertainty quantification. T-PAMI, 2023.
  26. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
  27. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 2023.
  28. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science, 2020.
  29. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 1990.
  30. Training independent subnetworks for robust prediction. In ICLR, 2021.
  31. Deep residual learning for image recognition. In CVPR, 2016.
  32. Mask r-cnn. In ICCV, 2017.
  33. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019.
  34. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
  35. A benchmark for anomaly segmentation. arXiv preprint arXiv:1911.11132, 2019.
  36. Jacob steinhardt et justin gilmer. the many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021a.
  37. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021b.
  38. Probabilistic backpropagation for scalable learning of bayesian neural networks. In ICML, 2015.
  39. Stephen C Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 1996.
  40. Wide bayesian neural networks have a simple weight posterior: theory and accelerated sampling. In ICML, pages 8926–8945. PMLR, 2022.
  41. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 2021.
  42. Openclip, 2021.
  43. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
  44. What are bayesian neural network posteriors really like? In ICML, 2021.
  45. An introduction to variational methods for graphical models. ML, 1999.
  46. What uncertainties do we need in bayesian deep learning for computer vision? NeurIPS, 2017.
  47. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  48. Being bayesian, even just a bit, fixes overconfidence in relu networks. In ICML, 2020.
  49. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, MIT, 2009.
  50. Imagenet classification with deep convolutional networks. In NeurIPS, 2012.
  51. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
  52. Deep learning under privileged information using heteroscedastic dropout. In CVPR, pages 8886–8895, 2018.
  53. A symmetry-aware exploration of bayesian neural network posteriors. arXiv preprint arXiv:2310.08287, 2023a.
  54. Packed-ensembles for efficient uncertainty estimation. In ICLR, 2023b.
  55. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, 2018.
  56. Coda: A real-world road corner case dataset for object detection in autonomous driving. In ECCV, 2022.
  57. A convnet for the 2020s. In CVPR, 2022.
  58. David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 1992.
  59. A simple baseline for bayesian uncertainty in deep learning. In NeurIPS, 2019.
  60. Predictive uncertainty estimation via prior networks. In NeurIPS, 2018.
  61. Obtaining well calibrated probabilities using bayesian binning. In AAAI, 2015.
  62. Eric Thomas Nalisnick. On priors for Bayesian neural networks. University of California, Irvine, 2018.
  63. Diverse imagenet models transfer better. arXiv preprint arXiv:2204.09134, 2022.
  64. Radford M Neal. Bayesian learning for neural networks. 2012.
  65. Reading digits in natural images with unsupervised feature learning. In NeurIPSW, 2011.
  66. Spurious features everywhere-large-scale detection of harmful spurious features in imagenet. In ICCV, 2023.
  67. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In NeurIPS, 2019.
  68. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  69. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018.
  70. You only look once: Unified, real-time object detection. In CVPR, 2016.
  71. Imagenet-21k pretraining for the masses, 2021.
  72. A scalable laplace approximation for neural networks. In ICLR, 2018.
  73. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  74. Uncertainty-guided source-free domain adaptation. In ECCV, 2022.
  75. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
  76. Learning to rank short text pairs with convolutional deep neural networks. In SIGIR, 2015.
  77. Better aggregation in test-time augmentation. In ICCV, pages 1214–1223, 2021.
  78. Efficient improvement of classification accuracy via selective test-time augmentation. Information Sciences, 642:119148, 2023.
  79. Rethinking the inception architecture for computer vision. In CVPR, 2016.
  80. Accurate approximations for posterior moments and marginal densities. Journal of the American Statistical Association, 1986.
  81. Consistent inference of probabilities in layered networks: predictions and generalizations. In IJCNN, 1989.
  82. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.07411, 2022.
  83. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
  84. ViM: Out-of-distribution with virtual-logit matching. In CVPR, 2022.
  85. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
  86. BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. In ICLR, 2019.
  87. Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
  88. Resnet strikes back: An improved training procedure in timm. In NeurIPSW, 2021.
  89. Bayesian deep learning and a probabilistic perspective of generalization. NeurIPS, 2020.
  90. Window-based early-exit cascades for uncertainty estimation: When deep ensembles are more efficient than single models. In ICCV, 2023.
  91. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
  92. Cutmix: Regularization strategy to train strong classifiers with localizable features. In CVPR, 2019.
  93. Explainability of deep vision-based autonomous driving systems: Review and challenges. IJCV, 2022.
  94. Wide residual networks. In BMVC, 2016.
  95. Sigmoid loss for language image pre-training. In ICCV, 2023.
  96. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets