A Framework for Variational Inference of Lightweight Bayesian Neural Networks with Heteroscedastic Uncertainties (2402.14532v1)
Abstract: Obtaining heteroscedastic predictive uncertainties from a Bayesian Neural Network (BNN) is vital to many applications. Often, heteroscedastic aleatoric uncertainties are learned as outputs of the BNN in addition to the predictive means, however doing so may necessitate adding more learnable parameters to the network. In this work, we demonstrate that both the heteroscedastic aleatoric and epistemic variance can be embedded into the variances of learned BNN parameters, improving predictive performance for lightweight networks. By complementing this approach with a moment propagation approach to inference, we introduce a relatively simple framework for sampling-free variational inference suitable for lightweight BNNs.
- David J. C. MacKay. A Practical Bayesian Framework for Backpropagation Networks. Neural Computation, 4(3):448–472, May 1992.
- Radford M. Neal. Bayesian Learning for Neural Networks, volume 118 of Lecture Notes in Statistics. Springer New York, New York, NY, 1996.
- What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
- Yarin Gal. Uncertainty in Deep Learning. 2016.
- Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA, 20–22 Jun 2016. PMLR.
- Deterministic Variational Inference for Robust Bayesian Neural Networks, March 2019. arXiv:1810.03958 [cs, stat].
- Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2931–2940, Seoul, Korea (South), October 2019. IEEE.
- Sampling-free variational inference of bayesian neural networks by variance backpropagation. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 563–573. PMLR, 22–25 Jul 2020.
- Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 1, pages 55–60 vol.1, 1994.
- Variational Autoencoder based Anomaly Detection using Reconstruction Probability, 2015.
- Getting a CLUE: A Method for Explaining Uncertainty Estimates, March 2021. arXiv:2006.06848 [cs, stat].
- A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference, January 2019. arXiv:1901.02731 [cs, stat].
- A Survey of Uncertainty in Deep Neural Networks, January 2022. arXiv:2107.03342 [cs, stat].
- Hands-on Bayesian Neural Networks – a Tutorial for Deep Learning Users. IEEE Computational Intelligence Magazine, 17(2):29–48, May 2022. arXiv:2007.06823 [cs, stat].
- Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
- Auto-Encoding Variational Bayes, 2013. arXiv:1312.6114 [cs, stat].
- Weight Uncertainty in Neural Networks, May 2015. arXiv:1505.05424 [cs, stat].
- Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
- Decoupled Weight Decay Regularization, January 2019. arXiv:1711.05101 [cs, math].
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.