Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tractable Function-Space Variational Inference in Bayesian Neural Networks (2312.17199v1)

Published 28 Dec 2023 in stat.ML, cs.AI, and cs.LG

Abstract: Reliable predictive uncertainty estimation plays an important role in enabling the deployment of neural networks to safety-critical settings. A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters, infer an approximate posterior distribution, and use it to make stochastic predictions. However, explicit inference over neural network parameters makes it difficult to incorporate meaningful prior information about the data-generating process into the model. In this paper, we pursue an alternative approach. Recognizing that the primary object of interest in most settings is the distribution over functions induced by the posterior distribution over neural network parameters, we frame Bayesian inference in neural networks explicitly as inferring a posterior distribution over functions and propose a scalable function-space variational inference method that allows incorporating prior information and results in reliable predictive uncertainty estimates. We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks and demonstrate that it performs well on a challenging safety-critical medical diagnosis task in which reliable uncertainty estimation is essential.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Concrete problems in ai safety, 2016.
  2. APTOS. APTOS 2019 Blindness Detection Dataset, 2019.
  3. Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks. 2021.
  4. Measuring and regularizing networks in function space. In International Conference on Learning Representations, 2019.
  5. Weight uncertainty in neural networks. volume 37 of Proceedings of Machine Learning Research, pages 1613–1622, Lille, France, 07–09 Jul 2015. PMLR.
  6. Understanding variational inference in function-space. In Third Symposium on Advances in Approximate Bayesian Inference, 2021.
  7. Scalable uncertainty for computer vision with functional variational inference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  8. EyePACS. Diabetic Retinopathy Detection Dataset, 2015.
  9. Radial Bayesian neural networks: Beyond discrete support in large-scale Bayesian deep learning. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1352–1362. PMLR, 26–28 Aug 2020a.
  10. Liberty or depth: Deep Bayesian neural nets do not need complex weight posterior approximations. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020b.
  11. A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks, 2019.
  12. ’in-between’ uncertainty in Bayesian neural networks, 2019.
  13. On the expressiveness of approximate inference in Bayesian neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  14. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML 2016, pages 1050–1059, 2016.
  15. Alex Graves. Practical variational inference for neural networks. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS’11, page 2348–2356, Red Hook, NY, USA, 2011. Curran Associates Inc. ISBN 9781618395993.
  16. Unsolved problems in ml safety, 2021.
  17. Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT ’93, page 5–13, New York, NY, USA, 1993. Association for Computing Machinery. ISBN 0897916115.
  18. Stochastic variational inference. Journal of Machine Learning Research, 14(1):1303–1347, May 2013. ISSN 1532-4435.
  19. Improving predictions of Bayesian neural networks via local linearization, 2020.
  20. Subspace inference for Bayesian deep learning. In Ryan P. Adams and Vibhav Gogate, editors, Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 1169–1179. PMLR, 22–25 Jul 2020.
  21. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583–589, 2021. doi: 10.1038/s41586-021-03819-2.
  22. Approximate inference turns deep networks into Gaussian processes. In Advances in Neural Information Processing Systems 32, pages 3094–3104. Curran Associates, Inc., 2019.
  23. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25:, pages 1106–1114, 2012.
  24. Leveraging Uncertainty Information From Deep Neural Networks for Disease Detection. Nature Scientific Reports, 7(1):17816, 2017.
  25. Functional variational inference based on stochastic process generators. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  26. Variational implicit processes. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4222–4233. PMLR, 09–15 Jun 2019.
  27. David J. C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Comput., 4(3):448–472, May 1992. ISSN 0899-7667. doi: 10.1162/neco.1992.4.3.448.
  28. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, pages 13153–13164, 2019.
  29. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. volume 51 of Proceedings of Machine Learning Research, pages 231–239, Cadiz, Spain, 09–11 May 2016. PMLR.
  30. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop. 2013.
  31. Radford M Neal. Bayesian Learning for Neural Networks. 1996.
  32. Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes, 2020.
  33. Practical deep learning with Bayesian principles. In Advances in Neural Information Processing Systems, volume 32, pages 4287–4299. Curran Associates, Inc., 2019.
  34. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32. 2019.
  35. Continual deep learning by functional regularisation of memorable past. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  36. Strong data-processing inequalities for channels and Bayesian networks. In Eric Carlen, Mokshay Madiman, and Elisabeth M. Werner, editors, Convexity and Concentration, pages 211–249, New York, NY, 2017. Springer New York. ISBN 978-1-4939-7005-6.
  37. Tim G. J. Rudner and Helen Toner. Key Concepts in AI Safety: An Overview. In CSET Issue Briefs, 2021a.
  38. Tim G. J. Rudner and Helen Toner. Key Concepts in AI Safety: Robustness and Adversarial Examples. In CSET Issue Briefs, 2021b.
  39. On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations. In Advances in Neural Information Processing Systems 34, 2021.
  40. Continual Learning via Sequential Function-Space Variational Inference. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2022.
  41. M. J. Schervish. Theory of Statistics. Springer-Verlag, New York, NY, 1995.
  42. Mastering the game of Go with deep neural networks and tree search. 529, 2016.
  43. Sparse Gaussian processes using pseudo-inputs. In Y. Weiss, B. Schölkopf, and J. C. Platt, editors, Advances in Neural Information Processing Systems 18, pages 1257–1264. MIT Press, 2006.
  44. Functional variational Bayesian neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  45. Functional regularisation for continual learning with Gaussian processes. In International Conference on Learning Representations, 2020.
  46. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning, 2020.
  47. Variational deterministic uncertainty quantification, 2021.
  48. Graphical Models, Exponential Families, and Variational Inference. Now Publishers Inc., Hanover, MA, USA, 2008. ISBN 1601981848.
  49. Function space particle optimization for Bayesian neural networks. In International Conference on Learning Representations, 2019.
  50. D. T. S. Widdowson. The Management of Grading Quality: Good Practice in the Quality Assurance of Grading. Tech. Rep., 2016.
  51. David H. Wolpert. Bayesian backpropagation over i-o functions rather than weights. In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, volume 6. Morgan-Kaufmann, 1993.
Citations (34)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now