Structured Partial Stochasticity in Bayesian Neural Networks (2405.17666v2)
Abstract: Bayesian neural network posterior distributions have a great number of modes that correspond to the same network function. The abundance of such modes can make it difficult for approximate inference methods to do their job. Recent work has demonstrated the benefits of partial stochasticity for approximate inference in Bayesian neural networks; inference can be less costly and performance can sometimes be improved. I propose a structured way to select the deterministic subset of weights that removes neuron permutation symmetries, and therefore the corresponding redundant posterior modes. With a drastically simplified posterior distribution, the performance of existing approximate inference schemes is found to be greatly improved.
- Git re-basin: Merging models modulo permutation symmetries, 2023.
- Weight uncertainty in neural networks, 2015.
- On the geometry of feedforward neural network error surfaces. Neural Computation, 5(6):910–927, 1993. 10.1162/neco.1993.5.6.910.
- Wide mean-field bayesian neural networks ignore the data, 2022.
- Bayesian deep learning via subnetwork inference. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2510–2521. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/daxberger21a.html.
- Efficient and scalable bayesian neural nets with rank-1 factors, 2020.
- The role of permutation invariance in linear mode connectivity of neural networks, 2022.
- Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
- Being bayesian, even just a bit, fixes overconfidence in relu networks, 2020.
- On symmetries in variational bayesian neural nets. In NeurIPS 2021 Workshop on Bayesian Deep Learning, 2021. URL https://www.amazon.science/publications/on-symmetries-in-variational-bayesian-neural-nets.
- On the detrimental effect of invariances in the likelihood for variational inference, 2022.
- Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.
- Radford Neal. Bayesian learning via stochastic dynamics. In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufmann, 1992. URL https://proceedings.neurips.cc/paper_files/paper/1992/file/f29c21d4897f78948b91f03172341b7b-Paper.pdf.
- Benchmarking the neural linear model for regression, 2019.
- Position paper: Bayesian deep learning in the age of large-scale ai, 2024.
- Improving the identifiability of neural networks for bayesian inference. 2017. URL https://api.semanticscholar.org/CorpusID:46932278.
- On permutation symmetries in bayesian neural network posteriors: a variational perspective, 2023.
- Do bayesian neural networks need to be fully stochastic? In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7694–7722. PMLR, 25–27 Apr 2023. URL https://proceedings.mlr.press/v206/sharma23a.html.
- Connecting the dots: Is mode-connectedness the key to feasible sample-based inference in bayesian neural networks?, 2024.
- Energy Efficiency. UCI Machine Learning Repository, 2012. DOI: https://doi.org/10.24432/C51307.
- Towards efficient mcmc sampling in bayesian neural networks by exploiting symmetry, 2023.
- A compact representation for bayesian neural networks by removing permutation symmetry, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Collections
Sign up for free to add this paper to one or more collections.
