2000 character limit reached
Generative Posterior Networks for Approximately Bayesian Epistemic Uncertainty Estimation (2312.17411v1)
Published 29 Dec 2023 in cs.LG and stat.ML
Abstract: In many real-world problems, there is a limited set of training data, but an abundance of unlabeled data. We propose a new method, Generative Posterior Networks (GPNs), that uses unlabeled data to estimate epistemic uncertainty in high-dimensional problems. A GPN is a generative model that, given a prior distribution over functions, approximates the posterior distribution directly by regularizing the network towards samples from the prior. We prove theoretically that our method indeed approximates the Bayesian posterior and show empirically that it improves epistemic uncertainty estimation and scalability over competing methods.
- Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859–877, 2017.
- Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613–1622. PMLR, 2015.
- A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020.
- Efficient model-based reinforcement learning through optimistic policy search and planning. In Neural Information Processing Systems (NeurIPS), 2020. URL https://arxiv.org/abs/2006.08684.
- Aleatory or epistemic? does it matter? Structural safety, 31(2):105–112, 2009.
- Gal, Y. Uncertainty in deep learning. PhD thesis, University of Cambridge, 2016.
- Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050–1059. PMLR, 2016.
- Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
- On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321–1330. PMLR, 2017.
- Hamidieh, K. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science, 154:346–354, 2018.
- Bayesian deep ensembles via the neural tangent kernel. arXiv preprint arXiv:2007.05864, 2020.
- Gaussian processes for big data. arXiv preprint arXiv:1309.6835, 2013.
- Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498–7512, 2020.
- Uncertainty in neural networks: Approximately bayesian ensembling. In International conference on artificial intelligence and statistics, pp. 234–244. PMLR, 2020.
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
- Bayesian optimization with robust bayesian neural networks. Advances in neural information processing systems, 29:4134–4142, 2016.
- Confidence-calibrated adversarial training: Generalizing to unseen attacks. In International Conference on Machine Learning, pp. 9155–9166. PMLR, 2020.
- The calculation of posterior distributions by data augmentation. Journal of the American statistical Association, 82(398):528–540, 1987.
- Titsias, M. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pp. 567–574. PMLR, 2009.
- Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning, pp. 9690–9700. PMLR, 2020.
- Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International Conference on Machine Learning, pp. 1775–1784. PMLR, 2015.
- Deep kernel learning. In Artificial intelligence and statistics, pp. 370–378. PMLR, 2016.