Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 54 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 18 tok/s Pro
GPT-5 High 31 tok/s Pro
GPT-4o 105 tok/s Pro
Kimi K2 182 tok/s Pro
GPT OSS 120B 466 tok/s Pro
Claude Sonnet 4 40 tok/s Pro
2000 character limit reached

Posterior Uncertainty Quantification in Neural Networks using Data Augmentation (2403.12729v1)

Published 18 Mar 2024 in stat.ML and cs.LG

Abstract: In this paper, we approach the problem of uncertainty quantification in deep learning through a predictive framework, which captures uncertainty in model parameters by specifying our assumptions about the predictive distribution of unseen future data. Under this view, we show that deep ensembling (Lakshminarayanan et al., 2017) is a fundamentally mis-specified model class, since it assumes that future data are supported on existing observations only -- a situation rarely encountered in practice. To address this limitation, we propose MixupMP, a method that constructs a more realistic predictive distribution using popular data augmentation techniques. MixupMP operates as a drop-in replacement for deep ensembles, where each ensemble member is trained on a random simulation from this predictive distribution. Grounded in the recently-proposed framework of Martingale posteriors (Fong et al., 2023), MixupMP returns samples from an implicitly defined Bayesian posterior. Our empirical analysis showcases that MixupMP achieves superior predictive performance and uncertainty quantification on various image classification datasets, when compared with existing Bayesian and non-Bayesian approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Deep ensembles work, but are they necessary? In Advances in Neural Information Processing Systems.
  2. Limit theorems for a class of identically distributed random variables. The Annals of Probability, 32(3):2029–2052.
  3. Weight uncertainty in neural network. In International Conference on Machine Learning.
  4. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning.
  5. Stochastic gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning.
  6. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory.
  7. Laplace redux-effortless Bayesian deep learning. In Advances in Neural Information Processing Systems.
  8. De Finetti, B. (1937). La prévision: ses lois logiques, ses sources subjectives. Annales de l’institut Henri Poincaré, 7(1):1–68.
  9. Doob, J. L. (1949). Application of the theory of martingales. Le calcul des probabilites et ses applications, pages 23–27.
  10. Efron, B. (1992). Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569–593. Springer.
  11. Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection. In International Conference on Intelligent Transportation Systems.
  12. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP.
  13. A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks. In NeurIPS Workshop on Bayesian Deep Learning.
  14. Martingale posterior distributions. Journal of the Royal Statistical Society, Series A, 85(5):1357–1391.
  15. Scalable nonparametric sampling from multimodal posteriors with the posterior bootstrap. In International Conference on Machine Learning.
  16. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning.
  17. Neural processes. arXiv preprint arXiv:1807.01622.
  18. Graves, A. (2011). Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, volume 24.
  19. Noise contrastive priors for functional uncertainty. In Uncertainty in Artificial Intelligence.
  20. Analysis of dropout learning regarded as ensemble learning. In International Conference on Artificial Neural Networks.
  21. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.
  22. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv:1807.01697.
  23. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning.
  24. What are Bayesian neural network posteriors really like? In International Conference on Machine Learning.
  25. Directional convergence and alignment in deep learning. In Advances in Neural Information Processing Systems.
  26. A simple resampling method by perturbing the minimand. Biometrika, 88(2):381–390.
  27. Learning multiple layers of features from tiny images. https://www.cs.utoronto.ca/ kriz/learning-features-2009-TR.pdf.
  28. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems.
  29. LeCun, Y. (1998). The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/.
  30. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.
  31. Martingale posterior neural processes. In International Conference on Learning Representations.
  32. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations.
  33. Nonparametric learning from Bayesian models with randomized objective functions. In Advances in Neural Information Processing Systems.
  34. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations.
  35. MacKay, D. J. (1992). A practical Bayesian framework for backpropagation networks. Neural computation, 4(3):448–472.
  36. Mixspeech: Data augmentation for low-resource automatic speech recognition. In International Conference on Acoustics, Speech and Signal Processing.
  37. Convergence of gradient descent on separable data. In International Conference on Artificial Intelligence and Statistics.
  38. Obtaining well calibrated probabilities using Bayesian binning. In AAAI Conference on Artificial Intelligence, volume 29.
  39. Neal, R. M. (2012). Bayesian learning for neural networks. Springer Science & Business Media.
  40. Approximate bayesian inference by the weighted likelihood bootstrap (with discussion). Journal of the Royal Statistical Society Series B, 56:1–48.
  41. Weighted Bayesian bootstrap for scalable posterior distributions. Canadian Journal of Statistics, 49(2):421–437.
  42. Why are bootstrapped deep ensembles not better? In ”I Can’t Believe It’s Not Better!”NeurIPS 2020 workshop.
  43. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems.
  44. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems.
  45. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems.
  46. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems.
  47. Uncertainty quantification and deep ensembles. In Advances in Neural Information Processing Systems.
  48. Rubin, D. B. (1981). The Bayesian bootstrap. The Annals of Statistics, pages 130–134.
  49. Tractable function-space variational inference in bayesian neural networks. In Advances in Neural Information Processing Systems.
  50. Neural bootstrapper. In Advances in Neural Information Processing Systems.
  51. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1–48.
  52. Understanding measures of uncertainty for adversarial example detection. In Uncertainty in Artificial Intelligence.
  53. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958.
  54. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems.
  55. Is importance weighting incompatible with interpolating classifiers? In International Conference on Learning Representations.
  56. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. In Advances in Neural Information Processing Systems.
  57. Combining ensembles and data augmentation can harm your calibration. In International Conference on Learning Representations.
  58. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems.
  59. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747.
  60. Understanding the role of importance weighting for deep learning. In International Conference on Learning Representations.
  61. Wide residual networks. arXiv preprint arXiv:1605.07146.
  62. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
  63. Cyclical stochastic gradient MCMC for Bayesian deep learning. In International Conference on Learning Representations.
Citations (1)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets