Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Gaussian Mixture Ensembles (2306.07235v1)

Published 12 Jun 2023 in stat.ML, cs.LG, and stat.ME

Abstract: This work introduces a novel probabilistic deep learning technique called deep Gaussian mixture ensembles (DGMEs), which enables accurate quantification of both epistemic and aleatoric uncertainty. By assuming the data generating process follows that of a Gaussian mixture, DGMEs are capable of approximating complex probability distributions, such as heavy-tailed or multimodal distributions. Our contributions include the derivation of an expectation-maximization (EM) algorithm used for learning the model parameters, which results in an upper-bound on the log-likelihood of training data over that of standard deep ensembles. Additionally, the proposed EM training procedure allows for learning of mixture weights, which is not commonly done in ensembles. Our experimental results demonstrate that DGMEs outperform state-of-the-art uncertainty quantifying deep learning models in handling complex predictive densities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297, 2021.
  2. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322–332. PMLR, 2019.
  3. Athanassia Bacharoglou. Approximation of probability distributions by convex mixtures of gaussian measures. Proceedings of the American Mathematical Society, 138(7):2619–2628, 2010.
  4. Recurrent mixture density network for spatiotemporal visual attention. arXiv preprint arXiv:1603.08199, 2016.
  5. The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence, 1(1):20–23, 2019.
  6. Christopher M Bishop. Mixture density networks. 1994.
  7. Bayesian neural networks for stock price forecasting before and during covid-19 pandemic. Plos one, 16(7):e0253217, 2021.
  8. Disentangling epistemic and aleatoric uncertainty in reinforcement learning. arXiv preprint arXiv:2206.01558, 2022.
  9. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pages 1683–1691. PMLR, 2014.
  10. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.09638, 2019.
  11. The use of mixture density networks in the emulation of complex epidemiological individual-based models. PLoS computational biology, 16(3):e1006869, 2020.
  12. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089–20103, 2021.
  13. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
  14. Photometric redshift estimation via deep learning-generalized and pre-classification-less, image based, fully probabilistic redshifts. Astronomy & Astrophysics, 609:A111, 2018.
  15. Deep neural networks for estimation and inference. Econometrica, 89(1):181–213, 2021.
  16. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  17. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342, 2021.
  18. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  19. Deep learning. MIT press, 2016.
  20. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  21. Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
  22. Neural expectation maximization. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  23. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International conference on machine learning, pages 1861–1869. PMLR, 2015.
  24. Fast predictive uncertainty for classification with bayesian deep networks. In Uncertainty in Artificial Intelligence, pages 822–832. PMLR, 2022.
  25. Long short-term memory. Neural computation, 9:1735–80, 12 1997.
  26. Stochastic variational inference. Journal of Machine Learning Research, 2013.
  27. Deep ensembles from a bayesian perspective. arXiv preprint arXiv:2105.13283, 2021.
  28. Aleatoric and epistemic uncertainty in machine learning: A tutorial introduction. 2019.
  29. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
  30. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
  31. Evaluating uncertainty quantification in end-to-end autonomous driving control. arXiv preprint arXiv:1811.06817, 2018.
  32. Thomas Minka. Expectation-maximization as lower bound maximization. Tutorial published on the web at http://www-white. media. mit. edu/tpminka/papers/em. html, 7(2), 1998.
  33. Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
  34. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
  35. Uncertainty in neural networks: Approximately bayesian ensembling. In International conference on artificial intelligence and statistics, pages 234–244. PMLR, 2020.
  36. Why have a unified predictive uncertainty? disentangling it using deep split ensembles. arXiv preprint arXiv:2009.12406, 2020.
  37. Bayesian uncertainty estimation for batch normalized deep networks. In International Conference on Machine Learning, pages 4907–4916. PMLR, 2018.
  38. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and buildings, 49:560–567, 2012.
  39. Pınar Tüfekci. Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods. International Journal of Electrical Power & Energy Systems, 60:126–140, 2014.
  40. Deep convolutional mixture density network for inverse design of layered photonic structures. ACS Photonics, 7(10):2703–2712, 2020.
  41. A deeper look into aleatoric and epistemic uncertainty disentanglement. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1508–1516. IEEE, 2022.
  42. Bayesian deep learning and a probabilistic perspective of generalization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4697–4708. Curran Associates, Inc., 2020a.
  43. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697–4708, 2020b.
  44. CF Jeff Wu. On the convergence properties of the em algorithm. The Annals of statistics, pages 95–103, 1983.
  45. I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete research, 28(12):1797–1808, 1998.
  46. Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 3844–3848. IEEE, 2014.
  47. Improved deep mixture density network for regional wind power probabilistic forecasting. IEEE Transactions on Power Systems, 35(4):2549–2560, 2020.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now