Papers
Topics
Authors
Recent
Search
2000 character limit reached

Deep Q-Exponential Processes

Published 29 Oct 2024 in stat.ML, cs.LG, and stat.ME | (2410.22119v1)

Abstract: Motivated by deep neural networks, the deep Gaussian process (DGP) generalizes the standard GP by stacking multiple layers of GPs. Despite the enhanced expressiveness, GP, as an $L_2$ regularization prior, tends to be over-smooth and sub-optimal for inhomogeneous subjects, such as images with edges. Recently, Q-exponential process (Q-EP) has been proposed as an $L_q$ relaxation to GP and demonstrated with more desirable regularization properties through a parameter $q>0$ with $q=2$ corresponding to GP. Sharing the similar tractability of posterior and predictive distributions with GP, Q-EP can also be stacked to improve its modeling flexibility. In this paper, we generalize Q-EP to deep Q-EP to enjoy both proper regularization and improved expressiveness. The generalization is realized by introducing shallow Q-EP as a latent variable model and then building a hierarchy of the shallow Q-EP layers. Sparse approximation by inducing points and scalable variational strategy are applied to facilitate the inference. We demonstrate the numerical advantages of the proposed deep Q-EP model by comparing with multiple state-of-the-art deep probabilistic models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Deep gaussian process priors for bayesian inference in nonlinear inverse problems. 12 2023.
  2. Deep kernel processes. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 130–140. PMLR, 18–24 Jul 2021.
  3. Deep convolutional gaussian processes. In Machine Learning and Knowledge Discovery in Databases, pages 582â–597. Springer International Publishing, 2020.
  4. Deep horseshoe gaussian processes. 03 2024.
  5. Deep Gaussian processes. In Carlos M. Carvalho and Pradeep Ravikumar, editors, Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31 of Proceedings of Machine Learning Research, pages 207–215, Scottsdale, Arizona, USA, 29 Apr–01 May 2013. PMLR.
  6. Besov priors for bayesian inverse problems. Inverse Problems and Imaging, 6(2):183–200, may 2012.
  7. Bayesian image classification with deep convolutional gaussian processes. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1529–1539. PMLR, 26–28 Aug 2020.
  8. K. Fang and Y.T. Zhang. Generalized Multivariate Analysis. Science Press, 1990.
  9. Posterior contraction for deep gaussian process priors. Journal of Machine Learning Research, 24(66):1–49, 2023.
  10. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  11. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
  12. Scalable Variational Gaussian Process Classification. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 351–360, San Diego, California, USA, 09–12 May 2015. PMLR.
  13. A. P. Dawid J. M. Bernardo, J. O. Berger and A. F. M. Smith. Regression and classification using gaussian process priors. Bayesian Statistics, 6:475–501, 1998.
  14. Deep sigma point processes. In Jonas Peters and David Sontag, editors, Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pages 789–798. PMLR, 03–06 Aug 2020.
  15. Parametric Gaussian process regressors. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4702–4712. PMLR, 13–18 Jul 2020.
  16. Inverse reinforcement learning via deep gaussian process. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence (UAI), volume abs/1512.08065, 2017.
  17. Mark E. Johnson. Multivariate Statistical Simulation, chapter 6 Elliptically Contoured Distributions, pages 106–124. Probability and Statistics. John Wiley & Sons, Ltd, 1987.
  18. Alignment of spatial genomics data using deep gaussian processes. Nature Methods, 20(9):1379–1387, August 2023.
  19. Multivariate generalized laplace distribution and related random fields. Journal of Multivariate Analysis, 113:59–72, 2013. Special Issue on Multivariate Distribution Theory in Memory of Samuel Kotz.
  20. Discretization-invariant bayesian inversion and besov space priors. Inverse Problems and Imaging, 3(1):87–122, 2009.
  21. Neil Lawrence. Gaussian process latent variable models for visualisation of high dimensional data. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2003.
  22. Neil Lawrence. Probabilistic non-linear principal component analysis with gaussian process latent variable models. Journal of Machine Learning Research, 6(60):1783–1816, 2005.
  23. Hierarchical gaussian process latent variable models. In Proceedings of the 24th international conference on Machine learning, ICML 2007. ACM, June 2007.
  24. Bayesian learning via q-exponential process. In Proceedings of the 37th Conference on Neural Information Processing Systems. NeurIPS, 12 2023. arxiv:2210.07987.
  25. Deep bayesian gaussian processes for uncertainty estimation in electronic health records. Scientific Reports, 11(1), October 2021.
  26. Thomas Minka. Automatic choice of dimensionality for pca. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
  27. Radford M. Neal. Bayesian Learning for Neural Networks. Springer New York, 1996.
  28. Bernt Øksendal. Stochastic Differential Equations. Springer Berlin Heidelberg, 2003.
  29. Deep variational implicit processes. In The Eleventh International Conference on Learning Representations, 2023.
  30. Estimation for stochastic models driven by laplace motion. Communications in Statistics - Theory and Methods, 40(18):3281–3302, sep 2011.
  31. Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2005.
  32. Doubly stochastic variational inference for deep gaussian processes. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  33. Probabilistic Principal Component Analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 61(3):611–622, 09 1999.
  34. Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In David van Dyk and Max Welling, editors, Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 567–574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009. PMLR.
  35. Bayesian gaussian process latent variable model. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 844–851, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
  36. Deep image prior. International Journal of Computer Vision, 128(7):1867–1888, Jul 2020.
  37. Deep kernel learning. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 370–378, Cadiz, Spain, 09–11 May 2016. PMLR.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 1 like about this paper.

Sign Up for Free