Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

VISA: Variational Inference with Sequential Sample-Average Approximations (2403.09429v2)

Published 14 Mar 2024 in stat.ML and cs.LG

Abstract: We present variational inference with sequential sample-average approximation (VISA), a method for approximate inference in computationally intensive models, such as those based on numerical simulations. VISA extends importance-weighted forward-KL variational inference by employing a sequence of sample-average approximations, which are considered valid inside a trust region. This makes it possible to reuse model evaluations across multiple gradient steps, thereby reducing computational cost. We perform experiments on high-dimensional Gaussians, Lotka-Volterra dynamics, and a Pickover attractor, which demonstrate that VISA can achieve comparable approximation accuracy to standard importance-weighted forward-KL variational inference with computational savings of a factor two or more for conservatively chosen learning rates.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Particle Markov Chain Monte Carlo Methods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 72(3):269–342, June 2010.
  2. Generalized Doubly Reparameterized Gradient Estimators. In International Conference on Machine Learning, pp. 738–747. PMLR, 2021.
  3. Reweighted wake-sleep. arXiv preprint arXiv:1406.2751, 2014.
  4. Sample Average Approximation for Black-Box VI. arXiv preprint arXiv:2304.06803, 2023.
  5. A Stochastic Quasi-Newton Method for Large-Scale Optimization. SIAM Journal on Optimization, 26(2):1008–1031, January 2016. ISSN 1052-6234. doi: 10.1137/140954362.
  6. Carpenter, B. Predator-Prey Population Dynamics: The Lotka-Volterra model in Stan. https://mc-stan.org/users/documentation/case-studies/lotka-volterra-predator-prey.html#abstract, January 2018.
  7. On importance-weighted autoencoders. arXiv preprint arXiv:1907.10477, 2019.
  8. Black box variational inference with a deterministic objective: Faster, more accurate, and even more black box. Journal of Machine Learning Research, 25(18):1–39, 2024.
  9. Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  10. Glynn, P. W. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75–84, October 1990. ISSN 0001-0782. doi: 10.1145/84537.84552.
  11. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proceedings F (Radar and Signal Processing), 140(2):107–113, April 1993. ISSN 2053-9045. doi: 10.1049/ip-f-2.1993.0015.
  12. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(47):1593–1623, 2014. ISSN 1533-7928.
  13. A Guide to Sample Average Approximation. Handbook of simulation optimization, pp.  207–243, 2015.
  14. Auto-Encoding Variational Bayes. ICLR 2014 conference submission, December 2013.
  15. Revisiting Reweighted Wake-Sleep. International Conference on Learning Representations, September 2018.
  16. Quasi-Monte Carlo Quasi-Newton in Variational Bayes. Journal of Machine Learning Research, 22(243):1–23, 2021. ISSN 1533-7928.
  17. Lotka, A. J. Principles of physical biology. 1925.
  18. Markovian Score Climbing: Variational Inference with KL(p || q). In Advances in Neural Information Processing Systems, volume 33, pp.  15499–15510. Curran Associates, Inc., 2020.
  19. Elements of Sequential Monte Carlo. Foundations and Trends in Machine Learning, 2019.
  20. Variational bayesian inference with stochastic search. In Proceedings of the 37th International Conference on Machine Learning, 2012.
  21. Pickover, C. A. The Pattern Book: Fractals, Art And Nature. World Scientific, June 1995. ISBN 978-981-4504-03-4.
  22. Bayesian optimization for probabilistic programs. Advances in Neural Information Processing Systems, 29, 2016.
  23. Black Box Variational Inference. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, pp.  814–822. PMLR, April 2014.
  24. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning, pp.  1278–1286. PMLR, June 2014.
  25. Learning proposals for probabilistic programs with inference combinators. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, pp.  1056–1066. PMLR, December 2021.
  26. Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives. arXiv preprint arXiv:1810.04152, 2018.
  27. An Introduction to Probabilistic Programming. arXiv preprint arXiv:1809.10756, 2018.
  28. Volterra, V. Fluctuations in the Abundance of a Species considered Mathematically. Nature, 119(2983):12–13, January 1927. ISSN 1476-4687. doi: 10.1038/119012b0.
  29. Automated variational inference in probabilistic programming. arXiv preprint arXiv:1301.1299, 2013.
  30. Amortized Population Gibbs Samplers with Neural Sufficient Statistics. In Proceedings of the 37th International Conference on Machine Learning, pp.  10421–10431. PMLR, November 2020.
  31. Pathfinder: Parallel quasi-Newton variational inference. Journal of Machine Learning Research, 23(306):1–49, 2022. ISSN 1533-7928.
  32. Transport Score Climbing: Variational Inference Using Forward KL and Adaptive Neural Transport. Transactions on Machine Learning Research, May 2023. ISSN 2835-8856.
  33. Nested Variational Inference. In Advances in Neural Information Processing Systems, volume 34, pp.  20423–20435. Curran Associates, Inc., 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets