Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Compositional Score Modeling for Simulation-based Inference (2209.14249v3)

Published 28 Sep 2022 in cs.LG and stat.ML

Abstract: Neural Posterior Estimation methods for simulation-based inference can be ill-suited for dealing with posterior distributions obtained by conditioning on multiple observations, as they tend to require a large number of simulator calls to learn accurate approximations. In contrast, Neural Likelihood Estimation methods can handle multiple observations at inference time after learning from individual observations, but they rely on standard inference methods, such as MCMC or variational inference, which come with certain performance drawbacks. We introduce a new method based on conditional score modeling that enjoys the benefits of both approaches. We model the scores of the (diffused) posterior distributions induced by individual observations, and introduce a way of combining the learned scores to approximately sample from the target posterior distribution. Our approach is sample-efficient, can naturally aggregate multiple observations at inference time, and avoids the drawbacks of standard inference methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  2. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13606, 2021.
  3. Beaumont, M. A. Approximate Bayesian computation. Annual review of statistics and its application, 6:379–403, 2019.
  4. Non-linear regression models for approximate Bayesian computation. Statistics and Computing, 20(1):63–73, 2010.
  5. A likelihood-free inference framework for population genetic data using exchangeable neural networks. Advances in Neural Information Processing Systems, 31, 2018.
  6. Density ratio estimation via infinitesimal classification. In International Conference on Artificial Intelligence and Statistics, pp.  2552–2573. PMLR, 2022.
  7. Approximating likelihood ratios with calibrated discriminative classifiers. arXiv preprint arXiv:1506.02169, 2015.
  8. Active sciencing with reusable workflows. https://https://github.com/cranmer/active_sciencing, 2017.
  9. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117(48):30055–30062, 2020.
  10. Diffusion Schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34:17695–17709, 2021.
  11. An adaptive sequential Monte Carlo method for approximate Bayesian computation. Statistics and computing, 22(5):1009–1020, 2012.
  12. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  13. Density estimation using Real NVP. arXiv preprint arXiv:1605.08803, 2016.
  14. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552, 2023.
  15. Sequential neural methods for likelihood-free inference. Bayesian Deep Learning Workshop at Neural Information Processing Systems, 2018.
  16. Friedman, J. H. On multivariate goodness–of–fit and two–sample testing. Statistical Problems in Particle Physics, Astrophysics, and Cosmology, 1:311, 2003.
  17. Variational methods for simulation-based inference. arXiv preprint arXiv:2203.04176, 2022.
  18. Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning, pp. 2404–2414. PMLR, 2019.
  19. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723–773, 2012.
  20. Exact analytical solutions of the Susceptible-Infected-Recovered (SIR) epidemic model and of the SIR model with equal death and birth rates. Applied Mathematics and Computation, 236:184–194, 2014.
  21. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  22. Likelihood-free MCMC with amortized approximate ratio estimators. In International Conference on Machine Learning, pp. 4239–4248. PMLR, 2020a.
  23. Likelihood-free MCMC with amortized approximate ratio estimators. In Proceedings of the 37th International Conference on Machine Learning, 2020b.
  24. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  25. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  26. The No-U-Turn Sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1):1593–1623, 2014.
  27. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
  28. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  29. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47–59, 2022.
  30. Compositional visual generation with composable diffusion models. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII, pp.  423–439. Springer, 2022.
  31. Adversarial variational optimization of non-differentiable simulators. arXiv preprint arXiv:1707.07113, 2017.
  32. Flexible statistical inference for mechanistic models of neural dynamics. Advances in Neural Information Processing Systems, 30, 2017.
  33. Likelihood-free inference with emulator networks. In Symposium on Advances in Approximate Bayesian Inference, pp.  32–53. PMLR, 2019.
  34. Benchmarking simulation-based inference. In International Conference on Artificial Intelligence and Statistics, pp.  343–351. PMLR, 2021.
  35. Luo, C. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.
  36. Markov chain Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences, 100(26):15324–15328, 2003.
  37. Neal, R. M. MCMC using Hamiltonian dynamics. Handbook of Markov chain Monte Carlo, 2(11):2, 2011.
  38. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning, 2022.
  39. Fast ε𝜀\varepsilonitalic_ε-free inference of simulation models with Bayesian conditional density estimation. Advances in Neural Information Processing Systems, 29, 2016.
  40. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.  837–848. PMLR, 2019.
  41. A note on approximating ABC-MCMC using flexible classifiers. Stat, 3(1):218–227, 2014.
  42. Composable effects for flexible and accelerated probabilistic programming in NumPyro. arXiv preprint arXiv:1912.11554, 2019.
  43. BayesFlow: Learning complex stochastic models with invertible neural networks. IEEE transactions on neural networks and learning systems, 2020.
  44. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  45. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022.
  46. Variational inference with normalizing flows. In International Conference on Machine Learning, pp. 1530–1538. PMLR, 2015.
  47. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905–4916, 2020.
  48. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341–363, 1996.
  49. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022a.
  50. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
  51. Should EBMs model the energy or the score? 2021.
  52. Sequential neural score estimation: Likelihood-free inference with conditional score based diffusion models. arXiv preprint arXiv:2210.04872, 2022.
  53. Conditional simulation using diffusion Schrödinger bridges. arXiv preprint arXiv:2202.13460, 2022.
  54. Sequential Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences, 104(6):1760–1765, 2007.
  55. Handbook of approximate Bayesian computation. CRC Press, 2018.
  56. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
  57. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  58. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  59. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415–1428, 2021.
  60. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145–164, 2013.
  61. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804–24816, 2021.
  62. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
  63. Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
  64. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, pp.  681–688. Citeseer, 2011.
  65. Learning likelihoods with conditional normalizing flows. arXiv preprint arXiv:1912.00042, 2019.
  66. Sequential neural posterior and likelihood approximation. arXiv preprint arXiv:2102.06522, 2021.
  67. Wood, S. N. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310):1102–1104, 2010.
  68. Deep sets. Advances in neural information processing systems, 30, 2017.
Citations (16)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now