Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
60 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Simulation-based Inference for High-dimensional Data using Surjective Sequential Neural Likelihood Estimation (2308.01054v3)

Published 2 Aug 2023 in stat.ML, cs.LG, and stat.ME

Abstract: Neural likelihood estimation methods for simulation-based inference can suffer from performance degradation when the modeled data is very high-dimensional or lies along a lower-dimensional manifold, which is due to the inability of the density estimator to accurately estimate a density function. We present Surjective Sequential Neural Likelihood (SSNL) estimation, a novel member in the family of methods for simulation-based inference (SBI). SSNL fits a dimensionality-reducing surjective normalizing flow model and uses it as a surrogate likelihood function, which allows for computational inference via Markov chain Monte Carlo or variational Bayes methods. Among other benefits, SSNL avoids the requirement to manually craft summary statistics for inference of high-dimensional data sets, since the lower-dimensional representation is computed simultaneously with learning the likelihood and without additional computational overhead. We evaluate SSNL on a wide variety of experiments, including two challenging real-world examples from the astrophysics and neuroscience literatures, and show that it either outperforms or is on par with state-of-the-art methods, making it an excellent off-the-shelf estimator for SBI for high-dimensional data sets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. A stochastic version of the Jansen and Rit neural mass model: analysis and numerics. The Journal of Mathematical Neuroscience, 7(1):1–35, 2017.
  2. A simulated annealing approach to approximate Bayes computations. Statistics and Computing, 25:1217–1232, 2015.
  3. Learning summary statistics for Bayesian inference with autoencoders. SciPost Physics Core, 5(3):043, 2022.
  4. Generalized massive optimal data compression. Monthly Notices of the Royal Astronomical Society: Letters, 476(1):L60–L64, 2018.
  5. Intrinsic dimensionality estimation within tight localities. In Proceedings of the 2019 SIAM international conference on data mining, pp.  181–189. SIAM, 2019.
  6. Intrinsic dimensionality estimation within tight localities: A theoretical and experimental analysis. arXiv preprint arXiv:2209.14475, 2022.
  7. The DeepMind JAX Ecosystem, 2020. URL http://github.com/deepmind.
  8. Scikit-dimension: a python package for intrinsic dimension estimation. Entropy, 23(10):1368, 2021.
  9. Adaptive approximate Bayesian computation. Biometrika, 96(4):983–990, 2009.
  10. Efficient identification of informative features in simulation-based inference. In Advances in Neural Information Processing Systems, 2022.
  11. Betancourt, M. A conceptual introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:1701.02434, 2017.
  12. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
  13. Brehmer, J. Simulation-based inference in particle physics. Nature Reviews Physics, 3(5):305–305, 2021.
  14. Constraining effective field theories with machine learning. Phys. Rev. Lett., 121:111801, 2018.
  15. Handbook of Markov Chain Monte Carlo. CRC press, 2011.
  16. Spectral density-based and measure-preserving ABC for partially observed diffusion processes. an illustration on Hamiltonian SDEs. Statistics and Computing, 30:627–648, 2020.
  17. Blackjax: A sampling library for JAX, 2023. URL http://github.com/blackjax-devs/blackjax.
  18. Fluctuations in Babcock-Leighton dynamos. i. period doubling and transition to chaos. The Astrophysical Journal, 619(1):613, 2005.
  19. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, 2018.
  20. Neural approximate sufficient statistics for implicit models. In International Conference on Learning Representations, 2021.
  21. Is learning summary statistics necessary for likelihood-free inference? In International Conference on Machine Learning, 2023.
  22. Approximating likelihood ratios with calibrated discriminative classifiers. arXiv preprint arXiv:1506.02169, 2015.
  23. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117(48):30055–30062, 2020.
  24. Normalizing flows across dimensions. In ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2020.
  25. Sliced iterative normalizing flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
  26. Real-time gravitational wave science with neural posterior estimation. Phys. Rev. Lett., 127:241103, 2021.
  27. Truncated proposals for scalable and hassle-free simulation-based inference. In Advances in Neural Information Processing Systems, 2022.
  28. Lightning-fast gravitational wave parameter inference through neural amortization. In Workshop on Machine Learning and the Physical Sciences, Advances in Neural Information Processing Systems, 2020.
  29. Towards reliable simulation-based inference with balanced neural ratio estimation. In Advances in Neural Information Processing Systems, 2022.
  30. Tensorflow distributions. arXiv preprint arXiv:1711.10604, 2017.
  31. NICE: non-linear independent components estimation. In Workshop Track, International Conference on Learning Representations, 2015.
  32. Density estimation using real NVP. In International Conference on Learning Representations, 2017.
  33. Neural spline flows. In Advances in Neural Information Processing Systems, 2019.
  34. On contrastive learning for likelihood-free inference. In Proceedings of the 37th International Conference on Machine Learning, 2020.
  35. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983–1049, 2016.
  36. Beta regression for modelling rates and proportions. Journal of applied statistics, 31(7):799–815, 2004.
  37. Summary statistics and discrepancy measures for approximate bayesian computation via surrogate posteriors. Statistics and Computing, 32(5):85, 2022.
  38. Visualization in bayesian workflow. Journal of the Royal Statistical Society Series A: Statistics in Society, 182(2):389–402, 2019.
  39. Compositional score modeling for simulation-based inference. In International Conference on Machine Learning, 2023.
  40. Inference from iterative simulation using multiple sequences. Statistical Science, 7(4):457 – 472, 1992.
  41. Made: Masked autoencoder for distribution estimation. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
  42. Maximum likelihood learning of energy-based models for simulation-based inference. arXiv preprint arXiv:2210.14756, 2022.
  43. Variational methods for simulation-based inference. In International Conference on Learning Representations, 2022.
  44. Training deep neural density estimators to identify mechanistic models of neural dynamics. Elife, 9:e56261, 2020.
  45. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Conference on Learning Representations, 2019.
  46. Automatic posterior transformation for likelihood-free inference. In Proceedings of the 36th International Conference on Machine Learning, 2019.
  47. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723–773, 2012.
  48. Pseudo-likelihood inference. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  49. Likelihood-free MCMC with amortized approximate ratio estimators. In Proceedings of the 37th International Conference on Machine Learning, 2020.
  50. Towards constraining warm dark matter with stellar streams through neural simulation-based inference. Monthly Notices of the Royal Astronomical Society, 507(2), 2021.
  51. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, 2019.
  52. The No-U-Turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593–1623, 2014.
  53. Minimizing the expected posterior entropy yields optimal summary statistics. arXiv preprint arXiv:2206.02340, 2022.
  54. Jia, H. Simulation-based inference with quantile regression. arXiv preprint arXiv:2401.02413, 2024.
  55. Kidger, P. On Neural Differential Equations. PhD thesis, University of Oxford, 2021.
  56. Glow: Generative flow with invertible 1x1 convolutions. Advances in Neural Information Processing Systems, 2018.
  57. Funnels: Exact maximum likelihood with dimensionality reduction. In Workshop on Bayesian Deep Learning, Advances in Neural Information Processing Systems, 2021.
  58. Adaptive approximate Bayesian computation for complex models. Computational Statistics, 28(6):2777–2796, 2013.
  59. L-C2ST: Local diagnostics for posterior approximations in simulation-based inference. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  60. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017.
  61. Flexible statistical inference for mechanistic models of neural dynamics. In Advances in Neural Information Processing Systems, 2017.
  62. Benchmarking simulation-based inference. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, 2021.
  63. Contrastive neural ratio estimation. In Advances in Neural Information Processing Systems, 2022.
  64. Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 2020.
  65. Score matched neural exponential families for likelihood-free inference. The Journal of Machine Learning Research, 23(1):1745–1815, 2022.
  66. Fast ε𝜀\varepsilonitalic_ε-free inference of simulation models with Bayesian conditional density estimation. In Advances in Neural Information Processing Systems, volume 29, 2016.
  67. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, 2017.
  68. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 2019.
  69. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617–2680, 2021.
  70. Population growth of human y chromosomes: a study of y chromosome microsatellites. Molecular Biology and Evolution, 16(12):1791–1798, 1999.
  71. JANA: Jointly amortized neural approximation of complex bayesian models. In The 39th Conference on Uncertainty in Artificial Intelligence, 2023.
  72. Using likelihood-free inference to compare evolutionary dynamics of the protein networks of H. pylori and P. falciparum. PLoS Computational Biology, 3(11):e230, 2007.
  73. HNPE: Leveraging global parameters for neural posterior estimation. In Advances in Neural Information Processing Systems, 2021.
  74. Applied stochastic differential equations. Cambridge University Press, 2019.
  75. Consistency models for scalable and fast simulation-based inference. arXiv preprint arXiv:2312.05440, 2023.
  76. Neural score estimation: Likelihood-free inference with conditional score based diffusion models. In Fifth Symposium on Advances in Approximate Bayesian Inference, 2023.
  77. Handbook of Approximate Bayesian Computation. CRC Press, 2018.
  78. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference on Learning Representations, 2017.
  79. Validating Bayesian inference algorithms with simulation-based calibration. arXiv preprint arXiv:1804.06788, 2018.
  80. sbi: A toolkit for simulation-based inference. Journal of Open Source Software, 5(52):2505, 2020.
  81. A note on the evaluation of generative models. In International Conference on Learning Representations, 2016.
  82. Likelihood-free inference by ratio estimation. Bayesian Analysis, 17(1):1–31, 2022.
  83. Rank-Normalization, Folding, and Localization: An Improved R^^𝑅\widehat{R}over^ start_ARG italic_R end_ARG for Assessing Convergence of MCMC. Bayesian Analysis, 16(2):667 – 718, 2021.
  84. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2):1–305, 2008.
  85. Flow matching for scalable simulation-based inference. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  86. Discriminative calibration: Check bayesian computation from simulations and flexible classifier. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  87. Simulation based stacking. arXiv preprint arXiv:2310.17009, 2023.
  88. Comparing distributions by measuring differences that affect decision making. In International Conference on Learning Representations, 2022.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets