Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors (2401.02739v4)
Abstract: We propose denoising diffusion variational inference (DDVI), a black-box variational inference algorithm for latent variable models which relies on diffusion models as flexible approximate posteriors. Specifically, our method introduces an expressive class of diffusion-based variational posteriors that perform iterative refinement in latent space; we train these posteriors with a novel regularized evidence lower bound (ELBO) on the marginal likelihood inspired by the wake-sleep algorithm. Our method is easy to implement (it fits a regularized extension of the ELBO), is compatible with black-box variational inference, and outperforms alternative classes of approximate posteriors based on normalizing flows or adversarial networks. We find that DDVI improves inference and learning in deep latent variable models across common benchmarks as well as on a motivating task in biology -- inferring latent ancestry from human genomes -- where it outperforms strong baselines on the Thousand Genomes dataset.
- Trimap: Large-scale dimensionality reduction using triplets. arXiv preprint arXiv:1910.00204, 2019.
- Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021.
- Linear discriminant analysis-a brief tutorial. Institute for Signal and information Processing, 18(1998):1–8, 1998.
- The isomap algorithm and topological stability. Science, 295(5552):7–7, 2002.
- Visualizing population structure with variational autoencoders. G3, 11(1):jkaa036, 2021.
- Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373–1396, 2003.
- Multidimensional scaling. Measurement, judgment and decision making, pp. 179–250, 1998.
- Generating long sequences with sparse transformers, 2019.
- Diffusion bridges vector quantized variational autoencoders. arXiv preprint arXiv:2202.04895, 2022.
- Diffusion maps. Applied and computational harmonic analysis, 21(1):5–30, 2006.
- Calibrated uncertainty estimation improves bayesian optimization, 2023.
- Deep multi-modal structural equations for causal effect estimation with unstructured proxies. Advances in Neural Information Processing Systems, 35:10931–10944, 2022.
- Umap reveals cryptic population structure and phenotype heterogeneity in large genomic cohorts. PLoS genetics, 15(11):e1008432, 2019.
- Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
- Diffusion maps for high-dimensional single-cell analysis of differentiation data. Bioinformatics, 31(18):2989–2998, 2015.
- Paul Henderson. Sammon mapping. Pattern Recognit. Lett, 18(11-13):1307–1316, 1997.
- beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2016.
- The” wake-sleep” algorithm for unsupervised neural networks. Science, 268(5214):1158–1161, 1995.
- Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
- Denoising criterion for variational auto-encoding framework. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
- Composing graphical models with neural networks for structured representations and fast inference. In Advances in Neural Information Processing Systems (NIPS) 29, 2016. arXiv:1603.06277 [stat.ML].
- Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021.
- Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
- Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
- Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
- Learning multiple layers of features from tiny images. 2009.
- Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791.
- Auxiliary deep generative models. In International conference on machine learning, pp. 1445–1453. PMLR, 2016.
- Adversarial autoencoders. arXiv preprint arXiv:1511.05644, 2015.
- Stephen Marsland. Machine Learning: An Algorithmic Perspective (2nd Edition). Chapman and Hall/CRC, 2014.
- Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
- On incorporating inductive biases into vaes. arXiv preprint arXiv:2106.13746, 2021.
- Phate: a dimensionality reduction method for visualizing trajectory structures in high-dimensional biological data. BioRxiv, 120378, 2017.
- Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16569–16594. PMLR, 2022. URL https://proceedings.mlr.press/v162/nguyen22b.html.
- Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162–8171. PMLR, 2021.
- Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619–10629, 2022.
- Semi-parametric inducing point networks and neural processes. 2023.
- Variational inference with normalizing flows. In International conference on machine learning, pp. 1530–1538. PMLR, 2015.
- High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022.
- Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2021. doi: 10.1162/tacl˙a˙00353. URL https://aclanthology.org/2021.tacl-1.4.
- Markov chain monte carlo and variational inference: Bridging the gap. In International conference on machine learning, pp. 1218–1226. PMLR, 2015.
- Autoregressive quantile flows for predictive uncertainty estimation. In International Conference on Learning Representations, 2022.
- Semi-autoregressive energy flows: Exploring likelihood-free training of normalizing flows. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31732–31753. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/si23a.html.
- Nayanah Siva. 1000 genomes project. Nature biotechnology, 26(3):256–257, 2008.
- Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
- Visualizing large-scale and high-dimensional data. In Proceedings of the 25th international conference on world wide web, pp. 287–297, 2016.
- Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pp. 1214–1223. PMLR, 2018.
- Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667–19679, 2020.
- Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
- Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
- Understanding how dimension reduction tools work: an empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization. The Journal of Machine Learning Research, 22(1):9129–9201, 2021.
- InfoDiffusion: Representation learning using information maximizing diffusion models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 36336–36354. PMLR, 23–29 Jul 2023a. URL https://proceedings.mlr.press/v202/wang23ah.html.
- Infodiffusion: Representation learning using information maximizing diffusion models. arXiv preprint arXiv:2306.08757, 2023b.
- Diffusion priors in variational autoencoders. arXiv preprint arXiv:2106.15671, 2021.
- Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37–52, 1987.
- Unsupervised representation learning from pre-trained diffusion probabilistic models. Advances in Neural Information Processing Systems, 35:22117–22130, 2022.
- Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262, 2017.