Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates (2311.13584v4)

Published 22 Nov 2023 in cs.LG, math.OC, math.PR, and stat.ML

Abstract: We provide full theoretical guarantees for the convergence behaviour of diffusion-based generative models under the assumption of strongly log-concave data distributions while our approximating class of functions used for score estimation is made of Lipschitz continuous functions avoiding any Lipschitzness assumption on the score function. We demonstrate via a motivating example, sampling from a Gaussian distribution with unknown mean, the powerfulness of our approach. In this case, explicit estimates are provided for the associated optimization problem, i.e. score approximation, while these are combined with the corresponding sampling estimates. As a result, we obtain the best known upper bound estimates in terms of key quantities of interest, such as the dimension and rates of convergence, for the Wasserstein-2 distance between the data distribution (Gaussian with unknown mean) and our sampling algorithm. Beyond the motivating example and in order to allow for the use of a diverse range of stochastic optimizers, we present our results using an $L2$-accurate score estimation assumption, which crucially is formed under an expectation with respect to the stochastic optimizer and our novel auxiliary process that uses only known information. This approach yields the best known convergence rate for our sampling algorithm.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. On stochastic gradient Langevin dynamics with dependent data streams in the logconcave case. Bernoulli, 27(1):1–33, 2021.
  2. Nearly d𝑑ditalic_d-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations, 2024.
  3. Generative modeling with denoising auto-encoders and Langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
  4. V. D. Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022.
  5. Tweedie moment projected diffusions for inverse problems. arXiv preprint arXiv:2310.06721, 2023.
  6. Monte carlo guided denoising diffusion models for Bayesian linear inverse problems. In The Twelfth International Conference on Learning Representations, 2024.
  7. Time reversal of diffusion processes under a finite entropy condition. In Annales de l’Institut Henri Poincaré (B) Probabilités et Statistiques, volume 59, pages 1844–1881, 2023.
  8. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735–4763, 2023a.
  9. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 4672–4712, 2023b.
  10. The probability flow ODE is provably fast. In Thirty-seventh Conference on Neural Information Processing Systems, 2023c.
  11. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, 2023d.
  12. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2022.
  13. Score diffusion models without early stopping: finite Fisher information is all you need. arXiv preprint arXiv:2308.12240, 2023.
  14. P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34, 2021.
  15. Efficient and accurate estimation of Lipschitz constants for deep neural networks. Advances in neural information processing systems, 32, 2019.
  16. Wasserstein convergence guarantees for a general class of score-based generative models. arXiv preprint arXiv:2311.11003, 2023.
  17. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  18. U. G. Haussmann and E. Pardoux. Time reversal of diffusions. The Annals of Probability, pages 1188–1205, 1986.
  19. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
  20. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  21. A. Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695–709, 2005.
  22. Strong error analysis for stochastic gradient descent optimization algorithms. IMA Journal of Numerical Analysis, 41(1):455–492, 2021.
  23. D. Kingma and J. Ba. ADAM: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  24. D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In The Eleventh International Conference on Learning Representations, 2014.
  25. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020.
  26. N. V. Krylov. A simple proof of the existence of a solution of Itô’s equation with monotone coefficients. Theory of Probability & Its Applications, 35(3):583–587, 1991.
  27. C. Kumar and S. Sabanis. On Milstein approximations with varying coefficients: The case of super-linear diffusion coefficients. BIT Numerical Mathematics, 59(4):929–968, 2019.
  28. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870–22882, 2022.
  29. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pages 946–985, 2023.
  30. Towards non-asymptotic convergence for diffusion-based generative models. In The Twelfth International Conference on Learning Representations, 2024.
  31. D.-Y. Lim and S. Sabanis. Polygonal unadjusted Langevin algorithms: Creating stable and efficient adaptive algorithms for neural networks. Journal of Machine Learning Research, 25(53):1–52, 2024.
  32. Diffusion models are minimax optimal distribution estimators. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 26517–26582, 2023.
  33. Improved convergence of score-based diffusion models via prediction-correction. arXiv preprint arXiv:2305.14164, 2023.
  34. D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530–1538, 2015.
  35. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  36. Can push-forward generative models fit multimodal distributions? Advances in Neural Information Processing Systems, 35:10766–10779, 2022.
  37. A. Saumard and J. A. Wellner. Log-concavity and strong log-concavity: A review. Statistics Surveys, pages 45 – 114, 2014.
  38. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265, 2015.
  39. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations, 2022.
  40. Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32, 2019.
  41. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  42. P. Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
  43. A. Virmaux and K. Scaman. Lipschitz regularity of deep neural networks: Analysis and efficient estimation. In Advances in Neural Information Processing Systems, 2018.
  44. K. Y. Yang and A. Wibisono. Convergence of the inexact Langevin algorithm and score-based generative models in KL divergence. arXiv preprint arXiv:2211.01512, 2023.
  45. Energy-based generative adversarial networks. In International Conference on Learning Representations, 2017.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets