Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
130 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
55 tokens/sec
2000 character limit reached

Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality (2410.18784v2)

Published 24 Oct 2024 in cs.LG, cs.NA, eess.SP, math.NA, math.ST, stat.ML, and stat.TH

Abstract: The denoising diffusion probabilistic model (DDPM) has emerged as a mainstream generative model in generative AI. While sharp convergence guarantees have been established for the DDPM, the iteration complexity is, in general, proportional to the ambient data dimension, resulting in overly conservative theory that fails to explain its practical efficiency. This has motivated the recent work Li and Yan (2024a) to investigate how the DDPM can achieve sampling speed-ups through automatic exploitation of intrinsic low dimensionality of data. We strengthen this line of work by demonstrating, in some sense, optimal adaptivity to unknown low dimensionality. For a broad class of data distributions with intrinsic dimension $k$, we prove that the iteration complexity of the DDPM scales nearly linearly with $k$, which is optimal when using KL divergence to measure distributional discrepancy. Notably, our work is closely aligned with the independent concurrent work Potaptchik et al. (2024) -- posted two weeks prior to ours -- in establishing nearly linear-$k$ convergence guarantees for the DDPM.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Anderson, B. D. (1982). Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313–326.
  2. Convergence of diffusion models under the manifold hypothesis in high-dimensions. arXiv preprint arXiv:2409.18804.
  3. Nearly $d$-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations.
  4. Intrinsic dimension estimation using wasserstein distance. Journal of Machine Learning Research, 23(313):1–37.
  5. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. arXiv preprint arXiv:2211.01916.
  6. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735–4763. PMLR.
  7. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning, pages 4672–4712. PMLR.
  8. Opportunities and challenges of diffusion models for generative ai. National Science Review, page nwae348.
  9. The probability flow ode is provably fast. arXiv preprint arXiv:2305.11798.
  10. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215.
  11. Restoration-degradation beyond linear diffusions: A non-asymptotic analysis for DDIM-type samplers. arXiv preprint arXiv:2303.03384.
  12. Exploring low-dimensional subspaces in diffusion models for controllable image editing. arXiv preprint arXiv:2409.02374.
  13. Random projection trees and low dimensional manifolds. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pages 537–546.
  14. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794.
  15. From optimal score matching to optimal sampling. arXiv preprint arXiv:2409.07032.
  16. Efron, B. (2011). Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614.
  17. An information-theoretic view of stochastic localization. IEEE Transactions on Information Theory, 68(11):7423–7426.
  18. Eldan, R. (2020). Taming correlations through entropy-efficient measure decompositions with applications to mean-field approximation. Probability Theory and Related Fields, 176(3):737–755.
  19. Convergence analysis for general probability flow odes of diffusion models in wasserstein distances. arXiv preprint arXiv:2401.17958.
  20. Time reversal of diffusions. The Annals of Probability, pages 1188–1205.
  21. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851.
  22. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646.
  23. Convergence analysis of probability flow ode for score-based generative models. arXiv preprint arXiv:2404.09730.
  24. Generalization in diffusion models arises from geometry-adaptive harmonic representation. arXiv preprint arXiv:2310.02557.
  25. A tree-based regressor that adapts to intrinsic dimension. Journal of Computer and System Sciences, 78(5):1496–1515.
  26. Le Gall, J.-F. (2016). Brownian motion, martingales, and stochastic calculus. Springer.
  27. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870–22882.
  28. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pages 946–985. PMLR.
  29. Lee, J. M. (2018). Introduction to Riemannian manifolds, volume 2. Springer.
  30. Accelerating convergence of score-based diffusion models, provably. arXiv preprint arXiv:2403.03852.
  31. Towards a mathematical theory for consistency training in diffusion models. arXiv preprint arXiv:2402.07802.
  32. Improved convergence rate for diffusion probabilistic models. arXiv preprint arXiv:2410.13738.
  33. Towards faster non-asymptotic convergence for diffusion-based generative models. arXiv preprint arXiv:2306.09251.
  34. A sharp convergence theory for the probability flow ODEs of diffusion models. arXiv preprint arXiv:2408.02320.
  35. Adapting to unknown low-dimensional structures in score-based diffusion models. arXiv preprint arXiv:2405.14861.
  36. O⁢(d/T)𝑂𝑑𝑇O(d/T)italic_O ( italic_d / italic_T ) convergence theory for diffusion probabilistic models under minimal assumptions. arXiv preprint arXiv:2409.18959.
  37. Broadening target distributions for accelerated diffusion models via a novel analysis approach. arXiv preprint arXiv:2402.13901.
  38. A note on the convergence of denoising diffusion probabilistic models. arXiv preprint arXiv:2312.05989.
  39. Manifold learning-based methods for analyzing single-cell rna-sequencing data. Current Opinion in Systems Biology, 7:36–46.
  40. Stochastic differential equations. Springer.
  41. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894.
  42. Linear convergence of diffusion models under the manifold hypothesis. arXiv preprint arXiv:2410.09046.
  43. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3.
  44. Natural image statistics and neural representation. Annual review of neuroscience, 24(1):1193–1216.
  45. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265.
  46. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations.
  47. Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statistics, pages 1648–1656. PMLR.
  48. Tang, W. (2023). Diffusion probabilistic models. preprint.
  49. Score-based diffusion models via stochastic differential equations–a technical tutorial. arXiv preprint arXiv:2402.07487.
  50. Vershynin, R. (2018). High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press.
  51. Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press.
  52. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426.
  53. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089–1100.
  54. Stochastic runge-kutta methods: Provable acceleration of diffusion models. arXiv preprint arXiv:2410.04760.
  55. Reconstructing spatial organizations of chromosomes through manifold learning. Nucleic acids research, 46(8):e50–e50.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now