Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Generative Modelling with High-Order Langevin Dynamics (2404.12814v3)

Published 19 Apr 2024 in cs.LG, cs.AI, and cs.CV

Abstract: Diffusion generative modelling (DGM) based on stochastic differential equations (SDEs) with score matching has achieved unprecedented results in data generation. In this paper, we propose a novel fast high-quality generative modelling method based on high-order Langevin dynamics (HOLD) with score matching. This motive is proved by third-order Langevin dynamics. By augmenting the previous SDEs, e.g. variance exploding or variance preserving SDEs for single-data variable processes, HOLD can simultaneously model position, velocity, and acceleration, thereby improving the quality and speed of the data generation at the same time. HOLD is composed of one Ornstein-Uhlenbeck process and two Hamiltonians, which reduce the mixing time by two orders of magnitude. Empirical experiments for unconditional image generation on the public data set CIFAR-10 and CelebA-HQ show that the effect is significant in both Frechet inception distance (FID) and negative log-likelihood, and achieves the state-of-the-art FID of 1.85 on CIFAR-10.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. B. D. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313–326, 1982.
  2. H. F. Baker. Further applications of metrix notation to integration problems. Proceedings of the London Mathematical Society, 1(1):347–360, 1901.
  3. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07309, 2022.
  4. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1xsqj09Fm.
  5. J. Campbell. On a law of combination of operators bearing on the theory of continuous transformation groups. Proceedings of the London Mathematical Society, 1(1):381–390, 1896.
  6. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  7. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
  8. Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:2112.07068, 2021.
  9. Genie: Higher-order denoising diffusion solvers. arXiv preprint arXiv:2210.05475, 2022.
  10. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19–26, 1980.
  11. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations, 2020.
  12. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, volume 27, pages 2672–2680, 2014.
  13. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018.
  14. F. Hausdorff. Die symbolische exponentialformel in der gruppentheorie. Ber. Verh. Kgl. SÃ chs. Ges. Wiss. Leipzig., Math.-phys. Kl., 58:19–48, 1906.
  15. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  16. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020.
  17. A. Hyvärinen and P. Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
  18. Gotta go fast with score-based generative models. In The Symbiosis of Deep Learning and Differential Equations, 2021.
  19. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  20. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104–12114, 2020.
  21. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.
  22. Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527, 2021.
  23. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  24. D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
  25. D. P. Kingma and M. Welling. Auto-encoding variational bayes. In ICLR 2014 : International Conference on Learning Representations (ICLR) 2014, 2014.
  26. Z. Kong and W. Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132, 2021.
  27. B. Leimkuhler and C. Matthews. Molecular Dynamics: With Deterministic and Stochastic Numerical Methods, volume 39. Springer, 2015.
  28. Diffusion-lm improves controllable text generation. arXiv preprint arXiv:2205.14217, 2022.
  29. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778, 2022.
  30. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.
  31. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
  32. High-order langevin diffusion yields an accelerated mcmc algorithm. J. Mach. Learn. Res., 22:42–1, 2021.
  33. R. M. Neal. Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
  34. A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  35. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 823–832, 2021.
  36. Diffusion-based voice conversion with fast maximum likelihood sampling scheme. arXiv preprint arXiv:2109.13821, 2021.
  37. E. J. Putzer. Avoiding the jordan canonical form in the discussion of linear systems with constant coefficients. The American Mathematical Monthly, 73(1):2–7, 1966.
  38. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  39. S. Särkkä and A. Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
  40. Z. Shi and S. Wu. Itôn: End-to-end audio generation with itô stochastic differential equations. Digital Signal Processing, page 103781, 2022.
  41. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  42. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020a.
  43. Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv:1907.05600, 2019.
  44. Y. Song and S. Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
  45. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  46. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415–1428, 2021.
  47. G. Strang. On the construction and comparison of difference schemes. SIAM journal on numerical analysis, 5(3):506–517, 1968.
  48. H. F. Trotter. On the product of semi-groups of operators. Proceedings of the American Mathematical Society, 10(4):545–551, 1959.
  49. M. Tuckerman. Statistical mechanics: theory and molecular simulation. Oxford university press, 2010.
  50. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
  51. P. Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
  52. Deep generative learning via schrödinger bridge. In International Conference on Machine Learning, pages 10794–10804. PMLR, 2021.
  53. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint arXiv:2106.03802, 2021.
  54. Vaebm: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2020.
  55. Pfgm++: Unlocking the potential of physics-inspired generative models. arXiv preprint arXiv:2302.04265, 2023.
  56. Q. Zhang and Y. Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets