Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Generative Modeling with Flow-Guided Density Ratio Learning (2303.03714v3)

Published 7 Mar 2023 in cs.LG and cs.AI

Abstract: We present Flow-Guided Density Ratio Learning (FDRL), a simple and scalable approach to generative modeling which builds on the stale (time-independent) approximation of the gradient flow of entropy-regularized f-divergences introduced in recent work. Specifically, the intractable time-dependent density ratio is approximated by a stale estimator given by a GAN discriminator. This is sufficient in the case of sample refinement, where the source and target distributions of the flow are close to each other. However, this assumption is invalid for generation and a naive application of the stale estimator fails due to the large chasm between the two distributions. FDRL proposes to train a density ratio estimator such that it learns from progressively improving samples during the training process. We show that this simple method alleviates the density chasm problem, allowing FDRL to generate images of dimensions as high as $128\times128$, as well as outperform existing gradient flow baselines on quantitative benchmarks. We also show the flexibility of FDRL with two use cases. First, unconditional FDRL can be easily composed with external classifiers to perform class-conditional generation. Second, FDRL can be directly applied to unpaired image-to-image translation with no modifications needed to the framework. Our code is publicly available at ttps://github.com/clear-nus/fdrl.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Refining deep generative models via discriminator gradient flow. In ICLR, 2021.
  2. Maximum mean discrepancy gradient flow. Advances in Neural Information Processing Systems, 32, 2019.
  3. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8188–8197, 2020.
  4. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  5. Nonparametric generative modeling with conditional and locally-connected sliced-wasserstein flows. arXiv preprint arXiv:2305.02164, 2023.
  6. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689, 2019.
  7. Robustness (python library), 2019.
  8. Variational wasserstein gradient flow. arXiv preprint arXiv:2112.02424, 2021.
  9. Deep generative learning via euler particle transport. In Mathematical and Scientific Machine Learning, pages 336–368. PMLR, 2022.
  10. Deep generative learning via variational gradient flow. In International Conference on Machine Learning, pages 2093–2101. PMLR, 2019.
  11. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  12. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
  13. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  14. The variational formulation of the fokker–planck equation. SIAM journal on mathematical analysis, 29(1):1–17, 1998.
  15. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
  16. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pages 35–51, 2018.
  17. Understanding mcmc dynamics as flows on the wasserstein space. In International Conference on Machine Learning, pages 4093–4103. PMLR, 2019.
  18. Sliced-wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. In International Conference on Machine Learning, pages 4104–4113. PMLR, 2019.
  19. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
  20. Large-scale wasserstein gradient flows. Advances in Neural Information Processing Systems, 34:15243–15256, 2021.
  21. On the convergence of gradient descent in gans: Mmd gan as a gradient flow. In International Conference on Artificial Intelligence and Statistics, pages 1720–1728. PMLR, 2021.
  22. Sobolev descent. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2976–2985. PMLR, 2019.
  23. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34:13497–13510, 2021.
  24. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems, 32, 2019.
  25. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33:21994–22008, 2020.
  26. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905–4916, 2020.
  27. The fokker-planck equation, methods of solution and applications. Journal of the Optical Society of America B Optical Physics, 2(3):508, 1985.
  28. Filippo Santambrogio. {{\{{Euclidean, metric, and Wasserstein}}\}} gradient flows: an overview. Bulletin of Mathematical Sciences, 7(1):87–154, 2017.
  29. Image synthesis with a single (robust) classifier. Advances in Neural Information Processing Systems, 32, 2019.
  30. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  31. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
  32. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  33. Density ratio estimation in machine learning. Cambridge University Press, 2012.
  34. Density-ratio matching under the bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64(5):1009–1044, 2012.
  35. Vaebm: A symbiosis between variational autoencoders and energy-based models. arXiv preprint arXiv:2010.00654, 2020.
  36. Cooperative learning of energy-based model and latent variable model via mcmc teaching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  37. A theory of generative convnet. In International Conference on Machine Learning, pages 2635–2644. PMLR, 2016.
  38. Unpaired image-to-image translation via latent energy transport. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16418–16427, 2021.
  39. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
Citations (1)

Summary

We haven't generated a summary for this paper yet.

Github Logo Streamline Icon: https://streamlinehq.com

GitHub