Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unifying GANs and Score-Based Diffusion as Generative Particle Models (2305.16150v3)

Published 25 May 2023 in cs.LG, cs.CV, cs.NE, and stat.ML

Abstract: Particle-based deep generative models, such as gradient flows and score-based diffusion models, have recently gained traction thanks to their striking performance. Their principle of displacing particle distributions using differential equations is conventionally seen as opposed to the previously widespread generative adversarial networks (GANs), which involve training a pushforward generator network. In this paper we challenge this interpretation, and propose a novel framework that unifies particle and adversarial generative models by framing generator training as a generalization of particle models. This suggests that a generator is an optional addition to any such generative model. Consequently, integrating a generator into a score-based diffusion model and training a GAN without a generator naturally emerge from our framework. We empirically test the viability of these original models as proofs of concepts of potential applications of our framework.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Optimizing functionals on the space of probabilities with input convex neural networks. Transactions on Machine Learning Research, 2022.
  2. Refining deep generative models via discriminator gradient flow. In International Conference on Learning Representations, 2021.
  3. Maximum mean discrepancy gradient flow. In Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp.  6484–6494. Curran Associates, Inc., 2019.
  4. Wasserstein generative adversarial networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.  214–223. PMLR, August 2017.
  5. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, August 2013.
  6. signSGD: Compressed optimisation for non-convex problems. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.  560–569. PMLR, July 2018.
  7. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
  8. Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F., and Lin, H.-T. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  12275–12287. Curran Associates, Inc., 2020.
  9. The equivalence between Stein variational gradient descent and black-box variational inference. arXiv preprint arXiv:2004.01822, 2020.
  10. DiffEdit: Diffusion-based semantic image editing with mask guidance. In International Conference on Learning Representations, 2023.
  11. Diffusion models beat GANs on image synthesis. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  8780–8794. Curran Associates, Inc., 2021.
  12. On the geometry of Steinl’heurrrr variational gradient descent. Journal of Machine Learning Research, 24(56):1–39, 2023.
  13. Effective dynamics of generative adversarial networks. arXiv preprint arXiv:2212.04580, 2022.
  14. Fallis, D. The epistemic threat of deepfakes. Philosophy & Technology, 34:623––643, 2021.
  15. Variational Wasserstein gradient flow. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  6185–6215. PMLR, July 2022.
  16. A neural tangent kernel perspective of GANs. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  6660–6704. PMLR, July 2022.
  17. Deep generative learning via variational gradient flow. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.  2093–2101. PMLR, June 2019.
  18. KALE flow: A relaxed KL gradient flow for probabilities with disjoint support. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  8018–8031. Curran Associates, Inc., 2021.
  19. Goodfellow, I. NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160, 2016.
  20. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems, volume 27, pp.  2672–2680. Curran Associates, Inc., 2014.
  21. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012.
  22. Improved training of Wasserstein GANs. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30, pp.  5769–5779. Curran Associates, Inc., 2017.
  23. Gaussian error linear units (GeLUs). arXiv preprint arXiv:1606.08415, 2016.
  24. Deep generative Wasserstein gradient flows, 2023. URL https://openreview.net/forum?id=zjSeBTEdXp1.
  25. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30, pp.  6629––6640. Curran Associates, Inc., 2017.
  26. Imagen Video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
  27. The limits of min-max optimization algorithms: Convergence to spurious non-critical sets. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.  4337–4348. PMLR, July 2021.
  28. Hyvärinen, A. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695–709, 2005.
  29. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp.  448–456, Lille, France, July 2015. PMLR.
  30. Neural tangent kernel: Convergence and generalization in neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31, pp.  8580–8589. Curran Associates, Inc., 2018.
  31. Dissecting adaptive methods in GANs. arXiv preprint arXiv:2210.04319, 2022.
  32. Adversarial score matching and improved sampling for image generation. In International Conference on Learning Representations, 2021.
  33. The variational formulation of the Fokker-Planck equation. SIAM Journal on Mathematical Analysis, 29(1):1–17, 1998.
  34. StudioGAN: A taxonomy and benchmark of GANs for image synthesis. arXiv preprint arXiv:2206.09479, 2022.
  35. Elucidating the design space of diffusion-based generative models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  26565–26577. Curran Associates, Inc., 2022.
  36. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  37. Klebaner, F. C. Introduction to Stochastic Calculus with Applications. Imperial College Press, 3rd edition, 2012.
  38. Introduction to Stochastic Time Discrete Approximation, pp.  305–337. Applications of Mathematics. Springer Berlin Heidelberg, Berlin - Heidelberg, Germany, 1992.
  39. Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, April 2009.
  40. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86, 1951.
  41. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998.
  42. Distribution-interpolation trade off in generative models. In International Conference on Learning Representations, 2018.
  43. Diffusion-LM improves controllable text generation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  4328–4343. Curran Associates, Inc., 2022.
  44. Geometric GAN. arXiv preprint arXiv:1705.02894, 2017.
  45. Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2021.
  46. On the linearity of large non-linear models: when and why the tangent kernel is constant. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F., and Lin, H.-T. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  15954–15964. Curran Associates, Inc., 2020.
  47. Generative adversarial network for abstractive text summarization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI’18, pp.  8109–8110. AAAI Press, 2018.
  48. Liu, Q. Stein variational gradient descent as gradient flow. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30, pp.  3118––3126. Curran Associates, Inc., 2017.
  49. Stein variational gradient descent: A general purpose Bayesian inference algorithm. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29, pp.  2378–2386. Curran Associates, Inc., 2016.
  50. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision (ICCV), pp.  3730–3738, December 2015.
  51. Sliced-Wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.  4104–4113. PMLR, June 2019.
  52. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pp.  48–55, Virtual, June 2021. Association for Computational Linguistics.
  53. Unrolled generative adversarial networks. In International Conference on Learning Representations, 2017.
  54. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
  55. Large-scale Wasserstein gradient flows. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  15243–15256. Curran Associates, Inc., 2021.
  56. Sobolev descent. In Chaudhuri, K. and Sugiyama, M. (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp.  2976–2985. PMLR, April 2019.
  57. Müller, A. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429–443, 1997.
  58. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  16784–16804. PMLR, July 2022.
  59. f𝑓fitalic_f-GAN: Training generative neural samplers using variational divergence minimization. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29, pp.  271––279. Curran Associates, Inc., 2016.
  60. PyTorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp.  8026–8037. Curran Associates, Inc., 2019.
  61. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations, 2016.
  62. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  10684–10695, June 2022.
  63. U-Net: Convolutional networks for biomedical image segmentation. In Navab, N., Hornegger, J., Wells, W. M., and Frangi, A. F. (eds.), Medical Image Computing and Computer-Assisted Intervention — MICCAI 2015, pp.  234–241, Cham, Switzerland, 2015. Springer International Publishing.
  64. Rényi, A. On measures of entropy and information. In Neyman, J. (ed.), Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pp.  547–561. University of California Press, 1961.
  65. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
  66. Santambrogio, F. {{\{{Euclidean, metric, and Wasserstein}}\}} gradient flows: an overview. Bulletin of Mathematical Sciences, 7:87–154, 2017.
  67. Generative models for spear phishing posts on social media. arXiv preprint arXiv:1802.05196, 2018.
  68. TorchMetrics – measuring reproducibility in PyTorch, February 2022. URL https://github.com/Lightning-AI/torchmetrics.
  69. Song, Y. Generative modeling by estimating gradients of the data distribution. https://yang-song.net/blog/2021/score/, 2021. Accessed: 2023-05-17.
  70. Generative modeling by estimating gradients of the data distribution. In Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp.  11918–11930. Curran Associates, Inc., 2019.
  71. Sliced score matching: A scalable approach to density and score estimation. In Adams, R. P. and Gogate, V. (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp.  574–584. PMLR, July 2020.
  72. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  73. Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research, 11(50):1517–1561, 2010.
  74. Tanaka, A. Discriminator optimal transport. In Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp.  6816––6826. Curran Associates, Inc., 2019.
  75. Learning disconnected manifolds: a no GAN’s land. In Daumé, III, H. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  9418–9427. PMLR, July 2020.
  76. Vincent, P. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661––1674, July 2011.
  77. MCVD - masked conditional video diffusion for prediction, generation, and interpolation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  23371–23385. Curran Associates, Inc., 2022.
  78. Man-in-the-middle attacks against machine learning classifiers via malicious generative models. IEEE Transactions on Dependable and Secure Computing, 18(5):1941–0018, September 2021.
  79. Diffusion-GAN: Training GANs with diffusion. arXiv preprint arXiv:2206.02262, 2022.
  80. On accelerated methods in optimization. arXiv preprint arXiv:1509.03616, 2015.
  81. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations, 2022.
  82. MonoFlow: Rethinking divergence GANs via the perspective of Wasserstein gradient flows. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  39984–40000. PMLR, July 2023.
  83. Self-attention generative adversarial networks. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.  7354–7363. PMLR, June 2019.
Citations (16)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now