Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Analyzing and Improving Optimal-Transport-based Adversarial Networks (2310.02611v2)

Published 4 Oct 2023 in cs.LG and cs.CV

Abstract: Optimal Transport (OT) problem aims to find a transport plan that bridges two distributions while minimizing a given cost function. OT theory has been widely utilized in generative modeling. In the beginning, OT distance has been used as a measure for assessing the distance between data and generated distributions. Recently, OT transport map between data and prior distributions has been utilized as a generative model. These OT-based generative models share a similar adversarial training objective. In this paper, we begin by unifying these OT-based adversarial methods within a single framework. Then, we elucidate the role of each component in training dynamics through a comprehensive analysis of this unified framework. Moreover, we suggest a simple but novel method that improves the previously best-performing OT-based model. Intuitively, our approach conducts a gradual refinement of the generated distribution, progressively aligning it with the data distribution. Our approach achieves a FID score of 2.51 on CIFAR-10 and 5.99 on CelebA-HQ-256, outperforming unified OT-based adversarial approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428–21439, 2020.
  2. Ae-ot: A new generative model based on extended semi-discrete optimal transport. ICLR, 2020a.
  3. Ae-ot-gan: Training gans from data specific latent distribution. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16, pp.  548–564. Springer, 2020b.
  4. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480–493, 2021.
  5. Refining deep generative models via discriminator gradient flow. arXiv preprint arXiv:2012.00780, 2020.
  6. Wasserstein generative adversarial networks. In International conference on machine learning, pp.  214–223. PMLR, 2017.
  7. Robust optimal transport with applications in generative modeling and domain adaptation. Advances in Neural Information Processing Systems, 33:12934–12944, 2020.
  8. Generative modeling through the semi-dual formulation of unbalanced optimal transport. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
  9. Restoration based generative models. In Proceedings of the 40th International Conference on Machine Learning, volume 202. PMLR, 2023b.
  10. Imre Csiszár. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica, 2(1-4):191–213, 1972.
  11. Score-based generative modeling with critically-damped langevin diffusion. The International Conference on Learning Representations, 2022.
  12. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  12873–12883, 2021.
  13. Scalable computation of monge maps with general costs. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
  14. Unbalanced minibatch optimal transport; applications to domain adaptation. In International Conference on Machine Learning, pp.  3186–3197. PMLR, 2021.
  15. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell, 1, 2016.
  16. Learning energy-based models by diffusion recovery likelihood. Advances in neural information processing systems, 2021.
  17. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  3224–3234, 2019.
  18. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  19. Multi-source domain adaptation via optimal transport for brain dementia identification. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp.  1514–1517. IEEE, 2021.
  20. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
  21. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  22. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  23. Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.07074, 1(3), 2021.
  24. Subspace diffusion generative models. arXiv preprint arXiv:2205.01490, 2022.
  25. Leonid Vitalevich Kantorovich. On a problem of monge. Uspekhi Mat. Nauk, pp.  225–226, 1948.
  26. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
  27. A style-based generator architecture for generative adversarial networks. 2018.
  28. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104–12114, 2020.
  29. Disconnected manifold learning for generative adversarial networks. Advances in Neural Information Processing Systems, 31, 2018.
  30. Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527, 2021.
  31. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
  32. Neural optimal transport. In The Eleventh International Conference on Learning Representations, 2023.
  33. Learning multiple layers of features from tiny images. 2009.
  34. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
  35. Optimal entropy-transport problems and a new hellinger–kantorovich distance between positive measures. Inventiones mathematicae, 211(3):969–1117, 2018.
  36. Wasserstein gan with quadratic transport cost. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  4832–4841, 2019.
  37. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Skq89Scxx.
  38. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pp.  6672–6681. PMLR, 2020.
  39. Non-asymptotic convergence bounds for wasserstein approximation using point clouds. Advances in Neural Information Processing Systems, 34:12810–12821, 2021.
  40. Which training methods for gans do actually converge? In International conference on machine learning, pp.  3481–3490. PMLR, 2018.
  41. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
  42. Gradient descent gan optimization is locally stable. Advances in neural information processing systems, 30, 2017.
  43. Is generator conditioning causally related to gan performance? In International conference on machine learning, pp.  3849–3858. PMLR, 2018.
  44. A Course in Game Theory. The MIT Press, 1994. ISBN 0262150417.
  45. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  823–832, 2021.
  46. On the regularization of wasserstein gans. In International Conference on Learning Representations, 2018.
  47. Computational optimal transport. Center for Research in Economics and Statistics Working Papers, (2017-86), 2017.
  48. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  14104–14113, 2020.
  49. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
  50. Stabilizing training of generative adversarial networks through regularization. Advances in neural information processing systems, 30, 2017.
  51. Generative modeling with optimal transport maps. In International Conference on Learning Representations, 2022.
  52. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, 2016.
  53. Can push-forward generative models fit multimodal distributions? Advances in Neural Information Processing Systems, 35:10766–10779, 2022.
  54. On the convergence and robustness of training gans with regularized optimal transport. Advances in Neural Information Processing Systems, 31, 2018.
  55. Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94, 2015.
  56. f𝑓fitalic_f-divergence inequalities. IEEE Transactions on Information Theory, 62(11):5973–6006, 2016.
  57. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  58. Denoising diffusion implicit models. Advances in Neural Information Processing Systems, 2021a.
  59. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  60. Score-based generative modeling through stochastic differential equations. The International Conference on Learning Representations, 2021b.
  61. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667–19679, 2020.
  62. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
  63. Pixel recurrent neural networks. In International conference on machine learning, pp.  1747–1756. PMLR, 2016.
  64. Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
  65. Vaebm: A symbiosis between variational autoencoders and energy-based models. arXiv preprint arXiv:2010.00654, 2020.
  66. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804, 2021.
  67. On scalable and efficient computation of large scale optimal transport. volume 97 of proceedings of machine learning research. Long Beach, California, USA, pp.  09–15, 2019.
  68. KD Yang and C Uhler. Scalable unbalanced optimal transport using generative adversarial networks. In International Conference on Learning Representations, 2019.
  69. Styleswin: Transformer-based gan for high-resolution image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11304–11314, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Jaemoo Choi (13 papers)
  2. Jaewoong Choi (26 papers)
  3. Myungjoo Kang (45 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now