Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Stabilizing GANs' Training with Brownian Motion Controller (2306.10468v1)

Published 18 Jun 2023 in cs.LG and math.DS

Abstract: The training process of generative adversarial networks (GANs) is unstable and does not converge globally. In this paper, we examine the stability of GANs from the perspective of control theory and propose a universal higher-order noise-based controller called Brownian Motion Controller (BMC). Starting with the prototypical case of Dirac-GANs, we design a BMC to retrieve precisely the same but reachable optimal equilibrium. We theoretically prove that the training process of DiracGANs-BMC is globally exponential stable and derive bounds on the rate of convergence. Then we extend our BMC to normal GANs and provide implementation instructions on GANs-BMC. Our experiments show that our GANs-BMC effectively stabilizes GANs' training under StyleGANv2-ada frameworks with a faster rate of convergence, a smaller range of oscillation, and better performance in terms of FID score.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Towards principled methods for training generative adversarial networks. International Conference on Learning Representations (ICLR), 2017.
  2. Stabilization of linear systems by noise. SIAM Journal on Control and Optimization, 21(3):451–461, 1983.
  3. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
  4. Generative multi-adversarial networks. International Conference on Learning Representations (ICLR), 2017.
  5. Do gans always have nash equilibria? In International conference on machine learning. PMLR, 2020.
  6. Many paths to equilibrium: Gans do not need to decrease a divergence at every step. International Conference on Learning Representations (ICLR), 2018.
  7. Stability and lyapunov stability of dynamical systems: A differential approach and a numerical method. Physica D: Nonlinear Phenomena, 27(3):311–337, 1987.
  8. Generative adversarial nets. Advances in Neural Information Processing Systems, pp. 2672–2680, 2014.
  9. Improved training of wasserstein gans. arXiv preprint arXiv:1704.00028, 2017.
  10. Noise-induced synchronization in realistic models. Physical Review E, 67(2):027201, 2003.
  11. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv:1706.08500, 2017a.
  12. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017b.
  13. Itô, K. On stochastic differential equations. Number 4. American Mathematical Soc., 1951.
  14. On computation and generalization of generative adversarial networks under spectrum control. In International Conference on Learning Representations, 2019.
  15. Progressive growing of gans for improved quality, stability, and variation. International Conference on Learning Representations (ICLR), 2018.
  16. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4401–4410, 2019.
  17. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104–12114, 2020a.
  18. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  8110–8119, 2020b.
  19. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852–863, 2021.
  20. Stochastic differential equations. In Numerical solution of stochastic differential equations, pp.  103–160. Springer, 1992.
  21. On convergence and stability of gans. arXiv preprint arXiv:1705.07215v5, 70, 2017.
  22. Learning multiple layers of features from tiny images. 2009.
  23. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
  24. Using white noise to enhance synchronization of coupled chaotic systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 16(1):013134, 2006.
  25. Pacgan: The power of two samples in generative adversarial networks. arXiv preprint arXiv:1712.04086, 2017.
  26. Mao, X. A note on global solution to stochastic differential equation based on a semimartingale with spatial parameters. Journal of Theoretical Probability, 4(1):161–167, 1991.
  27. Mao, X. Stochastic stabilization and destabilization. Systems & control letters, 23(4):279–290, 1994.
  28. Mao, X. Stochastic differential equations and applications. Elsevier, 2007.
  29. Stochastic differential equations with Markovian switching. Imperial college press, 2006.
  30. Environmental brownian noise suppresses explosions in population dynamic. Stochastic Processes and their Applications, 97, 2002.
  31. Disentangling factors of variation in deep representation using adversarial training. Advances in neural information processing systems, 29, 2016.
  32. Which training methods for gans do actually converge? In International conference on machine learning, pp. 3481–3490. PMLR, 2018.
  33. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
  34. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
  35. Fully coupled forward-backward stochastic differential equations and applications to optimal control. SIAM Journal on Control and Optimization, 37(3):825–843, 1999.
  36. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015a.
  37. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015b.
  38. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
  39. Sastry, S. Lyapunov stability theory. In Nonlinear systems, pp.  182–234. Springer, 1999.
  40. Scheutzow, M. Stabilization and destabilization by noise in the plane. Stochastic Analysis and Applications, 11(1):97–113, 1993.
  41. Lyapunov stability theory of nonsmooth systems. IEEE Transactions on automatic control, 39(9):1910–1914, 1994.
  42. Amortised map inference for image super-resolution. arXiv preprint arXiv:1610.04490, 2016.
  43. Generating and enhancing lag synchronization of chaotic systems by white noise. Chaos: An Interdisciplinary Journal of Nonlinear Science, 21(3):033114, 2011.
  44. Analytical and numerical studies of noise-induced synchronization of chaotic systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 11(3):665–673, 2001.
  45. Veretennikov, A. J. On strong solutions and explicit formulas for solutions of stochastic integral equations. Mathematics of the USSR-Sbornik, 39(3):387, 1981.
  46. Improving gan equilibrium by raising spatial awareness. arXiv preprint arXiv:2112.00718, 2021.
  47. Suppression and stabilisation of noise. International Journal of Control, 82(11):2150–2157, 2009.
  48. Understanding and stabilizing GANs’ training dynamics with control theory. In International conference on machine learning. PMLR, 2020.
  49. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  50. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126, 2016.
  51. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp.  2223–2232, 2017.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now