Noise Dimension of GAN: An Image Compression Perspective
Abstract: Generative adversial network (GAN) is a type of generative model that maps a high-dimensional noise to samples in target distribution. However, the dimension of noise required in GAN is not well understood. Previous approaches view GAN as a mapping from a continuous distribution to another continous distribution. In this paper, we propose to view GAN as a discrete sampler instead. From this perspective, we build a connection between the minimum noise required and the bits to losslessly compress the images. Furthermore, to understand the behaviour of GAN when noise dimension is limited, we propose divergence-entropy trade-off. This trade-off depicts the best divergence we can achieve when noise is limited. And as rate distortion trade-off, it can be numerically solved when source distribution is known. Finally, we verifies our theory with experiments on image generation.
- “Generative adversarial networks,” Communications of the ACM, vol. 63, pp. 139 – 144, 2014.
- “Towards principled methods for training generative adversarial networks,” ArXiv, vol. abs/1701.04862, 2017.
- “A style-based generator architecture for generative adversarial networks,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4396–4405, 2018.
- “Understanding noise injection in gans,” in International Conference on Machine Learning, 2020.
- “Effect of input noise dimension in gans,” in International Conference on Neural Information Processing, 2020.
- “Arithmetic coding for data compression,” Commun. ACM, vol. 30, pp. 520–540, 1987.
- “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” ArXiv, vol. abs/1506.03365, 2015.
- “Large scale gan training for high fidelity natural image synthesis,” ArXiv, vol. abs/1809.11096, 2018.
- “Training generative adversarial networks with limited data,” ArXiv, vol. abs/2006.06676, 2020.
- “Lecture notes on information theory,” Lecture Notes for ECE563 (UIUC) and, vol. 6, no. 2012-2016, pp. 7, 2014.
- “The numerics of gans,” Advances in neural information processing systems, vol. 30, 2017.
- “Size-noise tradeoffs in generative networks,” in Neural Information Processing Systems, 2018.
- “Elements of information theory,” 1991.
- Donald Ervin Knuth and Andrew Chi-Chih Yao, “The complexity of nonuniform random number generation,” 1976.
- Richard Blahut, “Computation of channel capacity and rate-distortion functions,” IEEE transactions on Information Theory, vol. 18, no. 4, pp. 460–473, 1972.
- “CVXPY: A Python-embedded modeling language for convex optimization,” Journal of Machine Learning Research, vol. 17, no. 83, pp. 1–5, 2016.
- Alex Krizhevsky, “Learning multiple layers of features from tiny images,” 2009.
Paper Prompts
Sign up for free to create and run prompts on this paper.
