2000 character limit reached
On the Convergence Rate of Gaussianization with Random Rotations
Published 23 Jun 2023 in cs.LG and stat.ML | (2306.13520v1)
Abstract: Gaussianization is a simple generative model that can be trained without backpropagation. It has shown compelling performance on low dimensional data. As the dimension increases, however, it has been observed that the convergence speed slows down. We show analytically that the number of required layers scales linearly with the dimension for Gaussian input. We argue that this is because the model is unable to capture dependencies between dimensions. Empirically, we find the same linear increase in cost for arbitrary input $p(x)$, but observe favorable scaling for some distributions. We explore potential speed-ups and formulate challenges for further research.
- TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
- Methods of Information Geometry. American Mathematical Society, 2007.
- Analyzing Inverse Problems with Invertible Neural Networks. In International Conference on Learning Representations, 2018.
- Projection Pursuit in High Dimensions. PNAS, 115(37):9151–9156, 2018.
- Cardoso, J.-F. Dependence, Correlation and Gaussianity in Independent Component Analysis. Journal of Machine Learning Research, 4:1177–1203, 2003.
- A refinement of the arithmetic mean-geometric mean inequality. Proceedings of the American Mathematical Society, 71(1):36–38, 1978.
- Gaussianization. In Leen, T., Dietterich, T., and Tresp, V. (eds.), Advances in Neural Information Processing Systems, volume 13, 2000.
- Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pp. 2921–2926. IEEE, 2017.
- How sharp is the Jensen inequality? Journal of Inequalities and Applications, 2015(1):69, December 2015.
- Some Theorems on Distribution Functions. Journal of the London Mathematical Society, s1-11(4):290–294, October 1936.
- Sliced iterative normalizing flows. In International Conference on Machine Learning, 2021.
- NICE: Non-linear Independent Components Estimation. In International Conference on Learning Representations, Workshop Track, 2015.
- Whitening Convergence Rate of Coupling-based Normalizing Flows. In Advances in Neural Information Processing Systems, 2022.
- Neural spline flows. In Advances in Neural Information Processing Systems, 2019.
- Falcon, W. and The PyTorch Lightning team. PyTorch Lightning, March 2019.
- On Choosing and Bounding Probability Metrics. International Statistical Review / Revue Internationale de Statistique, 70(3):419–435, 2002.
- Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
- Array programming with NumPy. Nature, 585(7825):357–362, 2020.
- Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 2020.
- Hunter, J. D. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90–95, 2007.
- An introduction to variational autoencoders. Foundations and Trends in Machine Learning, 12(4):307–392, 2019.
- Knothe, H. Contributions to the theory of convex bodies. Michigan Mathematical Journal, 4(1):39–52, 1957.
- Knuth, D. E. The art of computer programming, volume 3. 1997.
- Normalizing Flows: An Introduction and Review of Current Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):3964–3979, 2021.
- Representational aspects of depth and conditioning in normalizing flows. In International Conference on Machine Learning, 2021.
- Lightning trainable, 2023. Software available from https://github.com/larskue/lightning-trainable.
- Iterative Gaussianization: From ICA to Random Rotations. IEEE Transactions on Neural Networks, 22(4):537–549, April 2011.
- Sliced-Wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. In International Conference on Machine Learning, volume 97 of Proceedings of machine learning research, 2019.
- Gaussianization Flows. In International Conference on Artificial Intelligence and Statistics, 2020.
- Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
- Automated colour grading using colour distribution transfer. Computer Vision and Image Understanding, 107(1-2):123–137, July 2007.
- Variational inference with normalizing flows. In International Conference on Machine Learning, 2015.
- Rosenblatt, M. Remarks on a multivariate transformation. The annals of mathematical statistics, 23(3):470–472, 1952.
- Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators. In Advances in Neural Information Processing Systems, 2020a.
- Universal Approximation Property of Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, Workshop Track, 2020b.
- The pandas development team. pandas-dev/pandas: Pandas, February 2020.
- Wainwright, M. High-dimensional statistics: A non-asymptotic viewpoint. Cambridge University Press, 2019.
- Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), 9th Python in Science Conference, 2010.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
