2000 character limit reached
A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Data
Published 24 Feb 2024 in cs.LG, math.ST, stat.ML, and stat.TH | (2402.15710v1)
Abstract: Variational Autoencoders (VAEs) have gained significant popularity among researchers as a powerful tool for understanding unknown distributions based on limited samples. This popularity stems partly from their impressive performance and partly from their ability to provide meaningful feature representations in the latent space. Wasserstein Autoencoders (WAEs), a variant of VAEs, aim to not only improve model efficiency but also interpretability. However, there has been limited focus on analyzing their statistical guarantees. The matter is further complicated by the fact that the data distributions to which WAEs are applied - such as natural images - are often presumed to possess an underlying low-dimensional structure within a high-dimensional feature space, which current theory does not adequately account for, rendering known bounds inefficient. To bridge the gap between the theory and practice of WAEs, in this paper, we show that WAEs can learn the data distributions when the network architectures are properly chosen. We show that the convergence rates of the expected excess risk in the number of samples for WAEs are independent of the high feature dimension, instead relying only on the intrinsic dimension of the data distribution.
- Neural network learning: Theoretical foundations. Cambridge University Press.
- Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214–223. PMLR.
- Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285–2301.
- Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. Journal of Machine Learning Research, 3(Nov):463–482.
- Concentration inequalities: A nonasymptotic theory of independence. Oxford University Press.
- Statistical regeneration guarantees of the wasserstein autoencoder with latent space consistency. Advances in Neural Information Processing Systems, 34:17098–17110.
- On the statistical properties of generative adversarial models for low intrinsic data dimension. arXiv preprint arXiv:2401.15801.
- Distribution approximation and statistical estimation guarantees of generative adversarial networks. arXiv preprint arXiv:2002.03938.
- On deep generative models for approximation and estimation of distributions on manifolds. Advances in Neural Information Processing Systems, 35:10615–10628.
- Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255.
- Adversarial feature learning. In International Conference on Learning Representations.
- Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268–276.
- Generative Adversarial Nets. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
- Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462–1471. PMLR.
- A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773.
- Improved Training of Wasserstein GANs. Advances in Neural Information Processing Systems, 30.
- Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30.
- beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
- An error analysis of generative adversarial networks for learning distributions. Journal of Machine Learning Research, 23(116):1–43.
- A primal-dual link between gans and autoencoders. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- Karr, A. F. (1993). Probability. Springer Texts in Statistics. Springer New York, NY, 1 edition.
- Uniform convergence rate of the kernel density estimator adaptive to intrinsic volume dimension. In International Conference on Machine Learning, pages 3398–3407. PMLR.
- Adam: A method for stochastic optimization. International Conference on Learning Representations.
- Auto-encoding variational bayes. In International Conference on Learning Representations.
- Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias. In International Conference on Learning Representations.
- ϵitalic-ϵ\epsilonitalic_ϵ-entropy and ϵitalic-ϵ\epsilonitalic_ϵ-capacity of sets in function spaces. Translations of the American Mathematical Society, 17:277–364.
- Liang, T. (2021). How well generative adversarial networks learn distributions. The Journal of Machine Learning Research, 22(1):10366–10406.
- Deep nonparametric estimation of intrinsic data structures by chart autoencoders: Generalization error and robustness. arXiv preprint arXiv:2303.09863.
- Non-asymptotic error bounds for bidirectional gans. Advances in Neural Information Processing Systems, 34:12328–12339.
- Adversarial autoencoders. In International Conference on Learning Representations.
- Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2391–2400. JMLR. org.
- Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research, 21(174):1–38.
- The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations.
- Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR.
- Variational autoencoders pursue pca directions (by accident). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12406–12415.
- Hanson-Wright inequality and sub-gaussian concentration. Electronic Communications in Probability, 18(none):1 – 9.
- Statistical guarantees for generative models without domination. In Algorithmic Learning Theory, pages 1051–1071. PMLR.
- Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28.
- Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4784–4788. IEEE.
- On empirical bayes variational autoencoder: An excess risk bound. In Conference on Learning Theory, pages 4068–4125. PMLR.
- Wasserstein auto-encoders. International Conference on Learning Representations.
- Neural discrete representation learning. Advances in Neural Information Processing Systems, 30.
- Vershynin, R. (2018). High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press.
- Wainwright, M. J. (2019). High-dimensional statistics: A Non-asymptotic Viewpoint, volume 48. Cambridge University Press.
- Sharp asymptotic and finite-sample rates of convergence of empirical measures in Wasserstein distance. Bernoulli, 25(4A):2620 – 2648.
- Improved variational autoencoders for text modeling using dilated convolutions. In International Conference on Machine Learning, pages 3881–3890. PMLR.
- Yarotsky, D. (2017). Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114.
- Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
