Distributional Principal Autoencoders (2404.13649v1)
Abstract: Dimension reduction techniques usually lose information in the sense that reconstructed data are not identical to the original data. However, we argue that it is possible to have reconstructed data identically distributed as the original data, irrespective of the retained dimension or the specific mapping. This can be achieved by learning a distributional model that matches the conditional distribution of data given its low-dimensional latent variables. Motivated by this, we propose Distributional Principal Autoencoder (DPA) that consists of an encoder that maps high-dimensional data to low-dimensional latent variables and a decoder that maps the latent variables back to the data space. For reducing the dimension, the DPA encoder aims to minimise the unexplained variability of the data with an adaptive choice of the latent dimension. For reconstructing data, the DPA decoder aims to match the conditional distribution of all data that are mapped to a certain latent value, thus ensuring that the reconstructed data retains the original data distribution. Our numerical results on climate data, single-cell data, and image benchmarks demonstrate the practical feasibility and success of the approach in reconstructing the original distribution of the data. DPA embeddings are shown to preserve meaningful structures of data such as the seasonal cycle for precipitations and cell types for gene expression.
- Generalized denoising auto-encoders as generative models. Advances in neural information processing systems, 26.
- Robust principal component analysis? Journal of the ACM (JACM), 58(3):1–37.
- Copernicus Climate Change Service, C. D. S. (2019). Cordex regional climate model data on single levels. copernicus climate change service (c3s) climate data store (cds).
- Adversarial feature learning. In ICLR.
- Adversarially learned inference. In ICLR.
- Overview of the coupled model intercomparison project phase 6 (cmip6) experimental design and organization. Geoscientific Model Development, 9(5):1937–1958.
- Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378.
- Generative adversarial nets. Advances in neural information processing systems, 27.
- Dictionary learning for integrative, multimodal and scalable single-cell analysis. Nature Biotechnology.
- beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
- Reducing the dimensionality of data with neural networks. science, 313(5786):504–507.
- Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851.
- Information-ordered bottlenecks for adaptive semantic compression. arXiv preprint arXiv:2305.11213.
- Jolliffe, I. (2002). Principal Component Analysis. Springer Series in Statistics. Springer.
- Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR.
- Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
- Auto-encoding variational bayes. In International Conference on Learning Representations.
- Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687–15701.
- The geoengineering model intercomparison project phase 6 (geomip6): simulation design and preliminary results. Geoscientific Model Development, 8(10):3379–3392.
- The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/.
- On the conditional distribution of a multivariate normal given a transformation–the linear case. Heliyon, 5(2).
- Adversarial autoencoders. In International Conference on Learning Representations.
- Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861.
- Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1–64.
- Pca-ae: Principal component analysis autoencoder for organising the latent space of generative networks. Journal of Mathematical Imaging and Vision, 64(5):569–585.
- Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746–1754. PMLR.
- Learning representations by back-propagating errors. nature, 323(6088):533–536.
- Engression: Extrapolation for nonlinear regression? arXiv preprint arXiv:2307.00835.
- Bidirectional generative modeling using adversarial gradient estimation. arXiv preprint arXiv:2002.09161.
- Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR.
- Triangular dropout: Variable network width without retraining. arXiv preprint arXiv:2205.01235.
- Székely, G. J. (2003). E-statistics: The energy of statistical samples. Bowling Green State University, Department of Mathematics and Statistics Technical Report, 3(05):1–18.
- The Energy of Data and Distance Correlation. CRC Press.
- Wasserstein auto-encoders. In International Conference on Learning Representations.
- Visualizing data using t-sne. Journal of machine learning research, 9(11).
- Kernel-based data fusion for machine learning. Studies in Computational Intelligence: Springer Berlin Heidelberg.