Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Compressing Latent Space via Least Volume (2404.17773v1)

Published 27 Apr 2024 in cs.LG and cs.CV

Abstract: This paper introduces Least Volume-a simple yet effective regularization inspired by geometric intuition-that can reduce the necessary number of latent dimensions needed by an autoencoder without requiring any prior knowledge of the intrinsic dimensionality of the dataset. We show that the Lipschitz continuity of the decoder is the key to making it work, provide a proof that PCA is just a linear special case of it, and reveal that it has a similar PCA-like importance ordering effect when applied to nonlinear models. We demonstrate the intuition behind the regularization on some pedagogical toy problems, and its effectiveness on several benchmark problems, including MNIST, CIFAR-10 and CelebA.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  2. Pattern recognition and machine learning, volume 4. Springer, 2006.
  3. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4-5):291–294, 1988.
  4. Glen E Bredon. Topology and geometry, volume 139. Springer Science & Business Media, 2013.
  5. Learning flat latent manifolds with vaes. arXiv preprint arXiv:2002.04881, 2020.
  6. Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  7. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218, 1936.
  8. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983–1049, 2016.
  9. From variational to deterministic autoencoders. arXiv preprint arXiv:1903.12436, 2019.
  10. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110:393–416, 2021.
  11. Isometric autoencoders. arXiv preprint arXiv:2006.09289, 2020.
  12. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  13. Implicit rank-minimizing autoencoder. Advances in Neural Information Processing Systems, 33:14736–14746, 2020.
  14. Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233–243, 1991.
  15. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 55(5), 2014.
  16. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  17. On optimization methods for deep learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp.  265–272, 2011.
  18. Efficient sparse coding algorithms. Advances in neural information processing systems, 19, 2006.
  19. John Lee. Introduction to topological manifolds, volume 202. Springer Science & Business Media, 2010.
  20. John M Lee. Smooth manifolds. Springer, 2012.
  21. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp.  3730–3738, 2015.
  22. K-sparse autoencoders. arXiv preprint arXiv:1312.5663, 2013.
  23. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
  24. Topological autoencoders. In International conference on machine learning, pp.  7045–7054. PMLR, 2020.
  25. Geometric autoencoders-what you see is what you decode. 2023.
  26. Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1–19, 2011.
  27. Andrew Y Ng. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning, pp.  78, 2004.
  28. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996.
  29. Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559–572, 1901.
  30. Pca-ae: Principal component analysis autoencoder for organising the latent space of generative networks. Journal of Mathematical Imaging and Vision, 64(5):569–585, 2022.
  31. Elad Plaut. From principal subspaces to principal components with linear autoencoders. arXiv preprint arXiv:1804.10253, 2018.
  32. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp.  8821–8831. PMLR, 2021.
  33. Sparse feature learning for deep belief networks. Advances in neural information processing systems, 20, 2007.
  34. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pp.  1746–1754. PMLR, 2014.
  35. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10684–10695, 2022.
  36. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  37. Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996.
  38. Learning topology-preserving data representations. arXiv preprint arXiv:2302.00136, 2023.
  39. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
  40. Regularized autoencoders for isometric representation learning. In International Conference on Learning Representations, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets