Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Lifting Architectural Constraints of Injective Flows (2306.01843v5)

Published 2 Jun 2023 in cs.LG

Abstract: Normalizing Flows explicitly maximize a full-dimensional likelihood on the training data. However, real data is typically only supported on a lower-dimensional manifold leading the model to expend significant compute on modeling noise. Injective Flows fix this by jointly learning a manifold and the distribution on it. So far, they have been limited by restrictive architectures and/or high computational cost. We lift both constraints by a new efficient estimator for the maximum likelihood loss, compatible with free-form bottleneck architectures. We further show that naively learning both the data manifold and the distribution on it can lead to divergent solutions, and use this insight to motivate a stable maximum likelihood training objective. We perform extensive experiments on toy, tabular and image data, demonstrating the competitive performance of the resulting model.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org.
  2. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808.04730, 2018.
  3. Invertible residual networks. In International Conference on Machine Learning, pp. 573–582. PMLR, 2019.
  4. Pie: Pseudo-invertible encoder. arXiv:2111.00619, 2021.
  5. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  6. Probabilistic auto-encoder. arXiv:2006.05479, 2020.
  7. Flows for simultaneous manifold learning and density estimation. Advances in Neural Information Processing Systems, 33:442–453, 2020.
  8. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  9. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519, 2015.
  10. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 34:30228–30241, 2021.
  11. Pythae: Unifying Generative Autoencoders in Python – A Benchmarking Use Case. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35, pp. 21575–21589. Curran Associates, Inc., 2022. URL https://arxiv.org/abs/2206.08309.
  12. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018a.
  13. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018b.
  14. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
  15. Isometric manifold learning for injective normalizing flows. arXiv preprint arXiv:2203.03934, 2022.
  16. Normalizing flows across dimensions. arXiv:2006.13070, 2020.
  17. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
  18. Whitening convergence rate of coupling-based normalizing flows. In Advances in Neural Information Processing Systems, 2022.
  19. William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019. URL https://github.com/Lightning-AI/lightning.
  20. Batch norm with entropic regularization turns deterministic autoencoders into generative models. In Conference on Uncertainty in Artificial Intelligence, pp. 1079–1088. PMLR, 2020.
  21. From variational to deterministic autoencoders. arXiv:1903.12436, 2019.
  22. A Girard. A fast ‘Monte-Carlo cross-validation’ procedure for large least squares problems with noisy data. Numerische Mathematik, 56:1–23, 1989.
  23. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM Journal on numerical analysis, 10(2):413–432, 1973.
  24. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv:1810.01367, 2018.
  25. Relative gradient optimization of the jacobian term in unsupervised deep learning. Advances in neural information processing systems, 33:16567–16578, 2020.
  26. Array programming with NumPy. Nature, 585(7825):357–362, September 2020. doi: 10.1038/s41586-020-2649-2. URL https://doi.org/10.1038/s41586-020-2649-2.
  27. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
  28. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  29. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
  30. Denoising normalizing flow. Advances in Neural Information Processing Systems, 34:9099–9111, 2021.
  31. J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90–95, 2007. doi: 10.1109/MCSE.2007.55.
  32. Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3):1059–1076, 1989.
  33. Self normalizing flows. In International Conference on Machine Learning, pp. 5378–5387. PMLR, 2021.
  34. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649–2658. PMLR, 2018.
  35. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
  36. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
  37. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(11):3964–3979, 2020.
  38. Trumpets: Injective flows for inference and inverse problems. In Uncertainty in Artificial Intelligence, pp.  1269–1278. PMLR, 2021.
  39. Geometric integration theory. Springer Science & Business Media, 2008.
  40. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
  41. Regularized autoencoders via relaxed injective probability flow. In International Conference on Artificial Intelligence and Statistics, pp.  4292–4301. PMLR, 2020.
  42. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  43. Deep learning face attributes in the wild. In ICCV, pp.  3730–3738. IEEE Computer Society, 2015. ISBN 978-1-4673-8391-2. URL http://dblp.uni-trier.de/db/conf/iccv/iccv2015.html#LiuLWT15.
  44. Denoising deep generative models. In Proceedings on, pp.  41–50. PMLR, 2023.
  45. Adversarial autoencoders. arXiv:1511.05644, 2015.
  46. Francesco Mezzadri. How to generate random matrices from the classical compact groups. arXiv preprint math-ph/0609050, 2006.
  47. The pandas development team. pandas-dev/pandas: Pandas, February 2020. URL https://doi.org/10.5281/zenodo.3509134.
  48. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
  49. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  50. Bayesflow: Learning complex stochastic models with invertible neural networks. IEEE transactions on neural networks and learning systems, 33(4):1452–1466, 2020.
  51. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530–1538. PMLR, 2015.
  52. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10684–10695, 2022.
  53. Tractable density estimation on learned manifolds with conformal embedding flows. Advances in Neural Information Processing Systems, 34:26635–26648, 2021.
  54. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
  55. Deterministic training of generative autoencoders using invertible layers. In The Eleventh International Conference on Learning Representations, 2023.
  56. Learning to generate images with perceptual similarity metrics. In 2017 IEEE International Conference on Image Processing (ICIP), pp.  4277–4281. IEEE, 2017.
  57. Invertible autoencoder for domain adaptation. Computation, 7(2):20, 2019.
  58. Wasserstein auto-encoders. arXiv:1711.01558, 2017.
  59. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
  60. Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp.  56 – 61, 2010. doi: 10.25080/Majora-92bf1922-00a.
  61. Spread flows for manifold modelling. In International Conference on Artificial Intelligence and Statistics, pp.  11435–11456. PMLR, 2023.
  62. Perceptual generative autoencoders. In International Conference on Machine Learning, pp. 11298–11306, 2020.
  63. Infovae: Information maximizing variational autoencoders. arXiv:1706.02262, 2017.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now