Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
GPT-4o
Gemini 2.5 Pro Pro
o3 Pro
GPT-4.1 Pro
DeepSeek R1 via Azure Pro
2000 character limit reached

On the expressivity of bi-Lipschitz normalizing flows (2107.07232v3)

Published 15 Jul 2021 in cs.LG and stat.ML

Abstract: An invertible function is bi-Lipschitz if both the function and its inverse have bounded Lipschitz constants. Nowadays, most Normalizing Flows are bi-Lipschitz by design or by training to limit numerical errors (among other things). In this paper, we discuss the expressivity of bi-Lipschitz Normalizing Flows and identify several target distributions that are difficult to approximate using such models. Then, we characterize the expressivity of bi-Lipschitz Normalizing Flows by giving several lower bounds on the Total Variation distance between these particularly unfavorable distributions and their best possible approximation. Finally, we discuss potential remedies which include using more complex latent distributions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
  2. On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, November 2020. URL http://arxiv.org/abs/2006.08391. arXiv: 2006.08391.
  3. Wasserstein GAN. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia,, December 2017. URL http://arxiv.org/abs/1701.07875. arXiv: 1701.07875.
  4. Keith Ball. The reverse isoperimetric problem for Gaussian measure. Discrete & Computational Geometry volume, 10(4):411–420, December 1993. ISSN 1432-0444. URL https://doi.org/10.1007/BF02573986.
  5. Spectrally-normalized margin bounds for neural networks. In 30th Conference on Neural Information Processing Systems (NeurIPS 2017), December 2017. arXiv: 1706.08498.
  6. Invertible Residual Networks. In Proceedings of the 36 th International Conference on Machine Learning, Long Beach, California, PMLR 97, 2019, May 2019. arXiv: 1811.00995.
  7. Understanding and Mitigating Exploding Inverses in Invertible Neural Networks. In International Conference on Artificial Intelligence and Statistics, pages 1792–1800. PMLR, March 2021. URL http://proceedings.mlr.press/v130/behrmann21a.html.
  8. The Many Faces of 1-Lipschitz Neural Networks. arXiv:2104.05097 [cs, stat], May 2021. URL http://arxiv.org/abs/2104.05097. arXiv: 2104.05097.
  9. Residual Flows for Invertible Generative Modeling. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada., July 2020. arXiv: 1906.02735.
  10. Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators. SIAM Jour- nal on Mathematics of Data Science, 2020., June 2020. URL http://arxiv.org/abs/1903.01014. arXiv: 1903.01014.
  11. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. In Proceedings of the 37th International Conference on Machine Learning, April 2021. URL http://arxiv.org/abs/1909.13833. arXiv: 1909.13833.
  12. A RAD approach to deep mixture models. In ICLR 2019 Workshop DeepGenStruct, August 2020. URL http://arxiv.org/abs/1903.07714. arXiv: 1903.07714.
  13. Generalizable Adversarial Training via Spectral Normalization. In ICLR 2019, September 2018. URL https://openreview.net/forum?id=Hyx4knR9Ym.
  14. Herbert Federer. Geometric measure theory. Berlin, heidelberg, new york, springer edition, 1969.
  15. Generative Adversarial Networks. In 27th Conference on Neural Information Processing Systems (NeurIPS 2014), June 2014. URL http://arxiv.org/abs/1406.2661. arXiv: 1406.2661.
  16. Semi-Supervised Learning with Normalizing Flows. arXiv:1912.13025 [cs, stat], December 2019. URL http://arxiv.org/abs/1912.13025. arXiv: 1912.13025.
  17. Disconnected Manifold Learning for Generative Adversarial Networks. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada, January 2019. URL http://arxiv.org/abs/1806.00880.
  18. Glow: Generative Flow with Invertible 1x1 Convolutions. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada., volume 31, 2018.
  19. Benoît Kloeckner. Empirical measures: regularity is a counter-curse to dimensionality. arXiv:1802.04038 [math], February 2018. URL http://arxiv.org/abs/1802.04038. arXiv: 1802.04038.
  20. Representational aspects of depth and conditioning in normalizing flows. In Proceedings of the 38th International Conference on Machine Learning, pages 5628–5636. PMLR, July 2021. URL https://proceedings.mlr.press/v139/koehler21a.html. ISSN: 2640-3498.
  21. The Expressive Power of a Class of Normalizing Flow Models. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 3599–3609. PMLR, June 2020. URL https://proceedings.mlr.press/v108/kong20a.html. ISSN: 2640-3498.
  22. Universal Approximation of Residual Flows in Maximum Mean Discrepancy. arXiv, June 2021. URL http://arxiv.org/abs/2103.05793. Number: arXiv:2103.05793 arXiv:2103.05793 [cs, stat].
  23. Improved Precision and Recall Metric for Assessing Generative Models. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada., October 2019. arXiv: 1904.06991.
  24. Deep Learning Face Attributes in the Wild. Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  25. Spectral Normalization for Generative Adversarial Networks. In ICLR 2019, February 2018. URL http://arxiv.org/abs/1802.05957. arXiv: 1802.05957.
  26. Iosif Pinelis. Exact lower and upper bounds on the incomplete gamma function. Mathematical Inequalities & Applications, pages 1261–1278, 2020. 10.7153/mia-2020-23-95.
  27. Guilherme G. P. Freitas Pires and Mário A. T. Figueiredo. Variational Mixture of Normalizing Flows. arXiv:2009.00585 [cs, stat], September 2020. URL http://arxiv.org/abs/2009.00585. arXiv: 2009.00585.
  28. Assessing Generative Models via Precision and Recall. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada, October 2018. URL http://arxiv.org/abs/1806.00035. arXiv: 1806.00035.
  29. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada., October 2019. URL http://arxiv.org/abs/1805.10965. arXiv: 1805.10965.
  30. Intriguing properties of neural networks. In International Conference on Learning Represen- tations, 2014., February 2014. URL http://arxiv.org/abs/1312.6199. arXiv: 1312.6199.
  31. Learning disconnected manifolds: a no GANs land. In Proceedings of the 37 th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020, December 2020. arXiv: 2006.04596.
  32. Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators. In Advances in Neural Information Processing Systems, volume 33, pages 3362–3373. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/2290a7385ed77cc5592dc2153229f082-Abstract.html.
  33. MNIST handwritten digit database. ATT Labs, 2, 2010. URL http://yann.lecun.com/exdb/mnist.
  34. Approximation Capabilities of Neural ODEs and Invertible Residual Networks. arXiv, February 2020. URL http://arxiv.org/abs/1907.12998. Number: arXiv:1907.12998 arXiv:1907.12998 [cs, stat].
Citations (14)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets