2000 character limit reached
Local and global topological complexity measures OF ReLU neural network functions (2204.06062v2)
Published 12 Apr 2022 in math.AT, cs.CG, cs.LG, and math.GT
Abstract: We apply a generalized piecewise-linear (PL) version of Morse theory due to Grunert-Kuhnel-Rote to define and study new local and global notions of topological complexity for fully-connected feedforward ReLU neural network functions, F: Rn -> R. Along the way, we show how to construct, for each such F, a canonical polytopal complex K(F) and a deformation retract of the domain onto K(F), yielding a convenient compact model for performing calculations. We also give a construction showing that local complexity can be arbitrarily high.
- Understanding deep neural networks with rectified linear units. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
- Thomas Banchoff. Critical points and curvature for embedded polyhedra. J. Differential Geometry, 1:245–256, 1967.
- On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Trans. Neural Networks Learn. Syst., 25(8):1553–1565, 2014.
- On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks. Preprint: http://arxiv.org/abs/2008.09052, To appear: Siam Journal on Applied Algebra and Geometry, 2022.
- Branko Grünbaum. Convex polytopes, volume 221 of Graduate Texts in Mathematics. Springer-Verlag, New York, second edition, 2003. Prepared and with a preface by Volker Kaibel, Victor Klee and Günter M. Ziegler.
- Romain Grunert. Piecewise Linear Morse Theory. PhD thesis, Freie Universität Berlin, 2016. https://refubium.fu-berlin.de/handle/fub188/12531.
- PL Morse theory in low dimensions. Preprint: https://arxiv.org/pdf/1912.05054.pdf, 2019.
- On characterizing the capacity of neural networks using algebraic topology. CoRR, abs/1802.04443, 2018.
- Deep ReLU networks have surprisingly few activation patterns. CoRR, abs/1906.00904, 2019.
- Allen Hatcher. Algebraic topology. Cambridge University Press, Cambridge, 2002.
- Introduction to piecewise-linear topology. Ergebnisse der Mathematik und ihrer Grenzgebiete, Band 69. Springer-Verlag, New York-Heidelberg, 1972.
- Alexander Schrijver. Theory of linear and integer programming. Wiley-Interscience Series in Discrete Mathematics. John Wiley & Sons, Ltd., Chichester, 1986. A Wiley-Interscience Publication.
- Richard P. Stanley. An introduction to hyperplane arrangements. In Geometric combinatorics, volume 13 of IAS/Park City Math. Ser., pages 389–496. Amer. Math. Soc., Providence, RI, 2007.
- Günter M. Ziegler. Lectures on polytopes, volume 152 of Graduate Texts in Mathematics. Springer-Verlag, New York, 1995.
- Computing persistent homology. Discrete Comput. Geom., 33(2):249–274, 2005.