On Minimal Depth in Neural Networks (2402.15315v3)
Abstract: A characterization of the representability of neural networks is relevant to comprehend their success in artificial intelligence. This study investigate two topics on ReLU neural network expressivity and their connection with a conjecture related to the minimum depth required for representing any continuous piecewise linear (CPWL) function. The topics are the minimal depth representation of the sum and max operations, as well as the exploration of polytope neural networks. For the sum operation, we establish a sufficient condition on the minimal depth of the operands to find the minimal depth of the operation. In contrast, regarding the max operation, a comprehensive set of examples is presented, demonstrating that no sufficient conditions, depending solely on the depth of the operands, would imply a minimal depth for the operation. The study also examine the minimal depth relationship between convex CPWL functions. On polytope neural networks, we investigate basic depth properties from Minkowski sums, convex hulls, number of vertices, faces, affine transformations, and indecomposable polytopes. More significant findings include depth characterization of polygons; identification of polytopes with an increasing number of vertices, exhibiting small depth and others with arbitrary large depth; and most notably, the minimal depth of simplices, which is strictly related to the minimal depth conjecture in ReLU networks.
- On the decision boundaries of neural networks: A tropical geometry perspective. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):5027–5037, 2022.
- Understanding deep neural networks with rectified linear units. In International Conference on Learning Representations, 2018.
- Computing the continuous discretely: Integer-Point Enumeration in Polyhedra. Springer, 2015.
- Arne Brondsted. An Introduction to Convex Polytopes, volume 90. Springer-Verlag, 1983.
- Improved bounds on neural complexity for representing piecewise linear functions. In Advances in Neural Information Processing Systems, 2022.
- George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
- Nonlinear approximation and (deep) relu networks. Constructive Approximation, 55(1):127–172, 2022.
- Neural network approximation. Acta Numerica, 30:327–444, 2021.
- The power of depth for feedforward neural networks. In 29th Annual Conference on Learning Theory, volume 49, pages 907–940. PMLR, 2016.
- Branko Grünbaum. Convex Polytopes. Springer, 2003.
- Lower bounds on the depth of integral relu neural networks via lattice polytopes. In International Conference on Learning Representations, 2023.
- Relu deep neural networks and linear finite elements. Journal of Computational Mathematics, 38(3):502–527, 2020.
- Towards lower bounds on the depth of relu neural networks. Advances in Neural Information Processing Systems, 34:3336–3348, 2021.
- Towards lower bounds on the depth of relu neural networks. SIAM Journal on Discrete Mathematics, 37(2):997–1029, 2023.
- Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251–257, 1991.
- Neural networks with linear threshold activations: structure and algorithms. In International Conference on Integer Programming and Combinatorial Optimization, pages 347–360, 2022.
- Shiyu Liang and R. Srikant. Why deep neural networks for function approximation? In International Conference on Learning Representations, 2017.
- Introduction to tropical geometry, volume 161. American Mathematical Society, 2021.
- Lower bounds over boolean inputs for deep neural networks with relu gates. arXiv preprint arXiv:1711.03073, 2017.
- Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143–195, 1999.
- Matus Telgarsky. Benefits of depth in neural networks. In 29th Annual Conference on Learning Theory, volume 49, pages 1517–1539. PMLR, 2016.
- Generalization of hinging hyperplanes. IEEE Transactions on Information Theory, 51(12):4425–4431, 2005.
- Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114, 2017.
- Thomas Zaslavsky. Facing up to arrangements: Face-count formulas for partitions of space by hyperplanes: Face-count formulas for partitions of space by hyperplanes, volume 154. American Mathematical Soc., 1975.
- Tropical geometry of deep neural networks. In International Conference on Machine Learning, pages 5824–5832. PMLR, 2018.
- Günter M. Ziegler. Lectures on Polytopes. Springer, 1995.