On the Depth of Monotone ReLU Neural Networks and ICNNs (2505.06169v1)

Abstract: We study two models of ReLU neural networks: monotone networks (ReLU$^+$) and input convex neural networks (ICNN). Our focus is on expressivity, mostly in terms of depth, and we prove the following lower bounds. For the maximum function MAX$_n$ computing the maximum of $n$ real numbers, we show that ReLU$^+$ networks cannot compute MAX$_n$, or even approximate it. We prove a sharp $n$ lower bound on the ICNN depth complexity of MAX$_n$. We also prove depth separations between ReLU networks and ICNNs; for every $k$, there is a depth-2 ReLU network of size $O(k^2)$ that cannot be simulated by a depth-$k$ ICNN. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.

On the Depth of Monotone ReLU Neural Networks and ICNNs

The paper by Bakaev et al. thoroughly investigates monotone ReLU networks and input convex neural networks (ICNNs), focusing primarily on the expressivity and computational complexity aspects associated with the depth of these networks. In the context of neural networks, while the rectified linear unit (ReLU) activation function has been extensively studied, understanding the depth required for monotone and convex structures introduces a more refined analysis of network capabilities.

Expressivity Issues and Depth Lower Bounds

The authors analyze two specific models: monotone networks ($\relu^+$) and ICNNs. In each case, they explore the limitations and requirements in achieving expressivity equivalent to computing the maximum ($\MAX_n$) function, defined as the maximum of $n$ real values. $\MAX_n$ is pivotal since it is representative of the expressive capabilities of piecewise linear functions in general $\CPWL_n$ classes.

The findings within monotone ReLU networks ($\relu^+$) indicate that these networks cannot compute $\MAX_n$ or even approximate it. Monotone networks inherently struggle with functions exhibiting non-monotonic behavior because of their restriction to monotonic operations. For $\ICNN$, while it is capable of representing convex functions, including $\MAX_n$, they prove that it requires a depth linear in the input size to precisely compute these functions.

Technical Contributions

Several strong claims elucidate the inherent expressivity limitations of these network models. The paper establishes:

• Monotone Networks: A lower-bound proof demonstrates that $\MAX_n$ is non-representable by monotone networks even if the domain is limited to a bounded linear space such as $[0,1]^n$. This stems from a demonstration that functions computed by monotone networks have isotonic gradients, an essential limitation when handling piecewise linear transformations as required by $\MAX_n$.
• Depth Separation: Depth separations are drawn between monotone networks and general ReLU networks to highlight the efficiency of depth in expressive power. The authors introduce functions ($m_n$) requiring depth $n$ in monotone networks compared to logarithmic depth in general ReLU networks.
• ICNN Complexity: The research further extends to ICNNs, implying a depth complexity linear in $n$ for exact computation of functions like $\MAX_n$. Additionally, configurations of ICNNs are shown to represent various polytopes in convex geometry, proving useful for establishing bounds on depth through geometric interpretations.

Geometric Framework and Polyhedral Connections

One significant contribution of the paper is its establishment of a connection between neural network architecture and polyhedral geometry. Through Newton polytopes and polyhedral transformations, the authors translate neural network operations into manipulations of geometric objects—effectively allowing them to leverage known mathematical properties of polytopes to deduce neural network expressivity limitations. This approach extends beyond typical empirical analysis, providing a robust theoretical foundation and technical rigor.

Practical and Theoretical Implications

The implications extend to both practical boundaries of current neural network architecture development and theoretical foundations concerning computational complexity. Practically, understanding these depth limitations guides architectural choices in designing neural networks for tasks requiring rigorous monotonic or convex function representations. Theoretically, it enriches the discourse surrounding complexity requirements for neural networks in distinguishing between polynomial and super-polynomial depth requirements—valuable information for progressing towards optimal network designs used in AI applications.

Future Research Directions

Given these insights, potential research directions include exploring networks representing different constrained classes of functions, such as Lipschitz continuous or differentiable functions, and understanding their relationship with the geometry of non-linear manifolds. Furthermore, a deeper exploration into adaptive structures that intelligently balance depth and width could inspire innovations in scalable network models applicable to various complexity classes.

Overall, this paper by Bakaev et al. elevates the understanding of depth in specialized neural network configurations, encouraging a structured examination of how nuanced architecture choices impact computational and expressivity realms in artificial intelligence.