---
title: Rectified Linear Complexity in ReLU Networks
url: https://www.emergentmind.com/topics/rectified-linear-complexity
type: topic
---

# Rectified Linear Complexity in ReLU Networks

Rectified Linear Complexity denotes the interplay among depth, width, and the number of affine (piecewise linear) regions in functions realized by deep neural networks with rectified linear units (ReLU-DNNs). It quantifies the expressivity of ReLU networks by measuring how their architecture governs the partitioning of input space into regions where the computed function is affine. The notion synthesizes structural and functional complexity of ReLU-DNNs and establishes formal lower bounds relating network architecture to function representation and training complexity [1611.01491].

## 1. Function Class and Complexity Measures

A ReLU-DNN with input dimension $w_0$, output dimension $w_{k+1}$, and $k$ hidden layers of widths $w_1, \dots, w_k$ implements functions:

\[
f(x) = T_{k+1} \circ \sigma \circ T_k \circ \cdots \circ \sigma \circ T_1(x)
\]

where $T_i: \mathbb{R}^{w_{i-1}} \rightarrow \mathbb{R}^{w_i}$ is affine for $i=1\dots k$, $T_{k+1}$ is linear, and $\sigma$ applies coordinate-wise as $\sigma(t) = \max\{0, t\}$.

Key structural and functional measures:

| Term                 | Definition                                                                   | Notation                                          |
|----------------------|------------------------------------------------------------------------------|---------------------------------------------------|
| Depth                | Total number of layers, including output                                     | $k+1$                                             |
| Width                | Max hidden layer width                                                       | $\max\{w_1, \dots, w_k\}$                         |
| Size                 | Sum of hidden units across layers                                            | $\sum_{i=1}^k w_i$                                |
| Affine Pieces        | Maximal connected regions on which $f$ is affine                             | Number of PWL regions                             |

Any ReLU-DNN computes a continuous piecewise linear (PWL) function. Conversely, every PWL function $f : \mathbb{R}^n \rightarrow \mathbb{R}$ can be represented by a ReLU-DNN of depth $\lceil \log_2(n+1)\rceil + 1$. The number of affine regions, i.e., the cardinality of maximal connected input regions mapped affinely, serves as a fundamental complexity metric.

## 2. Global Optimization for One Hidden Layer

Empirical risk minimization over ReLU networks with one hidden layer and convex loss $\ell$ can be globally optimized as:

\[
\min_{A, b, a'} \frac{1}{D} \sum_{j=1}^{D} \ell\left(a' \cdot \sigma(A x_j + b), y_j\right)
\]

A globally optimal algorithm proceeds via:

1. Writing each hidden unit $i$ as $s_i \max\{0, \tilde{a}^i \cdot x + \tilde{b}_i\}$, $s_i \in \{\pm 1\}$.
2. Partitioning data by sign of $\tilde{a}^i \cdot x_j + \tilde{b}_i$ for all $i$.
3. Enumerating all sign and partition choices $(s_i)_{i=1}^w \in \{\pm1\}^w$ and all hyperplane partitions $(P^i_+, P^i_-)$, with possible count $2^w D^{n w}$.
4. For each, solving the induced convex program in $(\tilde{a}^i, \tilde{b}_i)$.

Total runtime:

\[
O \left( 2^w D^{n w} \mathrm{poly}(D, n, w) \right)
\]

This is polynomial in sample size $D$ for fixed $n, w$ but exponential in $n$ and $w$, matching known computational hardness bounds.

## 3. Depth–Size Gap Theorems

The expressivity of ReLU-DNNs grows rapidly with increased depth compared to width or overall size. For integers $k \geq 1$, $w \geq 2$, there exists a 1D function $f$ such that:

- A $(k+1)$-layer ReLU net of width $w$ represents $f$.
- Any representation by a shallower $(k'+1)$-layer net with $k' < k$ incurs a lower bound on required size:

\[
\text{size} \geq \frac{1}{2} k' w^{k/k' - 1}
\]

Furthermore, for every $k \in \mathbb{N}$, there exists a member of a smoothly-parameterized family (by $\bigcup_{M>0}\Delta_M^{k^3-1}$):

- $f$ is realized by a depth $k^2 + 1$ net of size $k^3$.
- Any depth $k + 1$ ReLU net computing $f$ requires at least:

\[
\frac{1}{2} k^{k+1} - 1
\]

The construction uses sawtooth-composed functions—composition amplifies the number of affine segments exponentially in depth.

## 4. Lower Bounds via Zonotope Constructions

A new lower bound for affine region count in ReLU-DNNs is established via the theory of zonotopes. For vectors $v_1,\dots,v_m \in \mathbb{R}^n$, the zonotope $Z(v_1,\dots,v_m)$ is:

\[
Z(v_1,\dots,v_m) = \left\{ \sum_{i=1}^m \lambda_i v_i : -1 \leq \lambda_i \leq 1 \right\}
\]

and its support function:

\[
\gamma_Z(x) = \max_{z \in Z(v_1,\dots,v_m)} \langle x, z \rangle = \sum_{i=1}^m |\langle x, v_i \rangle|
\]

For general $v_i$, $\gamma_Z$ has:

\[
|\mathrm{verts}(Z(v_1,\dots,v_m))| = \sum_{i=0}^{n-1} \binom{m-1}{i}
\]

distinct affine pieces. $\gamma_Z$ can be implemented by a two-layer ReLU net of size $2m$.

Composition with a $k$-fold sawtooth map $H_{t_1,\dots,t_k}$ yields a ReLU net of depth $k+2$, size $2m + wk$, and number of segments:

\[
\left( \sum_{i=0}^{n-1} \binom{m-1}{i} \right) w^k
\]

Asymptotically, choosing $m \gg n$ or $m \approx n$,

\[
\Omega(m^{n-1} w^k)
\]

For depth $\leq k+1$, matching this piece count requires size at least $\Omega((m-1)^{(n-1)/k'} w^{k/k'})$.

## 5. Synthesis and Implications of Rectified Linear Complexity

The composition of depth and width exponentially increases the count of affine regions:

- **Depth** acts as the exponential composition resource; each layer can multiply the region count.
- **Width** (or size) determines the parallel granularity per layer.
- The total number of affine pieces grows as $\text{width}^{\text{depth}}$ in 1D or $(\sum_{i=0}^{n-1} \binom{m-1}{i}) \, \text{width}^{\text{depth}}$ in $\mathbb{R}^n$.

The triplet $(\text{depth}, \text{width}, \text{number of affine regions})$ forms a natural complexity measure—*Rectified Linear Complexity*—which encapsulates the expressive power of ReLU networks. Deeper networks attain exponential region growth with moderate width, while shallow networks require super-polynomial size for function approximation equivalence.

A plausible implication is that for function classes requiring exponentially many affine pieces, depth is indispensable for architectural efficiency. Furthermore, computational hardness in training aligns with the representational barriers: even for one hidden layer, the exponential increase in complexity with input dimensionality indicates that training algorithms are fundamentally limited by both representational and computational regimes [1611.01491].

Source: https://www.emergentmind.com/topics/rectified-linear-complexity