Deep Feedforward ReLU Networks

• Deep Feedforward ReLU Networks are fully connected layers of affine maps interspersed with ReLU activations, which partition the input space into piecewise-linear regions.
• They leverage depth to exponentially increase the number and structure of linear regions, enhancing expressivity and enabling recursive compositions for complex function approximation.
• Their architecture supports theoretical analyses in geometry, optimization, and identifiability, contributing to robust initialization, verification, and training methods in deep learning.

Searching arXiv for the cited papers and closely related work on deep feedforward ReLU networks. arXiv Search Query: ids:(Vallin et al., 2024) OR ids:(Grigsby et al., 2020) OR ids:(Goubault et al., 2021) OR ids:(Aftabi et al., 2024) OR ids:(Lee et al., 2023) OR ids:(Telgarsky, 2015) OR ids:(Bona-Pellissier et al., 2021) OR ids:(Huybrechs, 23 Jun 2025) OR ids:(Awasthi et al., 2021) OR ids:(Schmidt-Hieber, 2017) OR ids:(Huang, 2022) OR ids:(Getu, 2021) OR ids:(Heinecke et al., 2019) OR ids:(Hinz et al., 2020) OR ids:(Opschoor et al., 2023) OR ids:(Bresler et al., 2020) OR ids:(Huang, 2022) OR ids:(Huang, 8 Jul 2026) OR ids:(Zhang et al., 30 Jan 2026) Deep feedforward ReLU networks are fully connected compositions of affine maps and the rectified linear unit $\sigma(z)=\max(0,z)$, and they constitute one of the standard architectural primitives of deep learning. In their most common scalar-output form, they compute

$$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$

or, equivalently, layerwise maps of the form $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$ followed by a final linear readout (Vallin et al., 2024). Their central mathematical feature is that they are continuous and piecewise affine: the input space is partitioned into finitely many regions determined by activation patterns, and on each region the network acts as an affine map (Grigsby et al., 2020). This piecewise-affine structure supports several complementary theories: geometric descriptions of decision boundaries, sharp depth-dependent expressivity results, approximation theorems in analytic and statistical settings, and formal analyses of optimization, verification, and identifiability (Heinecke et al., 2019).

1. Architectural definition and regional geometry

A deep feedforward ReLU network can be described either by nested affine-ReLU compositions or by gating matrices that make the piecewise-affine structure explicit. If $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$, then the hidden representation satisfies

$$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$

and repeated substitution yields a regionwise affine form in which the linear part is a product of weight matrices and diagonal gates (Huang, 8 Jul 2026). An equivalent formulation fixes an activation pattern $s$ and writes the network on the corresponding region as $f(x)=A_s x+c_s$, where $A_s$ is obtained by inserting the diagonal masks between successive weight matrices (Heinecke et al., 2019).

This regional viewpoint is formalized by the canonical polyhedral complex $C(F)$, on whose cells the network is affine-linear (Grigsby et al., 2020). The loci where preactivations vanish generate a bent hyperplane arrangement: in the first layer these sets are hyperplanes, while in later layers they are pullbacks of hyperplanes through preceding piecewise-affine maps, so they bend across activation regions (Grigsby et al., 2020). In a complementary signal-processing description, the affine map on a region admits an atomic decomposition into rank-one terms indexed by active paths, making the output matrix a sum of outer products of first-layer rows and last-layer columns with coefficients determined by intermediate active paths (Heinecke et al., 2019).

This geometry is not limited to shallow networks. In the two-layer case, a hidden unit partitions the input space by a hyperplane. In deeper networks, the preimage of a later-layer threshold is generally a piecewise-linear manifold in input space rather than a single hyperplane, because earlier ReLU layers change the affine chart across regions (Huang, 8 Jul 2026). A plausible implication is that the expressive role of depth is not only to increase the number of linear pieces, but also to allow later units to operate on already-folded coordinates, thereby generating more structured partitions than a single hidden layer can realize.

2. Depth, oscillation, and representation power

The most explicit depth-separation statement is Telgarsky’s construction of the $n$-alternating-point problem. For every positive integer $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$0, there exists a classification task on $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$1 points in $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$2 such that a ReLU network with $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$3 nodes in each of $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$4 layers achieves zero error, whereas shallower networks with width

$$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$5

incur error at least $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$6 (Telgarsky, 2015). The construction is driven by repeated composition of a “mirror map,” which doubles the number of oscillations at each step. This gives an elementary mechanism by which depth converts repeated folding into exponentially many threshold crossings.

A finer depth-dependent theory is given by the representation classes $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$7 defined through Fourier norms. For $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$8 and a $$f(x)=W_L\,\sigma\big(W_{L-1}\,\sigma(\cdots \sigma(W_1 x + b_1)\cdots)+b_{L-1}\big)+b_L,$$9-layer ReLU network with $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$0, the square-loss approximation rate is

$x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$1

and the lower bound matches this exponent (Bresler et al., 2020). The classes satisfy $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$2, and as $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$3 grows they contain progressively less smooth functions (Bresler et al., 2020). In that sense, deeper networks are not merely more expressive in aggregate; they are better matched to representing less smooth functions by exploiting oscillatory structure with few activation functions.

The algorithmic-construction literature makes the same point in constructive rather than existential terms. Exact sorting can be realized by ReLU networks using min-max comparator gadgets, and bitonic sorting yields a network with parallel depth

$x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$4

for $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$5 inputs, width $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$6 per hidden layer, and sparse parameter count $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$7 (Huybrechs, 23 Jun 2025). The same fold/unfold viewpoint gives exponentially accurate approximations for $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$8, exponentials, trigonometric functions, monomials, and multiplication, with depth encoding recursion depth (Huybrechs, 23 Jun 2025). This suggests that depth in feedforward ReLU networks functions as a resource for controlled recursive composition rather than as a mere stacking of affine maps.

3. Decision boundaries, manifolds, and topology

Because a ReLU network is piecewise affine, its zero set and decision regions are polyhedral on each activation region. This observation is sharpened by a constructive hypersurface approximation theorem: if $x^{(\ell)}=\sigma(W_\ell^\top x^{(\ell-1)}+b_\ell)$9 has bounded second derivatives, then there exists a fully connected ReLU network of constant width $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$0 whose zero contour is the graph of a continuous piecewise-linear function $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$1 satisfying

$D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$2

with depth bounded by

$D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$3

The proof interprets each hidden layer as a projection onto a polyhedral cone and, on bounded sets, as a parallel projection onto a hyperplane outside a half-space (Vallin et al., 2024). This provides an explicit geometric mechanism by which depth sculpts a decision boundary while width remains fixed at $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$4.

Topology imposes additional architectural constraints. For a generic, transversal ReLU network with a single hidden layer of width $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$5, any decision region has at most one bounded connected component (Grigsby et al., 2020). More generally, if every hidden layer has width at most $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$6 in $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$7, then each decision region and its boundary are either empty or unbounded (Grigsby et al., 2020). The proof uses the canonical polyhedral complex, a bent hyperplane arrangement, and a partially oriented $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$8-complex that records gradient flow across the $D_\ell(x)=\operatorname{diag}(1_{a_{\ell,j}(x)>0})$9-skeleton (Grigsby et al., 2020).

These geometric and topological statements align with more recent path-based interpretations in which adjacent regions differ by one newly active unit and satisfy a recurrence

$$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$0

with continuity enforced by $$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$1 (Huang, 8 Jul 2026). A plausible implication is that deep decision boundaries are assembled incrementally by adjoining compatible local affine pieces, rather than by globally optimizing an unconstrained polyhedral surface.

4. Approximation theory and statistical guarantees

In nonparametric regression, sparsely connected deep ReLU networks achieve minimax rates up to logarithmic factors under a hierarchical composition model

$$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$2

With effective smoothness indices

$$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$3

the benchmark rate is

$$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$4

and, for suitable depth, widths, and sparsity, the prediction risk satisfies

$$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$5

with $$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$6 yielding $$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$7 (Schmidt-Hieber, 2017). The theory explicitly attributes the benefit of depth to the compositional structure of the target and notes that wavelet estimators are suboptimal under this assumption (Schmidt-Hieber, 2017).

Constructive approximation theory gives complementary bounds for specific operators and function classes. For matrix-vector products $$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$8, one can construct deep ReLU networks with

$$h_\ell(x)=D_\ell(x)\big(W_\ell h_{\ell-1}(x)+b_\ell\big),$$9

and uniform approximation error at most $s$0 on $s$1; analogous bounds hold in $s$2 and for complex-valued matrix-vector products via real-imaginary decomposition (Getu, 2021). The proof reduces products to approximations of squares and composes these approximants, so error decays exponentially in depth for fixed width (Getu, 2021).

For continuous piecewise polynomial targets on arbitrary finite partitions of an interval, Chebyshev-based constructions improve on monomial-based emulations. Chebyshev coefficients are computed from Clenshaw–Curtis values by an inverse FFT, and the resulting ReLU emulators obtain explicit Sobolev-stable bounds with fewer neurons (Opschoor et al., 2023). For analytic functions with point singularities, the paper proves exponential ReLU emulation rate bounds in the number of parameters through graded meshes and hp-approximation (Opschoor et al., 2023). Taken together, these results place deep feedforward ReLU networks within a constructive approximation hierarchy spanning operator emulation, piecewise polynomial synthesis, and minimax-optimal statistical estimation.

5. Training structure, initialization, and interpolation viewpoints

Several recent analyses treat trained ReLU networks as structured objects rather than purely statistical predictors. A path-based account explains a trained solution through adjacent-path recurrences, multiple strict partial orders on knots, and a continuity restriction that determines neighboring linear pieces once two nonparallel boundaries are fixed (Huang, 8 Jul 2026). In this view, backpropagation adjusts path coefficients and local manifolds while preserving continuity across shared boundaries, providing a geometric explanation of trained piecewise-linear solutions (Huang, 8 Jul 2026).

A related interpolation perspective organizes exact fitting through an interpolation matrix whose entries are hidden-unit activations on the training data. In three-layer networks, the interpolation equation is $s$3, and solvability depends on the zero/nonzero block structure induced by activations (Huang, 2022). In deep networks, the last three layers can be separated from an upstream bijective map, leading to a reduced interpolation matrix $s$4 whose sparsity increases with depth through what the paper terms the sparse-matrix principle (Huang, 2022). This principle is used to construct deep solutions for classification and to explain data-disentangling behavior of encoders (Huang, 2022).

Initialization matters acutely in deep and narrow settings. A deterministic orthogonally based scheme constructs

$s$5

where $s$6 comes from the QR decomposition of $s$7 (Lee et al., 2023). The resulting weights are orthogonal or isometric, align a principal direction with the all-ones vector, and preserve mean and angle in a way intended to mitigate dying ReLU in deep narrow feedforward networks (Lee et al., 2023). The paper reports stable training in experiments with up to $s$8 layers and alternating widths $s$9 and $f(x)=A_s x+c_s$0, a regime in which several standard initializations degrade (Lee et al., 2023). For depth-$f(x)=A_s x+c_s$1 networks with general biases under Gaussian inputs, efficient learning is possible by decomposing higher-order Hermite tensors and exploiting identifiability up to signs under nondegeneracy assumptions (Awasthi et al., 2021).

6. Verification, identifiability, and formal analysis

Because deep feedforward ReLU networks are piecewise affine, they admit exact or sound formal encodings. Fixed-parameter ReLU networks can be represented as mixed-integer programs with one binary variable per ReLU unit and standard big-$f(x)=A_s x+c_s$2, extended, disjunctive, or convex-hull formulations (Aftabi et al., 2024). On a narrow MNIST network, extended and disjunctive formulations solved adversarial-generation instances to optimality in seconds, whereas on a wider network all tested formulations hit a $f(x)=A_s x+c_s$3s time limit with large gaps (Aftabi et al., 2024). In parallel, tropical-polyhedral abstractions give exact handling of the ReLU nonlinearity and tight layerwise range analysis without enumerating activation patterns, yielding a sound abstract-interpretation framework for robustness and reachability questions (Goubault et al., 2021).

Identifiability is subtler. For fully connected deep ReLU networks, parameters are generically identifiable only modulo permutation and positive rescaling of hidden units, reflecting the symmetries induced by ReLU homogeneity (Bona-Pellissier et al., 2021). Under Conditions $f(x)=A_s x+c_s$4—full row rank, visibility of gate boundaries on the observed domain, nondegenerate propagation of kinks, and boundary separation—equality of the realized function on a subset of the input space implies equivalence of parameters up to these symmetries (Bona-Pellissier et al., 2021). A more recent logic-based treatment translates ReLU networks into Łukasiewicz logic formulae and proves that every functional equivalence class is generated by a finite set of symmetry-inducing rewrites corresponding to the logic axioms, yielding a complete identification program for networks over $f(x)=A_s x+c_s$5 with integer, rational, or real parameters (Zhang et al., 30 Jan 2026).

Optimization can also be formulated directly on the polyhedral complex induced by activation patterns. Deep ReLU Programming models the network as a continuous piecewise-affine function and extends the Simplex method to move not only along vertices of one affine region but also across adjacent affine regions by single activation flips (Hinz et al., 2020). The resulting DRLSimplex algorithm recovers the Barrodale–Roberts algorithm for LAD regression as a special case and trains the first layer of a neural network under an $f(x)=A_s x+c_s$6 loss with monotone decrease at each step (Hinz et al., 2020). This line of work reinforces the broader theme that deep feedforward ReLU networks are amenable to exact combinatorial and polyhedral analysis precisely because their nonlinearity is localized to sign changes of affine forms.