On the Principles of Deep Feedforward ReLU Networks

Abstract: The architecture of deep feedforward neural networks is ubiquitous in deep learning, either as a whole system or as a subnetwork of other architectures, and thus its mechanism is a key ingredient of the black box of neural networks. On the basis of the simplest two-layer ReLU network, this paper systematically studies the mechanism of deep feedforward ReLU networks with multiple hidden layers and successfully explains the training solution obtained by the back-propagation algorithm. The concept of a path, especially in terms of the relationships between paths, plays a central role in uncovering the mystery of the black box. It is shown that a unit of a deep ReLU network can form a piecewise linear manifold to divide the input space, instead of a hyperplane of the two-layer case. How to efficiently use the hidden-layer units to produce both linear functions and partitions of the input space is also a central problem. The principles of a two-layer ReLU network can be generalized to the deeper case to a large extent, such as multiple strict partial orders and continuity restriction. The combination of the basic and simple principles proposed can yield complicated instantiations including the training solutions, and in this sense the black box of deep feedforward ReLU networks is revealed.

• The paper introduces a formal mechanism using paths to partition input space and generate continuous piecewise linear functions.
• It details how recursive activations and multiple-knot control enable efficient function approximation and expressive network behavior.
• The study connects theoretical predictions with backpropagation training, offering practical algorithms for visualizing and controlling network partitions.

Principles and Mechanism of Deep Feedforward ReLU Networks

Introduction

The paper "On the Principles of Deep Feedforward ReLU Networks" (2607.07035) presents a rigorous theoretical foundation for understanding the mechanistic underpinnings of deep feedforward neural networks employing the ReLU activation function. Building directly upon prior analysis of two-layer ReLU networks, the author systematizes the geometric and functional principles for multi-layer architectures, incorporating deductive reasoning analogous to methodologies in theoretical physics. The central focus is to elucidate how training solutions are obtained by backpropagation, how the input space is partitioned, and how piecewise linear functions are instantiated via paths in the network. The study aims to bridge gaps between ad hoc empirical observations and formal, practically relevant theory.

Formalization of Paths and Partitioning Mechanisms

A fundamental conceptual advance in the paper is the abstraction of ‘paths’—sequences of activations through the network—serving as the primary units for encoding input space partitions and function construction. Each region in the partition of the input domain $U \subset \mathbb{R}^n$ corresponds to a unique path, characterized by the subset of ReLU units activated.

Crucially, the analysis demonstrates that, unlike two-layer ReLU networks wherein hidden units induce hyperplane partitions, in deep networks a single hidden unit can create complex piecewise linear manifolds of dimension $n-1$. Recursive interactions between units across layers allow a single unit to modify the geometry of regions it participates in, enabling direction changes and concatenation of linear segments in the partition boundaries. Formal proofs establish that the number of regions generated by deep ReLU networks grows at least exponentially with depth, subject to constraints on unit activations.

The paper establishes continuity and adjacency of paths and regions: for two adjacent paths (differing by a single unit), their corresponding regions are adjacent and separated by a knot (intersecting $n-1$ dimensional manifold), and the function output is continuous across the boundary.

Principles of Function Construction

From the geometric partition, the paper advances to the principles for constructing piecewise linear functions, crucial for understanding approximation properties.

It formally proves that the output of a deep ReLU network is necessarily a continuous piecewise linear function on the regions induced by the partition. The process is recursive: an ‘initial path’ yields the first linear region, and subsequent adjacent paths—each formed by the addition of a single unit—induce additional linear pieces by the recurrence $$s_{j+1}(\boldsymbol{x}) = s_j(\boldsymbol{x}) + \lambda_j\sigma(\boldsymbol{w}_j^T\boldsymbol{x} + b_j)$$ for each new region.

An influence coefficient vector is introduced, which controls the simultaneous realization of knots (partition boundaries) and the corresponding linear function on a region. The rank and dimensionality requirements for these influence vectors are derived, ensuring the feasibility of constructing arbitrary linear pieces, contingent on sufficient network width and nondegeneracy of parameter matrices.

The paper presents the 'continuity-restriction principle': if linear functions are realized on two adjacent regions sharing a boundary, the linear function on their mutual region is automatically constrained by continuity, and the construction propagates through adjacency in the partition graph.

Piecewise Linear Manifold Generation and Knot Control

A core claim substantiated by theorems and corollaries is that hidden units in deep layers can form not just hyperplanes but continuous piecewise linear manifolds, or even closed curves partitioning the input space. Recursive application of unit activations in paths enables flexible geometric boundaries, which can be used to fit complex data distributions and function geometry. Mechanisms for controlling and generating knots are provided, either by adjusting input parameters of a unit or by leveraging output weights of units in shallower layers.

Principles of 'multiple-knot control' are formalized: a single unit may produce multiple knots by carefully tuning associated parameters, offering increased representational efficiency.

Two-Sided Solutions and Solution Space Complexity

The analysis extends to consideration of 'two-sided' solutions, where units are allowed to switch the sign of their associated partition manifolds (negative knots), greatly enlarging the functional solution space. Rigorous constructions for these cases are given, specifying the required adjustments to both input and output parameters and their effects on other units in the network.

Strong numerical results are provided for minimum unit counts required for universal approximation in both univariate and multivariate cases. For one-dimensional input, the minimal number is shown to be $\Phi+\zeta$, where $\Phi$ is depth and $\zeta$ is the number of linear segments; in the multivariate scenario, the count scales as $n\zeta^{1/n} + 1 + n(\Phi-1)$.

Mechanism of Multiple Outputs

The paper generalizes principles to networks with multiple linear outputs. Using influence coefficient matrices, it demonstrates that arbitrary sets of piecewise linear functions can be simultaneously constructed on the same partition, given sufficient network width and proper parameterization. Each output can share hidden-layer units, maximizing efficiency.

Interpretation of Training Solutions and Practical Algorithm

A dedicated section interprets training solutions from backpropagation, demonstrating how the formal path and partition framework explains observed parameter configurations. Theoretical guarantees are reconciled tightly with empirical solutions via algorithms for visualizing knots and partitions, enabling practical interpretability.

Implications and Speculation

This formalization of deep ReLU network operation has substantial theoretical and practical implications:

• Theory: It advances a universal geometric and functional theory for deep feedforward networks, connecting previously fragmented empirical and mathematical analyses. The framework enables systematic derivation and prediction of network behavior, including expressive capacity and partition complexity.
• Practical Devleopments: The principles translate readily to improved interpretability and controllability of trained models, offering pathways for manually constructing solutions for desired function approximation and data fitting. This has implications for safe and economic AI, as interpretable, efficiently designed architectures can mitigate risks and reduce energy expenditure.
• Future Directions: The work suggests that the richness of solution space in deep ReLU networks derives from high-dimensional geometry. Further exploration may yield new representations, more sophisticated partition strategies, and progress in automated design of architectures tailored for use cases in scientific computing, neural modeling, and energy-efficient AI.

Conclusion

The paper (2607.07035) introduces a comprehensive deductive theory for deep feedforward ReLU networks, formalizing the principles of path construction, partition formation, function implementation, and knot control. Explicit relationships between network architecture and geometric/fuctional expressivity are established, with recursive mechanisms enabling controlled instantiation of complex piecewise linear partitions and functions. The theorems provide rigorous bounds on region formation and approximation capabilities, and practical algorithms reconcile theory with observed training solutions. This work lays a scalable geometric foundation with both immediate and far-reaching ramifications for the development, analysis, and application of deep neural networks employing the ReLU activation function.