Complexity of One-Dimensional ReLU DNNs

Abstract: We study the expressivity of one-dimensional (1D) ReLU deep neural networks through the lens of their linear regions. For randomly initialized, fully connected 1D ReLU networks (He scaling with nonzero bias) in the infinite-width limit, we prove that the expected number of linear regions grows as $\sum_{i = 1}^L n_i + \mathop{o}\left(\sum_{i = 1}^L{n_i}\right) + 1$, where $n_\ell$ denotes the number of neurons in the $\ell$-th hidden layer. We also propose a function-adaptive notion of sparsity that compares the expected regions used by the network to the minimal number needed to approximate a target within a fixed tolerance.

• The paper derives a closed-form expression for the expected number of linear regions, showing each neuron contributes one breakpoint in the infinite-width limit.
• The analysis leverages Gaussian process theory to track how breakpoints propagate across layers, confirming that network width predominantly controls expressivity in one-dimensional settings.
• The study introduces region-adaptive sparsity, a metric that compares the network's linear region count to the minimal regions required for a target approximation error, paving the way for principled pruning.

Complexity Analysis of One-Dimensional ReLU DNNs

Introduction

This paper, "Complexity of One-Dimensional ReLU DNNs" (2512.08091), provides a rigorous study of the expressive power of one-dimensional (1D) fully connected deep neural networks (DNNs) with ReLU activations. The focus is on quantifying expressivity via the number of linear regions, which in the 1D case equates to the number of breakpoints (non-differentiable points) the network can realize as a function of input. Beyond this, the paper addresses the inadequacy of classical, parameter-based sparsity metrics by proposing a notion of sparsity grounded in the functional complexity of the target, measured through linear regions. This work is motivated by the need for precise understanding of model complexity and efficient network design, particularly as it applies to pruned or sparse architectures.

Formalization and Setup

The study is conducted in the infinite-width regime, a setting where fully connected DNNs with He-initialized weights ($W_{ij}^{(\ell)} \sim N(0, 2/n_{\ell-1})$) and nonzero bias ($b_j^{(\ell)} \sim N(0, \sigma_b^2)$) converge to Gaussian processes, per the neural network Gaussian process correspondence [lee2017deep]. The networks considered are strictly 1D-in, 1D-out, and all results are characterized with respect to randomly initialized weights and biases in the infinite-width limit.

An essential aspect of the analysis is the concept of "linear regions," which are maximal open intervals on which the network function is affine. For 1D inputs, the count of linear regions equals the number of breakpoints plus one. This correspondence enables a direct statistical analysis of ReLU breakpoint formation via probabilistic properties of pre-activations and their sign changes, leveraging GP theory and asymptotic analysis.

Main Theoretical Contributions

Expected Number of Linear Regions

The central result is a closed-form characterization (in the limit) for the expected number of breakpoints in such a random DNN. The paper proves that for a network of depth $L$ and hidden widths $n_1, \ldots, n_L$, the expected number of linear regions (breakpoints plus one) satisfies:

$$\mathbb{E}\left[R(T)\right] = \sum_{\ell=1}^L n_\ell + o\left(\sum_{\ell=1}^L n_\ell\right) + 1$$

as $\min\{n_1, \ldots, n_L\} \to \infty$, where $R(T)$ is the random variable denoting the number of breakpoints of the DNN (2512.08091). Thus, in expectation, each neuron in each hidden layer contributes precisely one region, in the asymptotic mean. This result analytically confirms and generalizes earlier conjectures and empirical findings regarding the scaling of region complexity with network size [hanin2019complexity, montufar2014number].

The derivation proceeds by analyzing the propagation of breakpoints through layers, using detailed expansions of correlation and covariance functions for pre-activations viewed as GPs. The recursive structure enables quantitative tracking of how often sign changes are introduced and preserved at each layer.

Function-Adaptive Sparsity Notion

Parameter-based sparsity metrics are highly architecture-dependent and often fail to convey information about practical expressivity, especially under reparametrizations or with structurally atypical architectures (e.g., sparse networks, RadiX-Nets [kepner2019radix]). To address this, the paper introduces region-adaptive sparsity, which is defined relative to the minimal number of linear regions required to approximate a target function $f$ to a prescribed uniform error $\varepsilon_0$. Explicitly,

$$\eta_{\mathrm{region}}(\Phi; f, \varepsilon_0) = \frac{L(\Phi)}{L_{\min}(f, \varepsilon_0)},$$

where $L(\Phi)$ is the number of regions the network $\Phi$ uses, and $L_{\min}(f, \varepsilon_0)$ is the smallest number required for any continuous piecewise-linear function to achieve the target accuracy.

A ReLU network is declared $(f, \varepsilon_0, \alpha, c)$-region-adaptively sparse if it achieves an approximation error at most $\alpha \varepsilon_0$ and does not use more than $c$ times the minimal number of required regions. This formulation ties sparsity directly to functional approximation complexity and invariance under architectural transformation, facilitating more meaningful comparison across pruning algorithms, architectures, or training regimes.

Numerical and Analytical Claims

• The paper establishes, without reliance on empirical experimentation, that in the infinite-width random initialization setting, the average number of linear regions is linear in the number of neurons across all layers, and the proportionality coefficient is one.
• It is shown that each neuron in the first layer introduces exactly one breakpoint, and each neuron in subsequent layers, in expectation, also makes a unit contribution.
• The analysis is robust to the choice of depth and widths as long as all widths are taken to infinity.

Implications and Theoretical Impact

The existence of a precise formula for the expected number of 1D regions impacts theoretical studies of network expressivity, statistical generalization, and function complexity. The results restrict possible gains in region complexity from naive width expansion, reinforcing the qualitative observation that width, rather than depth, dominates in the 1D setting. The architecture-independent notion of region-adaptive sparsity mitigates architectural and implementation biases inherent to previous parameter-complexity frameworks, offering a path toward function-centric model selection and principled pruning, particularly relevant for structured sparse regimes like RadiX-Nets and network lottery ticket paradigms [frankle2018lottery].

Further, this work provides foundational machinery for the study of overparameterization in the context of true functional complexity, independent of confounds from parameter redundancy, and enables future explorations into the cost-benefit tradeoffs in model design and compression.

Limitations and Prospects for Future Work

• All results are confined to the strictly 1D-in, 1D-out setting. Extension to higher input/output dimensions will involve handling exponentially more complex combinatorial geometry and the intricacies of multi-dimensional region counting, possibly requiring alternative machinery such as polyhedral theory or Morse-theoretic techniques [serra2018bounding].
• The paper provides only the expected value of the number of regions; the distributional properties (variance, tail bounds, concentration) remain open, and are crucial for applications in reliability and worst-case analysis.
• The region-adaptive sparsity definition, while robust in theory, invites further empirical grounding and application to real-world pruning scenarios and learned networks.

Conclusion

This paper delivers a comprehensive, asymptotically exact result concerning the region complexity (number of 1D linear regions) of wide, random ReLU networks. It establishes that the expected number of breakpoints is the sum of hidden layer widths, a property robust to depth and width scalings under infinite-width initialization. The introduction of a region-based, function-adaptive sparsity definition provides a rigorous alternative to parameter-count-based measures and sets the stage for future research into principled compression and expressive capacity analysis in DNNs. Extensions to higher dimensions, analysis of fluctuation statistics, and practical exploitation of region-adaptive metrics remain as promising directions.