Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

Abstract: We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as $\ell<sup>1$-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories that are largely specialized to homogeneous activations such as the ReLU, our framework provides a meaningful notion of functional complexity for deep networks with a broad range of homogeneous and non-homogeneous activation functions commonly used in practice. This simple construction unites several seemingly disparate ideas from the literature, including norm-based complexity bounds and variational characterizations of depth, and facilitates novel analyses of what kinds of functions deep norm-constrained networks can represent. To this end, we prove a novel representer theorem for our spaces and establish novel function-space complexity bounds showing that the associated function classes remain qualitatively small at arbitrary depth. In the univariate ReLU case, we prove a "depth saturation" result: depth in this setting yields only a small constant rescaling of the function class, with no added functional diversity. As a consequence, we show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This finding reveals that some commonly cited expressivity benefits of depth disappear once network complexity is controlled by an appropriate function space norm, rather than parameter count or other representational costs that permit compounded rescaling across layers. Overall, our results illustrate how a function space perspective yields new structural insights into the relationship between depth and complexity.

• The paper introduces a unifying function-space theory that generalizes variation norms to analyze depth and complexity in deep neural networks.
• It rigorously compares the new function-space norm to standard parametric costs while providing explicit bounds on Rademacher complexity and metric entropy.
• The analysis reveals that under norm control, increased depth does not inherently boost function richness, ensuring controlled expressivity for various activations.

Deep Neural Variation Spaces: A Unified Function Space Perspective on Depth and Complexity

Overview of the Function-Space Framework

This work introduces a unifying function-space theory for deep, fully connected neural networks based on recursively constructed variation (atomic/gauge) norms. Rather than focusing on parametric representations or architecture size (width, parameter count), the theory centers on the functional objects implemented by networks and defines complexity using norms over recursively applied, normalized activation feature spaces. The construction generalizes the path norm for ReLU networks to a broad class of activation functions, both homogeneous (e.g., ReLU, Leaky ReLU) and non-homogeneous (e.g., GELU, SiLU, Mish), via scale-normalized nonlinear activation families.

The depth-$L$ function classes $_L$ are defined recursively, with each layer corresponding to absolutely convex combinations of scale-normalized features applied to the previous layer’s function space. For homogeneous activations, this generalization aligns with previously-studied function space norms (e.g., path norm, Barron spaces), but the framework extends naturally to non-homogeneous cases through a normalization that effectively decouples feature shape from scale.

Figure 1: Rescaled and normalized GELU activations $\sigma(st)$ (left) versus $\sigma(st)/s$ (right), illustrating scale-dependent feature variation and Lipschitz control.

Function-Space Norms and Activation-Dependence

A distinct advantage of the proposed framework is its ability to make explicit the separation between compounded linear rescaling and true functional enrichment due to repeated nonlinear transformation. By using normalized activations $\sigma_s(t) = \sigma(st)/s$ (or $s^m$ for degree-$m$ homogeneous activations), the proposed spaces maintain control over the Lipschitz constants of basis elements regardless of scale. This normalization is critical for non-homogeneous activations, where naive input rescaling alters feature shape and local smoothness—a key issue for generalization analysis and for comparisons across network architectures.

The function space $\mathcal{V}_L$ at depth $L$ is constructed as the closed absolutely convex hull of all functions $\sigma_s \circ f$ for $_L$0 in $_L$1 and $_L$2, with the initial dictionary given by affine functions over compact parameter sets. This provides a unified framework accommodating all fully connected networks with controlled parameter magnitudes at all layers.

Relationship with Parametric Costs and Banach Structure

The theory systematically compares the new function-space norm with several prevalent representation costs (e.g., sum-of-squares, additive group/layerwise penalties). A crucial structural property is that $_L$3 is a Banach norm on the function, not merely on parameters: it properly reflects exponential effects of layerwise rescaling via network depth, which additive or quadratic costs (SOSW, group norms) cannot.

Explicitly, for ReLU, the function-space norm matches the path norm in the sense of Neyshabur et al. (2015) and Barron & Klusowski (2019). For general activations, this provides tight upper and lower bounds relating the norm to standard parametric costs for networks with hidden-layer $_L$4 constraints and bounded width—so the framework strictly generalizes prior function space models.

Representer Theorem and Integral Representations

A central technical contribution is a comprehensive representer theorem for the constructed function spaces. Solutions to norm-penalized empirical risk minimization problems (or interpolations) over $_L$5 correspond to finite-width (width $_L$6 for $_L$7 data) deep neural nets whose constituent layer representations are atomic with respect to the gauge norm. The proof leverages the convex-analytic structure and integral representation (Radon) theorems, showing that every minimizer admits a measure-theoretic representation over the atomic feature set, and ultimately a finite decomposition. For ReLU, this bridges exactly to classical neural tree (Barron/neural tree) spaces.

Complexity Bounds: Metric Entropy and Rademacher Complexity

An extensive analysis yields tight, explicit upper bounds on both Rademacher complexity and metric entropy/n-widths of norm balls in $_L$8, for arbitrary depth and a range of $_L$9 norms over data distributions. Notably, the worst-case empirical Rademacher complexity and covering numbers grow at most polynomially in depth for a wide range of relevant activations (often linearly or sublinearly due to global Lipschitz regularity). In all cases, these complexity measures remain "mild" at large depth under norm control, sharply contrasting with the combinatorial parameter-count-based capacity of unrestricted architectures.

Figure 2: Functions $\sigma(st)$0 (see \eqref{eq:f_alpha}), illustrating dependence of higher layer positive-part operations on depth.

Depth Saturation and Expressivity in Uni-/Multi-variate Regimes

A stark, technically robust univariate "depth saturation" phenomenon is established: for ReLU on $\sigma(st)$1, the norm balls $\sigma(st)$2 essentially coincide with the two-layer space ($\sigma(st)$3) up to constant rescaling, independent of $\sigma(st)$4. Thus, depth does not increase the range or complexity of functions attainable under norm control. This result is exact, and the constant is provably optimal.

Figure 3: Iterated compositions $\sigma(st)$5 of the Telgarsky sawtooth illustrating norm blowup with depth for oscillatory functions.

In multivariate settings, this univariate saturation leads to a strong restriction: every function in the deep space must exhibit controlled second variation (low-frequency components) when restricted to any line in input space—thus excluding all high-frequency oscillatory functions in any direction, regardless of depth (unless the norm is allowed to blow up). Empirically, depth-based representational advantages observed for highly oscillatory functions (e.g., Telgarsky’s functions, depth separation theorems) disappear when complexity is measured as a function-space norm rather than parameter counting.

Figure 4: High-frequency univariate trigonometric functions $\sigma(st)$6, excluded from deep variation spaces due to excessive $\sigma(st)$7 variation.

Norm-Equivalence for ReLU-like Activations and Depth Dependence

For a large family of activations (Leaky ReLU, ELU, Swish/SiLU, Mish), the depth-$\sigma(st)$8 variation spaces are norm equivalent to their ReLU counterparts, sometimes with only mild (linear or low-exponential) dependence on $\sigma(st)$9 in the equivalence constants. Thus, for all practical purposes, the structuring effect of depth on complexity and expressivity is robust across common neural nonlinearities.

Implications, Theoretical Significance, and Open Problems

The main theoretical implication is that, under proper function-space norm control, depth alone does not yield increased function space richness—most especially for univariate ReLU and norm-equivalent activations. Commonly cited expressivity benefits of deep compositions are not present for norm-bounded infinite-width networks. In practical deep learning, where width is large and norm control is common (via explicit/implicit regularization), these results suggest that increased depth primarily supports expressivity through unpenalized exponential scaling effects, which may not reflect the structure favored by learning dynamics.

From a statistical learning theory perspective, the results provide robust generalization guarantees for deep networks across a wide range of nonlinearities, offer fundamental insight into the structural induction biases conferred by norm-based regularization, and clarify connections and limitations relative to traditional depth separation and universal approximation theorems.

Open problems center around characterizing functions genuinely unique to deeper spaces in higher dimensions, the fine structure of the deep variation balls (for $\sigma(st)/s$0), and the behavior for "borderline" activations where the norm equivalence constants may increase rapidly with depth. Additionally, understanding the practical implications for regularization in overparameterized regimes, and the role of implicit bias induced by SGD-like optimizers within these function spaces, stands as an important direction.

Conclusion

The paper presents a unified, highly general function-space framework for deep neural networks, providing extensive theoretical tools (variation norms, measure-theoretic representer theorems, explicit complexity bounds) and deepening understanding of how depth interacts with function complexity under norm control. The resulting insights challenge folklore about the expressive potential of depth, especially for norm-bounded, infinite-width networks, and lay groundwork for principled regularization and capacity estimation in both theory and practice.