Learning Sparse Compositional Functions with Norm-Constrained Neural Networks

Abstract: The ability of deep neural networks to learn hierarchical features is widely regarded as a key mechanism underlying their success in high-dimensional learning. Existing theory partially supports this view by establishing approximation rates based on parameter counts and sample complexity guarantees for compositional models without incurring the curse of dimensionality (CoD). To study overparameterized regimes, where the number of parameters exceeds the sample size, we develop a framework that measures complexity via the parameter norm. Within this approach, we establish approximation rates and excess risk bounds for learning sparse compositional functions whose compositional structure is represented by directed acyclic graphs (DAGs), using Frobenius norm-constrained deep neural networks. Our results have broad applicability since every function that is efficiently Turing computable admits sparse compositional representations. In particular, we cover a range of representative models, including multi-index models, binary tree structures, and general compositional architectures. The rates we derive show that deep networks can exploit the compositional structure of the target functions, effectively avoiding the CoD through hierarchical representations.

• The paper presents a novel analytic framework that uses Frobenius norm constraints to control capacity in deep neural networks.
• It derives norm-dependent approximation rates and excess risk bounds for learning hierarchically sparse compositional Hӧlder functions.
• It demonstrates that modeling compositional structures via DAGs enables deep networks to overcome the curse of dimensionality.

Norm-Based Analysis of Hierarchical Sparse Compositional Function Learning

Introduction and Motivation

The paper "Learning Sparse Compositional Functions with Norm-Constrained Neural Networks" (2605.25608) presents an analytic framework for understanding the statistical efficiency of deep neural networks (DNNs) in learning high-dimensional compositional functions, particularly in the overparameterized regime. Instead of parameter-count complexity, the analysis centers on the Frobenius norm as a capacity control mechanism, which is especially relevant for modern DNNs where network width vastly exceeds sample size. The compositional structure of targets is formalized via directed acyclic graphs (DAGs), reflecting the hierarchically sparse dependencies prevalent in vision, language, and reasoning systems. The paper rigorously establishes approximation rates and excess risk bounds for compositional H\"older function classes under norm-constraints and demonstrates mathematically that hierarchical representation mitigates the curse of dimensionality (CoD) in both classical and overparameterized settings.

Frobenius Norm Constraint for Deep Networks

In contrast to analyses grounded in VC-dimension or parameter counts, the paper scrutinizes deep ReLU networks under multiplicative Frobenius norm constraints across layers, constructing the hypothesis class

$$\mathcal{N}\mathcal{N}(W,D,K) = \left\{ f_\theta \in (W,D) : \kappa(\theta) \le K \right\},\quad \kappa(\theta) = \|A_D\|_{F}\prod_{\ell=0}^{D-1} \|\tilde{A}_\ell\|_F,$$

where $A_\ell$ and $b_\ell$ parameterize each layer, and the norm integrates both weight matrices and bias terms. The product structure mirrors the compositionality and layer-wise multiplicative Lipschitz constants, motivating the shift from additive constraints. This norm constraint yields a hypothesis class consistent with weight decay regularization and the implicit bias of gradient-based optimization in contemporary deep learning.

Approximation Rates for Compositional Functions

The core theoretical contribution is the derivation of norm-dependent approximation rates for both single H\"older spaces and DAG-represented sparse compositional function classes. The approximation rate for unit ball H\"older functions $h \in \mathcal{C}^\alpha([0,1]^d)$, with depth $D$ and Frobenius-norm budget $K$, is given by

$$\|h - \phi\|_{L^\infty} \lesssim K^{-\frac{2\alpha}{2 + d(D+1)}}$$

for sufficiently large network width (polynomial in $K$). Notably, the error exponent depends polynomially on local smoothness and input dimension and only logarithmically on depth, enabling effective function approximation with shallow, wide networks. The construction relies on partition-of-unity techniques with localized hat functions for Taylor expansion, and on shallow wide ReLU subnets for polynomial approximation, decoupling norm budget from width.

Figure 1: Partition of unity achieved by localized hat functions, providing a stable basis for norm-induced approximation.

For sparse compositional functions encoded by DAGs, the error rate is characterized path-wise, with bounds determined by the critical path of constituent node functions: $$\|f - \phi\|_{\infty} \lesssim \max_{\text{path}} K^{-\frac{2\alpha^*}{2L + (D+L)d_{\text{in}}}}$$ where $\alpha^*$ is the accumulated local regularity and $A_\ell$0 the local input dimension along the DAG's path. Representative architectures such as multi-index models, binary trees, and general constant-depth DAGs all yield exponential dependence on local dimensions rather than the ambient $A_\ell$1, e.g., binary tree structures scaling as $A_\ell$2, thus alleviating the CoD.

Statistical Analysis and Excess Risk Bounds

Empirical risk minimization within the norm-constrained class is analyzed, yielding excess risk bounds for supervised regression: $A_\ell$3 with explicit scaling for the norm constraint $A_\ell$4 balancing approximation and Rademacher complexity–driven estimation error. The rate depends exclusively on critical-path local structure rather than global ambient dimensionality. The derivation leverages concentration inequalities, Rademacher complexity contraction principles, and norm-based generalization bounds for ReLU networks. The results cover both classical and overparameterized scenarios, with optimal scaling for $A_\ell$5 yielding statistically efficient learning in the limit as network width grows.

Structural Implications: Compositionality and Efficiency

From a theoretical perspective, the norm-based analysis demonstrates that DNNs can exploit compositional structure to overcome classical limitations imposed by parameter-count complexity. The path-wise accumulation of smoothness indices and input dimensions directly translates to adaptability and efficiency in high-dimensional regimes. Approximation and excess risk rates remain exponential only in local latent dimensions, confirming deep network adaptivity to data structure. The generality of the DAG formalism covers multi-index models, trees, and stair-case functions, as well as architectures underlying reasoning in LLMs and hierarchical data manifolds. The partition-of-unity and localized function block techniques also have direct implications for stable numerical approximation and function decomposition.

Practical Relevance and Future Directions

Practically, these findings clarify why norm-based regularization (e.g., weight decay) is effective for DNNs in overparameterized contexts. Gradient descent and related optimization algorithms induce an implicit bias toward minimal-norm solutions, aligning with statistical guarantees derived here. The analysis raises compelling questions about further refining capacity control, e.g., via margin or sharpness-based measures, and extending norm-based bounds to refine generalization guarantees for Tikhonov-regularized models or implicitly regularized interpolators. The methodology provides a pathway for function-space analytic characterization of deep networks, which is critical for understanding compositionality in large-scale AI models, including transformers and CNNs. Robustness, computational efficiency, and adaptivity to mixed smoothness or latent manifold structure remain important directions for further research.

Conclusion

By establishing norm-constrained approximation and generalization rates for sparse compositional functions, the paper rigorously demonstrates that deep networks equipped with hierarchical structure and explicit norm control can circumvent the curse of dimensionality, even in highly overparameterized regimes. The dependence on local rather than global complexity marks a substantial theoretical advance for the analysis of large-scale deep learning systems. The norm-based analytic perspective outlined here is of direct utility for both fundamental research and practical model selection in contemporary AI architectures.