Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks

Abstract: This paper studies how efficiently deep ReLU neural networks can approximate and learn smooth functions. When the error is measured in $L<sup>p([0,1]<sup>d)$ norm and the approximator is a network with width $W$ and depth $L$, recent works have proven the supper approximation rate $\mathcal{O}((WL)<sup>{-2s/d})$ for Besov space $\mathcal{B}<sup>s_{q,r}([0,1]<sup>d)$ under the Sobolev embedding condition $s/d&gt;1/q-1/p$. In order to overcome the curse of dimensionality in this rate, we extent this result to anisotropic and mixed smooth function classes. We establish the approximation rate $\mathcal{O}((WL)<sup>{-2\tilde{s}})$ for anisotropic Besov space $$\mathcal{B}<sup>{\boldsymbol{s}}_{q,r}([0,1]<sup>d)$$ with anisotropic smoothness $\boldsymbol{s}=(s_1,\dots,s_d)$ under the embedding condition $\tilde{s} &gt; 1/q-1/p$, where the mean smoothness $$\tilde{s} = (\sum_{i=1}^d s_i<sup>{-1})<sup>{-1}$$. For mixed smooth Besov space $\mathcal{MB}<sup>s_{q,r}([0,1]<sup>d)$ with mixed smoothness $s&gt;1/q-1/p$, we show that the approximation rate $\mathcal{O}((WL)<sup>{-2s})$ holds up to logarithmic factors. Using these results, we also derive approximation bounds for the composition of anisotropic Besov functions. As an application, it is shown that deep ReLU neural networks can achieve minimax optimal rates up to logarithmic factors for a wide range of smooth function classes.

• The paper demonstrates that deep ReLU networks achieve optimal Lp approximation rates for anisotropic Besov spaces up to logarithmic factors.
• The paper shows that exploiting mixed smoothness and compositional structures circumvents the curse of dimensionality in high-dimensional regression.
• The paper provides explicit width-depth tradeoffs, offering practical guidelines for network design in scientific computing and high-dimensional learning.

Approximation and Learning of Anisotropic and Mixed Smooth Functions by Deep ReLU Networks

Introduction and Problem Setting

This paper rigorously analyzes the approximation and statistical learning capacities of fully-connected feedforward ReLU networks when the target functions exhibit either anisotropic or mixed smoothness, as described in the framework of (anisotropic and mixed) Besov spaces. Recent deep learning theory has characterized sharp approximation rates for isotropic, globally smooth function classes such as Sobolev and (isotropic) Besov spaces; however, such rates degrade rapidly in high dimensions due to the curse of dimensionality. Practical data often possess intrinsic low-dimensional or structured geometry—manifested as anisotropic or mixed smoothness—where smoothness varies across coordinates or is dominated by mixed partial derivatives. This work extends and strengthens previous results by providing optimal (up to log factors) rates for neural network approximation and minimax learning in these structured function classes, with full explicit trade-offs between width and depth.

Main Contributions

1. Supper Approximation Rates for Anisotropic Besov Spaces

Previous results established that deep ReLU networks can achieve $L^p$-error at the rate $\mathcal{O}((WL)^{-2s/d})$ for isotropic Besov spaces $\mathcal{B}^s_{q,r}([0,1]^d)$ under the Sobolev embedding condition $s/d > 1/q - 1/p$, where $W$ is the network width and $L$ the depth. This paper generalizes to anisotropic Besov spaces $\mathcal{B}^{\mathbf{s}}_{q,r}$, where $\mathbf{s} = (s_1,\dots,s_d)$ is a vector of per-coordinate smoothness. It establishes that, under the embedding condition $\tilde{s} > 1/q - 1/p$ with mean smoothness $\tilde{s} = (\sum_{j=1}^d s_j^{-1})^{-1}$, ReLU networks achieve

$\mathcal{O}((WL)^{-2s/d})$0

for any $\mathcal{O}((WL)^{-2s/d})$1. This rate is optimal up to logarithmic factors, as shown by matching lower bounds.

2. Supper Approximation for Mixed Smooth Besov Spaces

For spaces of mixed smoothness $\mathcal{O}((WL)^{-2s/d})$2 (dominating mixed derivatives), where functions may be highly non-smooth except along higher-order mixed derivatives, the curse of dimensionality can be circumvented. The paper proves that, up to log factors, the optimal rate in $\mathcal{O}((WL)^{-2s/d})$3-approximation is

$\mathcal{O}((WL)^{-2s/d})$4

with the threshold $\mathcal{O}((WL)^{-2s/d})$5, again with nearly matching lower bounds. This demonstrates that the effective approximation power depends only polynomially (and not exponentially) in $\mathcal{O}((WL)^{-2s/d})$6, provided one exploits mixed smoothness.

3. Generalization to Deep Compositional Models

Extension is given to deep compositional classes (functions of functions) in which each layer is a vector of functions in anisotropic Besov spaces. By deriving a careful propagation of approximation error and smoothness loss through layers, the optimal rate is shown to depend on an explicit "compositional smoothness index" $\mathcal{O}((WL)^{-2s/d})$7 capturing the weakest smoothness and contraction at each composition level.

4. Supervised Learning: Minimax Rates

These approximation results are leveraged to analyze minimax nonparametric regression with least-squares neural network fits. For a sample size $\mathcal{O}((WL)^{-2s/d})$8, minimax optimal learning rate bounds (up to log factors) are achieved for the various structured spaces, showing, e.g.,

$\mathcal{O}((WL)^{-2s/d})$9

in anisotropic cases, and dimension-independent rates (modulo logs) under mixed smoothness, thus closing the gap between achievable rates by sparsely-connected and fully-connected deep nets.

Methodological Innovations

The analytical machinery combines multiscale, adaptive anisotropic grid constructions, sharp decompositions of anisotropic/mixed Besov functions into localized polynomial pieces, and recent advances in neural network-based realization of piecewise polynomials. The construction leverages technical results on network depth/width tradeoffs for interpolation of multidimensional index maps, and employs nontrivial use of entropy, pseudo-dimension, and packing arguments for lower bounds.

Strong claims are made in several respects:

• Sparsity is proved unnecessary: fully-connected (dense) networks admit rates previously only possible for specialized architectures with nonzero parameter constraints.
• The width-depth tradeoff is completely explicit (whereas previous work mostly counted parameters).
• In linear ($\mathcal{B}^s_{q,r}([0,1]^d)$0) and nonlinear ($\mathcal{B}^s_{q,r}([0,1]^d)$1) approximation regimes, the analysis achieves the sharpest known exponents in all relevant Besov scales.

Validation of Optimality and Limitations

Lower bounds are established via metric entropy and pseudo-dimension arguments, showing optimality of the upper rates up to log factors, and demonstrating that no ReLU network class of polynomial size in $\mathcal{B}^s_{q,r}([0,1]^d)$2 can do better, even when broadening the hypothesis class to all bounded-depth ReLU architectures. Small remaining gaps in log factors are noted, and in the case of general $\mathcal{B}^s_{q,r}([0,1]^d)$3 (third Besov parameter) for mixed smoothness, an explicit conjecture is left for optimal log scaling.

Implications and Future Developments

Theoretical

• The results unify and generalize the theory of function approximation and statistical learning by neural networks for a variety of structured function spaces relevant in high dimensions.
• They provide theoretical justification for the empirical observation that deep networks can efficiently learn high-dimensional targets if the function possesses certain structural regularities (anisotropic or mixed smoothness, compositionality).

Practical

• Optimal width/depth scaling guidance is given for practitioners fitting deep nets to data known or conjectured to conform to structured smoothness.
• These findings can guide architectural design and hyperparameter selection for applications in scientific computing, PDE solvers, and high-dimensional regression where model smoothness is structured.

Potential Extensions

• Extension of results to convolutional networks is addressed in the appendix and can be further strengthened to more general architectures.
• The framework suggests generalization to data lying on (unknown) low-dimensional manifolds or under more general integral representations.
• Open questions remain on adaptation to unknown smoothness parameters and on sharp characterization of minimax rates for other classes such as manifold learning or more general mixing/anisotropy scenarios.

Conclusion

This work gives a technical and comprehensive answer to the question of how well—and under which precise conditions—deep ReLU networks can approximate and learn highly structured smooth functions, eliminating the curse of dimensionality in appropriate regimes. The match between upper and lower bounds, explicit network construction, and the breadth of function classes covered advance the theoretical understanding of deep learning in high-dimensional approximation and statistical learning settings.

Reference: "Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks" (2605.31152)