---
title: Error Decay in Network Approximation
url: https://www.emergentmind.com/topics/error-decay-in-network-approximation
type: topic
---

# Error Decay in Network Approximation

Error decay in network approximation quantifies how the best-achievable approximation error for a target function decreases as the complexity of a neural network, measured by its width, depth, or total number of nonzero parameters, increases. Recent research rigorously formalizes this using approximation spaces, establishes sharp decay rates for a wide variety of architectures and function classes, and connects these rates to the network's structural and training properties. The regime of possible error-decay rates encompasses polynomial, root-exponential, exponential, and even geometric rates, depending on architecture, activation functions, target regularity, width-depth allocations, and algorithmic considerations.

## 1. Formalization via Approximation Spaces

Error decay is naturally captured by approximation spaces $A^r_q(X, \Sigma)$, which classify functions $f$ in a normed space $X$ by the rate at which the best-approximation error $E_n(f) = \inf_{g \in \Sigma_n} \|f - g\|_X$ decays as the network complexity $n$ increases, where $\Sigma_n$ is the set of functions that can be realized by networks with complexity at most $n$ and bounded depth (or depth growing as a prescribed function of $n$).

The space $A^r(X, \Sigma)$ consists of those $f$ for which $\sup_{n \ge 1} n^r E_n(f) < \infty$, corresponding to error decay $E_n(f) = O(n^{-r})$. One can refine this to $A^r_q(X, \Sigma)$ by requiring $(\sum_n [n^r E_n(f)]^q n^{-1})^{1/q} < \infty$; for $q = \infty$ this is the same as $A^r$.

Under mild hypotheses on the function spaces and network classes (e.g., nestedness, scale-invariance, density), $A^r_q$ are (quasi-)Banach spaces with inclusion relations mirroring those of classical interpolation spaces. Key direct ("Jackson") and inverse ("Bernstein") inequalities ensure that for $f \in A^r$,
$$
E_n(f) \leq C n^{-r} \|f\|_{A^r}, \quad \text{and conversely, } \|g\|_{A^r} \leq C n^r \|g\|_X \text{ for } g \in \Sigma_n.
$$
This precisely captures the tradeoff between network complexity and error decay [1905.01208].

## 2. Depth, Activation, and Structural Constraints

The achievable exponent $r$ in $E_n(f) = O(n^{-r})$ is fundamentally limited by depth and activation choice:

- **Depth Limitation:** For ReLU activation, fixed depth $L$ severely restricts attainable $r$. One cannot have a nontrivial function in $A^r$ for $r > 2\lfloor L/2\rfloor$ (or $r > 2(L-1)$ in neuron-count complexity). Constructing highly oscillatory "sawtooth" functions and counting the number of linear/polynomial pieces per depth provides tight lower bounds [1905.01208].
- **Activation Functions:** For $\sigma_r(x) = (x_+)^r$, the approximation spaces $A^r_q(X, \Sigma)$ remain nontrivial for any $r$ and $L \ge 2$, and for spline activations of degree $r$ (with at least one knot), the same spaces are realized. These approximation spaces coincide with the universality properties of classical piecewise polynomial approximation.
- **Skip-connections:** Allowing generalized layerwise activations (per-neuron "id" or activation) or skip-connections does not alter the resulting approximation spaces $A^r_q$. Thus, the universality class is robust to such architecture modifications [1905.01208].

## 3. Direct and Inverse Estimates: Relation to Smoothness

For ReLU and higher-power activations, rigorous embeddings relate these network approximation spaces to classical smoothness (e.g., Besov spaces $B^s_{p, q}$):

- **Direct estimate:** Using $\sigma_r(x)$ with sufficient depth, functions in $B^{d s}_{p, q}$ can be embedded into $A^s_q$, providing
  $$
  E_n(f) \leq C n^{-s} \quad \text{for } s < \min\{r + 1, 1 + 1/p\}/d.
  $$
  For $d=1$ and $L \geq 2$, $s < r + 1$ is admissible.
- **Inverse estimate:** With finite depth $L$ and ReLU, any embedding $A^r_q(\sigma_1) \hookrightarrow B^s_{*, *}$ requires $s \leq 2\lfloor L/2\rfloor$, sharply limiting recoverable Besov smoothness [1905.01208].

Approximation rates are thus bounded above (direct) and below (inverse) by the available network complexity and target smoothness, matching the best-known Jackson/Bernstein bounds for splines and wavelets, and linking deep network approximation directly to the classic theory of function spaces.

## 4. Architectural and Training Regimes: Beyond Polynomial Decay

Network approximation rates exhibit various regimes beyond the classical algebraic decay:

- **Exponential Error Decay:** For many classes of smooth or oscillatory functions (e.g., polynomials, sinusoids, certain $C^\infty$ functions, Besov or modulation space balls), deep networks of polylogarithmic depth and finite width achieve exponential decrease:
  $$
  \varepsilon \leq C e^{-cM}
  $$
  where $M$ is the number of nonzero parameters or connections. This aligns with Kolmogorov-Donoho optimality [1901.02220].
- **Root-exponential and superexponential rates:** For Floor–ReLU architectures, uniform approximation of Hölder functions achieves
  $$
  \|f - f_N\|_\infty \leq 3\lambda d^{\alpha/2} N^{-\alpha \sqrt{L}}
  $$
  with $N$ width and $L$ depth, i.e., error is "reciprocal of width to the power $\sqrt{\text{depth}}$," yielding root-exponential convergence in depth and demonstrating substantial mitigation of the curse of dimensionality [2006.12231].
- **Geometric Error Decay:** Multilevel neural network architectures (e.g., for AFEM-adapted solvers or extended Galerkin frameworks) can exhibit geometric error decay in the number of network levels/basis functions, e.g., $||u-u_n|| \le (2\epsilon)^n ||u-u_0||$, given subspace enrichment that precisely targets singular structures and regular remainder [2403.12650, 2405.00815].

## 5. Algorithmic and Training Effects on Error Decay

Error decay is also heavily influenced by training algorithms and initialization regimes:

- **Gradient Descent Dynamics:** In the over-parameterized (so-called “lazy” or NTK) regime, gradient descent iteratively applies (in function space) powers of an integral operator $T$ whose eigenvalues control the geometric decay:
  $$
  \|e(t)\|_2 \leq (1 - \eta\lambda_\ell)^t \|f^*\|_{L^2} + O(\epsilon(f^*, \ell)) + O(n^{-1/2})
  $$
  where $\epsilon(f^*, \ell)$ is the error projecting onto the top-$\ell$ eigenspaces of $T$ [1905.10826].
- **Empirical Error Decay under Gradient Flow:** For networks trained on Sobolev-smooth targets, with constant depth and increasing width, the $L^2$-error converges at rates $O(m^{-r/[4(r-\beta)]})$, with exponents determined by smoothness $r$ and the NTK coercivity gap $\beta$ [2309.04860].
- **Taylorized Training:** The error of the $k$-th order Taylor expansion (centered at initialization) of a wide network decays exponentially with $k$, i.e., $O(m^{-k/2}) = O(e^{-\alpha k})$, showing that only modest $k$ is needed to close the gap with exact network dynamics for wide architectures [2002.04010].
- **Random Initialization Ensemble Methods:** In stochastic training with multiple restarts, the overall error can be decomposed into explicit approximation, generalization, and optimization components, with improved exponential rates in depth and logarithmic scaling in width [2012.08443].

## 6. Limitations, Instabilities, and Lower Bounds

Fundamental limits and instabilities are intrinsic to network approximation:

- **Stability vs. Expressivity:** The "space-filling" property of the manifold of network outputs under parameter optimization confers expressivity but can undermine numerical stability in finding best parameterizations. Stable parameter selection (Lipschitz maps) cannot outperform entropy-bounded rates $n^{-r}$, while unstable/delicate methods could achieve superrates (log-factor improvements) in some regimes [2012.14501].
- **Curse of Dimensionality and Activation Function:** With general activation functions, error decay can be dimension-independent—$O(n^{-1/2})$ for polynomial decay activations or $O(n^{-1/4})$ for bounded, integrable activations—but with explicit $d$-dependence for stratified sampling under increased smoothness [1904.02311].
- **Width-Depth Trade-offs and Limits:** Sharp lower bounds prove that fixed-depth, wide networks cannot achieve the same error decay (for high-frequency or highly oscillatory targets) as deep, narrow networks. Polylogarithmic-depth, finite-width constructions match Kolmogorov-Donoho optimal exponents only if a matching dictionary with polylog-depth implementations exists [1901.02220].

## 7. Summary Table: Representative Error Decay Regimes

| Network/Function Class     | Error Decay Rate                            | Limiting Factor / Notes            | Reference      |
|---------------------------|---------------------------------------------|------------------------------------|---------------|
| ReLU, depth $L$, width $n$ | $O(n^{-r}),\ r\leq 2\lfloor L/2\rfloor$    | Depth $L$ (sawtooth lower bound)   | [1905.01208]  |
| Floor–ReLU, width $N$, depth $L$ | $O(N^{-\alpha \sqrt{L}})$             | "Root-exponential" in $\sqrt{L}$   | [2006.12231]  |
| Deep (polylog-depth), $M$ params | $O(e^{-cM})$                         | Exponential in parameter count     | [1901.02220]  |
| Two-layer, poly-decay activation | $O(n^{-1/2})$                         | Polynomial decay of activation     | [1904.02311]  |
| Bounded activation         | $O(n^{-1/4})$                               | No decay; $L^\infty$ activation    | [1904.02311]  |
| Super-convergence (deep, Besov-Sobolev) | $O(N^{-2s/d})$               | Requires sufficient depth          | [2012.14501]  |
| Gradient descent, NTK regime | $O((1-\eta\lambda_i)^t)$ (geometric in $t$) | Controlled by leading eigenvalue   | [1905.10826]  |
| Multilevel adaptive net    | $O(\rho^L)$                                | Geometric in levels ($\rho < 1$)   | [2403.12650]  |

## References

- [1905.01208] Approximation spaces of deep neural networks
- [2006.12231] Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
- [1901.02220] Deep Neural Network Approximation Theory
- [2012.14501] Neural Network Approximation
- [2012.08443] Strong overall error analysis for the training of artificial neural networks via random initializations
- [2405.06727] Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces
- [2511.15326] A Quantifier-Reversal Approximation Paradigm for Recurrent Neural Networks
- [2403.12650] Adaptive Multilevel Neural Networks for Parametric PDEs with Error Estimation
- [2002.04010] Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
- [2309.04860] Approximation Results for Gradient Descent trained Neural Networks
- [1904.02311] Approximation Rates for Neural Networks with General Activation Functions
- [1404.6449] Univariate error function based neural network approximation

A comprehensive understanding of error decay in network approximation thus integrates sharp theoretical decay rates, robust structural principles, links to classical smoothness theory, algorithmic and training effects, as well as explicit limitations imposed by both network and problem class properties.

Source: https://www.emergentmind.com/topics/error-decay-in-network-approximation