Continuous Neural Function Approximation

• Continuous neural function approximation is a framework for representing and approximating continuous scalar, vector, or operator-valued functions using diverse neural network architectures.
• It leverages universal approximation theorems, quantitative error bounds, and constructive algorithms to optimize network depth, width, and parameterization.
• The approach extends to infinite-dimensional spaces and non-compact domains through operator-valued networks, neural ODE/SDE, and structured decompositions.

Continuous neural function approximation refers to the theoretical and algorithmic framework for representing and uniformly approximating continuous functions—scalar-, vector-, or operator-valued—on compact subsets (and, with suitable modifications, on non-compact domains and even infinite-dimensional spaces) using neural network architectures. Core to this field are universal approximation theorems, quantitative and constructive bounds, complexity analyses (in terms of depth, width, and parameterization), and algorithmic realizations for diverse architectures such as feed-forward, convolutional, recurrent, neural ODE, neural SDE networks, and functional (operator-valued) networks.

1. Universal Approximation Theorems: Foundational Results

The classical universal approximation theorems assert that feedforward neural networks with a single hidden layer, equipped with any continuous non-polynomial activation function, are dense in the space of continuous functions on any compact subset $K\subset\mathbb{R}^d$. That is, given any $f\in C(K)$ and any $\epsilon > 0$, there exist weights and biases such that the neural network $$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$ satisfies $\|f - F_N\|_{C(K)} < \epsilon$ (Mukherjee et al., 20 May 2026, Chong, 2020). The necessity of the non-polynomial condition on $\sigma$ is strict; polynomial activations can only reproduce polynomial targets.

For general continuous targets $f$, the required number of hidden units scales as $O(\epsilon^{-d})$, corresponding to the classical curse of dimensionality. For polynomial targets of degree $d$, networks with $\binom{n+d}{d}$ neurons suffice—independent of $f\in C(K)$0 and of output dimension (Chong, 2020).

Extensions include vector-valued functions, $f\in C(K)$1-spaces, uniform approximation on all of $f\in C(K)$2 under envelope conditions (Kratsios, 2019), and functions on arbitrary separable Banach and Fréchet spaces. In the infinite-dimensional (functional data) regime, Benth–Detering–Galimberti (Benth et al., 2021) generalize this theorem by introducing "neurons" defined by nonlinear activation operators and affine maps, providing density in $f\in C(K)$3 for any Fréchet space $f\in C(K)$4 when the activation operator is discriminatory or satisfies a separating limit condition.

2. Quantitative and Constructive Approximation Schemes

While qualitative density results are classical (e.g., Cybenko 1989), contemporary research seeks explicit error bounds as a function of smoothness, dimension, and network size.

• Modulus of Continuity and Hölder Classes: For $f\in C(K)$5, the modulus $f\in C(K)$6 yields error rates of the form

$f\in C(K)$7

for ReLU or sigmoidal networks of width $f\in C(K)$8 (Mukherjee et al., 20 May 2026, Shen et al., 2020). FLES (Floor-Exponential-Step) architectures achieve even super-exponential rates in $f\in C(K)$9 for Hölder targets, with error $\epsilon > 0$0 and only mild $\epsilon > 0$1-dependence (Shen et al., 2020).

• Barron Class: For functions with bounded Fourier-Barron seminorm, error decays as $\epsilon > 0$2 in mean-square (Mukherjee et al., 20 May 2026). Explicit constructive schemes exist for special activations such as the error function (Anastassiou, 2014), providing Jackson-type inequalities in the uniform norm.
• One-Shot Closed-Form Algorithms: The Universal Function Algorithm (UFA) realizes arbitrary continuous scalar or vector-valued $\epsilon > 0$3 by explicit parameter choice (with non-iterative, closed-form solutions for the weights) and selects one hidden neuron per sampled point to achieve a prescribed error, at the cost of linear network size in $\epsilon > 0$4 (Zaresky-Williams, 2019).
• Random Feature Methods: For activations in a broad class, mollified integral representations yield non-asymptotic $\epsilon > 0$5 and (with smoothness) $\epsilon > 0$6 error bounds for networks with random weights, and sample complexity scaling polynomially in $\epsilon > 0$7 (Lekang et al., 2023).

Table: Parameter Scaling for Uniform Approximation on $\epsilon > 0$8

Target $\epsilon > 0$9	Architecture	Parametric Complexity	Error Rate
General $$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$0	1-hidden-layer	$$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$1	$$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$2
Barron class	1-hidden-layer	$$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$3	$$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$4 in $$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$5
EUAF, fixed-depth [SYZ/EUAF]	Specialized (depth 11)	$$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$6	Exact for any $$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$7, linear in $$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$8
FLES (Shen et al., 2020)	3 layers, width $$F_N(x) = \sum_{i=1}^N a_i\,\sigma(w_i\cdot x + b_i)$$9	$\|f - F_N\|_{C(K)} < \epsilon$0	Super-exponential in $\|f - F_N\|_{C(K)} < \epsilon$1
Barycentric NN (CPLF)	Shallow	$\|f - F_N\|_{C(K)} < \epsilon$2, $\|f - F_N\|_{C(K)} < \epsilon$3 = # simplices	Arbitrarily accurate for sufficient base points

3. Depth, Width, and Architectural Trade-Offs

Width vs. Depth: Although one-hidden-layer networks are universal, increasing depth exponentially enhances expressivity and can dramatically reduce the required parameter count for certain compositional targets (Mukherjee et al., 20 May 2026). E.g., Telgarsky's sawtooth functions and Eldan–Shamir's constructions establish situations where shallow networks incur exponential width requirements compared to deep narrow architectures.

Fixed-width, Deep Networks: KANs (Kolmogorov–Arnold Networks), inspired by the Kolmogorov superposition theorem, achieve dimension-independent constructive rates by learning univariate functions along graph edges; for analytic targets, deep architectures and KANs admit optimal, sometimes linear-in-$\|f - F_N\|_{C(K)} < \epsilon$4, parameter scaling (Mukherjee et al., 20 May 2026, Maiti et al., 2024).

Super-Approximation Property: With specialized activation functions (e.g., the elementary universal activation function, EUAF), one can build fixed-size, fixed-depth architectures with parameter count $\|f - F_N\|_{C(K)} < \epsilon$5 (linear in input dimension) that realize $\|f - F_N\|_{C(K)} < \epsilon$6-approximation for all $\|f - F_N\|_{C(K)} < \epsilon$7, bypassing the classical curse of dimensionality (Maiti et al., 2024, Shen et al., 2021). The optimality (for fixed depth) is established, as at least width $\|f - F_N\|_{C(K)} < \epsilon$8 is necessary in worst case (Maiti et al., 2024).

Convolutional and Structured Networks: For continuous function approximation over high-dimensional spaces or latent spaces with low intrinsic dimension, convolutional and block-term neural decompositions (e.g., in NeuApprox (Wu et al., 4 Mar 2026)) recover optimal error and sample complexity. CNN architectures with sigmoidal activation are provably universal for continuous targets on compact input sets, with parameter scaling as in the MLP case but with favorable noise insensitivity and efficiency (Chang, 2022).

4. Infinite-Dimensional, Operator, and Temporal Function Approximation

Function approximation extends to Banach and Fréchet space input domains using networks whose "activations" are nonlinear operators (Benth et al., 2021). For any compact subset of a separable Fréchet space $\|f - F_N\|_{C(K)} < \epsilon$9, suitable networks can approximate all $\sigma$0, provided the activation operator is discriminatory (roughly, separating all measures via neuron-integral tests).

Finite-dimensional truncation/projection arguments allow for practical reduction of infinite-dimensional networks to computable finite networks, with the error controlled by the truncation error of a Schauder basis. This directly supports functional data analysis and neural operator (e.g., DeepONet) applications.

Neural ODEs, RNNs, and their stochastic generalizations can universally approximate continuous functionals and operators. For neural ODEs, universal approximation holds for suitable non-polynomial activations; under additional Lipschitz or spectral norm constraints, explicit error bounds quantify the tradeoff between stability and approximation error (Marinis et al., 19 Mar 2025). Neural SDEs can realize a wide range of continuous mappings, with the richness controlled by the dimension and Lie-algebraic complexity of the underlying stochastic process (Veeravalli et al., 2023).

Recurrent neural networks of fixed size and fixed weights, when run for sufficiently many steps, can uniformly approximate any continuous $\sigma$1—the approximation error can be made arbitrarily small by increasing runtime, with explicit convergence rates governed by Chebyshev/polynomial approximation schemes (Abadie et al., 18 Jun 2026).

5. High-Dimensional Sets: Intrinsic vs. Ambient Complexity

Modern high-dimensional applications often involve latent or model sets $\sigma$2 of low intrinsic complexity (e.g., sparse, low-rank, or manifold-structured). The effective dimensionality of continuous neural approximation for such sets is governed not by $\sigma$3 but by the Gaussian width or covering number scaling of the set of unit secants of $\sigma$4 (Karnik et al., 2022). Johnson–Lindenstrauss embeddings allow for initial linear compression to dimension $\sigma$5, after which the required neural network size for Hölder targets scales as $\sigma$6. Three typical cases analyzed:

• $\sigma$7-sparse: $\sigma$8,
• rank-$\sigma$9 matrices: $f$0,
• low-dimensional manifolds: $f$1 (intrinsic dim).

Hence, practical neural approximation in high dimensions is feasible when the data or target function class is restricted to subsets of low complexity.

6. Alternative Paradigms, Topological and Geometric Interpretability

Beyond standard parametric networks, recent approaches utilize structural or geometric representations:

• Barycentric Neural Networks (BNN): Networks parameterized directly by domain base points, exactly representing continuous piecewise linear functions and achieving universal approximation for continuous targets on compacta when the set of base points is sufficiently rich (Toscano-Duran et al., 8 Sep 2025).
• Topological Losses: Training objectives based on topological signatures, such as length-weighted persistent entropy, can lead to significantly more robust and interpretable approximations in low-resource and function-reconstruction scenarios (Toscano-Duran et al., 8 Sep 2025).
• Block-term Neural Approximations: Decomposing multivariate functions into sums of tensor products of univariate neural basis functions, trained end-to-end, enables state-of-the-art data adaptation, low-rank modeling, and modular learning in practical settings (Wu et al., 4 Mar 2026).

These approaches provide alternative means for interpreting, controlling, and regularizing neural function approximators in context-driven applications.

7. Limitations, Open Directions, and Ongoing Developments

While the theoretical landscape of continuous neural function approximation is comprehensive, open challenges remain:

• For standard activations (ReLU, sigmoid), the curse of dimensionality persists in worst-case approximation rates unless one leverages structural prior (low-dimensional sets, compositionality).
• Although super-approximation via fixed-size networks is possible with EUAF-type activations, these are nonstandard and have not yet seen widespread computational adoption.
• Extension to non-compact domains, $f$2, and L$f$3-spaces (including closed-form closure descriptions) depends sensitively on the growth and limit behavior of the activation; the algebraic structure of function classes realizable with deep networks is a recent subject of advanced study (Nuland, 2023).
• Infinite-dimensional and operator settings require more elaborate activation/operator designs; projection-based methods are practical but may not capture all relevant structure in very high-complexity tasks.

Ongoing research addresses these issues by further refining depth-width trade-offs, constructing universal networks with explicit error quantification in complex regimes, and integrating advances from operator theory, random matrix theory, topological data analysis, and functional analysis. The highly interdisciplinary and constructive nature of this area continues to yield new methods and theoretical clarifications, with deep connections to both classical and modern approximation theory.