Universal Approximation: Multi-Input Operators

• Universal approximation theory for multi-input operators is a framework that establishes how neural networks approximate continuous mappings on infinite-dimensional spaces using structured architectures.
• It extends classical approximation theorems from Euclidean domains to multi-branch neural networks in Banach and topological vector spaces, emphasizing precise error estimates and scaling laws.
• The theory underpins scientific applications by guiding the design of operator learning models like DeepONet, MIONet, and transformer-based architectures.

Universal Approximation Theory for Multi-Input Operators

Universal approximation theory for multi-input operators characterizes the ability of neural network architectures to approximate arbitrary continuous mappings where the input variable is a collection of functions, sequences, matrices, or general elements of infinite-dimensional (often topological vector) spaces. This line of research rigorously extends the classical universal approximation theorem—from functions on Euclidean domains to nonlinear operators acting on products of Banach or topological vector spaces—and delivers a theoretical foundation for the design of modern operator-learning neural architectures, such as DeepONet and MIONet. The theory also encompasses universality results for specialized architectures, including mixture-of-experts neural operators, projection-based operator networks, and transformer in-context learners.

1. Mathematical Framework: From Topological Vector Spaces to Banach Products

Multi-input operator approximation theorems operate within the framework of topological vector spaces (TVS) or Banach spaces. Let $X_1, \dots, X_n$ be Banach spaces (or more generally, real TVSs with the Hahn–Banach extension property), and $Y$ a Banach space as the output space. The joint input is then an element of the product space $X = X_1 \times \cdots \times X_n$, and the target operator is a continuous map $\mathcal{G}: K_1 \times \cdots \times K_n \to Y$ where each $K_i \subset X_i$ is compact.

Key properties:

• The dual space $X^*$ must separate points, guaranteed if each $X_i$ enjoys the HBEP, which is automatic for locally convex TVSs (Ismailov, 19 Sep 2024).
• Activation functions $\sigma: \mathbb{R} \to \mathbb{R}$ must be continuous and non-polynomial (Tauber–Wiener class).
• Universal approximation is established in the topology of uniform convergence on compacts.

Multi-input operator learning is essential for scientific domains requiring the solution of parametric families of ODEs or PDEs, with distinct function-valued inputs such as boundary conditions, initial data, and coefficients (Jin et al., 2022).

2. Core Universal Approximation Theorems

Several lines of universal approximation results underpin the theory:

1. TVS-FNN Universality: For any TVS $X$ with HBEP and continuous non-polynomial $\sigma$, the linear span of $$\{ x \mapsto \sigma(f(x) - \theta): f \in X^*, \theta \in \mathbb{R} \}$$ is dense in $C(K)$ for every compact $K \subset X$. This result holds in all locally convex TVSs, encompassing functions, sequences, matrices, etc. The proof involves a reduction to the Stone–Weierstrass theorem and shows that shallow feedforward networks suffice for operator universality in this broad setting (Ismailov, 19 Sep 2024).
2. MIONet Theorem: Any continuous operator $\mathcal{G}: K_1 \times \cdots \times K_n \to Y$ can be uniformly approximated on compacts by a multilinear finite-rank expansion of the form

$$\mathcal{G}(v_1, \dots, v_n) \approx \sum_{k=1}^p \left(g^1_k(\varphi^1_{q_1}(v_1)) \cdots g^n_k(\varphi^n_{q_n}(v_n))\right) u_k$$

with each $v_i$ embedded via a finite-dimensional projection, and $g^i_k,$ $u_k$ realized by feedforward networks (Jin et al., 2022). This formalizes the multi-branch low-rank structure that motivates MIONet.

1. Encoder–Decoder Universality: For any normed spaces $X$, $Y$ with the encoder–decoder approximation property (EDAP), all continuous operators $G: X \to Y$ can be approximated by sequences of encoder/decoder neural architectures, where the same sequence achieves uniform convergence on every compact subset. Multi-input architectures arise as encoder–decoder nets on product spaces (Gödeke et al., 31 Mar 2025).
2. Extension to Banach Spaces and Polynomial Projection: Any continuous $T: X \to Y$ (with $X, Y$ Banach) can be approximated by composing a finite-dimensional (e.g., polynomial) projection with a neural network $G_N$ (Zappala, 18 Jun 2024). For $X = L^p(\Omega)$, this is realized via orthogonal projection onto multivariate polynomials followed by a NN on the resulting coefficient vector.

3. Architectural Constructions and Operator Network Instantiations

These theoretical foundations support a variety of neural operator architectures, with explicit constructions for multi-input cases:

• TVS-FNNs: Shallow networks with generalized inputs and functional connections corresponding to dual elements of $X^*$, permitting direct embeddings for matrices, sequences, and functions (Ismailov, 19 Sep 2024).
• Branch–Trunk Structures: Generalization of DeepONet and MIONet, where each input function feeds into its own branch net, mixed via a hadamard or multilinear fusion with a trunk net for output coordinates (Jin et al., 2022).
• Nonlocal Neural Operators and ANO: Concatenation of multiple input channels, lifted into a high-dimensional embedding, followed by global averaging and pointwise nonlinearity, yields universal approximation in operator norms—even in minimal architectures (ANO) (Lanthaler et al., 2023).
• Mixture-of-Experts Neural Operators (MoNO): Partitioning the function space into local regions, each delegated to a small neural operator, allows distributed universal approximation with per-expert depth, width, and rank $\mathcal{O}(\varepsilon^{-1})$ (Kratsios et al., 13 Apr 2024).
• Transformer-based Operator Learning: For multi-example, multi-input settings, transformer architectures can approximate mappings from in-context prompts $\{(x_i, y_i)\}_{i=1}^N, x_\star \mapsto f(x_\star)$ universally, using Barron features and in-prompt Lasso regression (Li et al., 5 Jun 2025).

4. Error Estimates and Scaling Laws

Quantitative aspects of the theory include explicit error decompositions:

• Projection Error: Controlled by the choice of finite-dimensional projections, e.g., truncating at $q_i$ basis elements yields error $L_i(q_i) \to 0$ as $q_i \to \infty$ (Jin et al., 2022).
• Network Approximation Error: For each subnetwork, error decays with width/depth according to standard finite-dimensional UAT rates.
• Total Error: For tensorized expansions, total error bounds are additive over projection and network terms: $$\|\mathcal{G} - \tilde{\mathcal{G}}\| \leq M \sum_i L_i(q_i) + \|\mathbf{u}\| \sum_i C_i W_i^{-\alpha_i} D_i^{-\beta_i} + \dots$$ (Jin et al., 2022).
• Scaling Laws in Multi-Operator Architectures: In MNO/MONet, to achieve uniform error $\varepsilon$, the count of subnetworks (in parameters, function nets, spatial nets) scales polynomially or logarithmically, depending on the intrinsic dimensions of parameter, function, and space variables, with explicit balancing tradeoffs for depth, width, and sample size (Weihs et al., 29 Oct 2025).
• Distributed MoNO Complexity: The number of experts grows rapidly with intrinsic dimension, but each expert’s size is held to $O(\varepsilon^{-3})$ in typical settings, mitigating—but not eliminating—dimensionality-induced network growth (Kratsios et al., 13 Apr 2024).

5. Representative Examples and Special Cases

The universality framework applies to a wide spectrum of input domains:

• Matrix Inputs: $X = M_{n \times m}(\mathbb{R})$, with linear functionals implemented as trace pairings (Ismailov, 19 Sep 2024).
• $\ell^p$ Sequences: $X = \ell^p$, $X^* \cong \ell^q$; shallow networks parameterized by functional coefficients $a_n$ (Ismailov, 19 Sep 2024).
• Function Spaces:
  • $X = L^p(\mu)$, $X^* = L^q(\mu)$, where neurons correspond to functionals $\int f(x) g(x) d\mu$ (Ismailov, 19 Sep 2024).
  • $X = C(\Omega)$, $X^* = M(\Omega)$ (signed measures) (Ismailov, 19 Sep 2024).
  • Multi-input settings: $X = X_1 \times \cdots \times X_n$ with each $X_i$ a Banach space (Jin et al., 2022).
• Operator Families: Unified multi-operator maps $G[\alpha][u]$ for parametric PDEs, with universality in both strong and weak norms (Weihs et al., 29 Oct 2025).

The encoder-decoder framework also covers DeepONet, BasisONet, frame-based nets, and MIONet as special instances (Gödeke et al., 31 Mar 2025).

6. Limitations and Extensions

Established results are subject to necessary hypotheses:

• Activation Function: $\sigma$ must be continuous and non-polynomial (or Tauber–Wiener); polynomial or affine activations fail universality except in specially designed architectures (Ismailov, 19 Sep 2024, Yu et al., 2021).
• Input Space Conditions: HBEP or local convexity is required; non-separable or non-metrizable spaces fall outside standard theorems (Ismailov, 19 Sep 2024, Gödeke et al., 31 Mar 2025).
• Depth vs. Width: Arbitrary depth can exponentially reduce width (for ReLU NNs and operator settings), but explicit scaling is architecture-dependent (Yu et al., 2021).
• Output Structure: The majority of results are formulated for scalar-valued output or coordinate-wise for $\mathbb{R}^d$; vector-valued extensions are immediate though rate guarantees may differ (Ismailov, 19 Sep 2024).

Extensions encompass:

• Universality for shallow and deep architectures (with different scaling behaviors) (Yu et al., 2021, Ismailov, 19 Sep 2024).
• Compact-set-independent approximations (i.e., a single sequence achieving uniform convergence on all compacts) (Gödeke et al., 31 Mar 2025).
• Quantitative rates in Banach or Sobolev norms, particularly for mixture-of-expert and projection-based architectures (Kratsios et al., 13 Apr 2024, Zappala, 18 Jun 2024).
• Learning hypercomplex-valued operators and generalizations to operators on spaces of measures or distributions (Ismailov, 19 Sep 2024).

7. Implications for Operator Learning and Scientific Computing

Universal approximation theorems for multi-input operators fundamentally support the functional-analytic foundation of neural operator learning. They guarantee that, given suitable architecture and size, one can learn nonlinear mappings from high-dimensional or infinite-dimensional input collections (functions, fields, sequences) to rich output spaces, for arbitrary continuous, and often parametric, solution operators arising in dynamical systems and PDEs (Ismailov, 19 Sep 2024, Jin et al., 2022, Weihs et al., 29 Oct 2025). The detailed error decompositions and explicit constructions provide design principles for practical architectures, guiding the allocation of depth, width, and sensor density, and offering routes to reduce complexity growth in high-dimensional scenarios (Kratsios et al., 13 Apr 2024, Weihs et al., 29 Oct 2025).

This unification places multi-input operator learning on a rigorous theoretical footing, informing the development of efficient, robust operator neural networks for scientific machine learning and beyond.