Universal Approximation Theorem for MoE

• Universal Approximation Theorem for MoE is a foundational result proving that MoE architectures can approximate any continuous function when gating functions form a soft partition and expert classes are dense.
• MoE models include variants like Mixture of Linear Experts, deep multilayer MoE, and MoNO, each addressing challenges of dimensionality, memory scaling, and hierarchical data dependencies.
• Practical design of MoE requires balancing the number of experts, gating function complexity, and model depth to achieve efficient, scalable approximations of functions and operators.

A mixture of experts (MoE) model is a partitioned neural architecture combining multiple “expert” subnetworks, each specialized to a region of the input space, coordinated by a gating function. The universal approximation theorem for MoE asserts that, under general conditions, MoE architectures can approximate a wide range of functions or conditional distributions to arbitrary accuracy. This result is of foundational importance, both theoretically and practically, in informing the design and scope of MoE networks in machine learning and statistics.

1. Precise Formulations of Universal Approximation for MoE

Universal approximation for MoE is formalized in relation to the denseness of the MoE hypothesis class in various function spaces. Consider functions $f:X\to\mathbb{R}^q$, with $X\subset\mathbb{R}^p$ compact. The mean function of a generic MoE takes the form

$$m(x) = \sum_{k=1}^n \text{Gate}_k(x; \alpha)\, h_k(x; \psi_k)$$

where $\text{Gate}_k(x; \alpha)\geq 0$, $\sum_{k} \text{Gate}_k(x; \alpha)=1$ are gating functions (softmax, Gaussian/RBF, or nearest-neighbor), and $h_k$ are expert functions (linear, polynomial, MLP, operator, etc.).

For real-valued outputs ($q=1$), it is established that MoE mean functions are dense in $C(X)$, the space of continuous functions on $X$ with the supremum norm, provided gating functions can generate soft partitions of unity and expert functions are dense in $C(X)$ (Nguyen et al., 2016). For multivariate outputs, the class is dense in $C_q(X)$ under the sum-of-max-norms (Nguyen et al., 2017). For conditional density modeling, Gaussian-gated MoE models can approximate arbitrary continuous conditional densities in marginal KL divergence.

These theorems extend immediately to multi-output settings and conditional densities for multivariate responses. In hierarchical or multilevel data settings, MMoE (Mixture of Experts with Mixed Effects) are shown to be dense in the space of continuous mixed effects models in the weak topology (Fung et al., 2022).

2. Model Classes and Architectural Variants

Several distinct MoE model classes and architectural variants are encompassed by these results:

• Mixture of Linear Experts (MoLE): Experts $h_k(x; \psi_k) = a_k + B_k^Tx$, gating networks are either softmax (affine in $x$) or Gaussian-RBF (Nguyen et al., 2017). For multivariate output spaces, both mean functions and conditional densities satisfy universal approximation properties.
• General MoE with Nonlinear Experts: Experts can be any dense function class (e.g., polynomial, shallow/deep MLPs). If experts are ReLU/MLP subnetworks and gating admits partitions of unity, universal approximation holds over $C(X)$ (Nguyen et al., 2016).
• Deep and Multilayer MoE: When layers of MoE are stacked, the expressive power increases exponentially in the number of layers; $L$ layers with $E$ experts per layer yield models capable of representing $E^L$ compositional pieces, supporting efficient approximation for structured targets (Wang et al., 30 May 2025).
• MoE with Neural Operators (MoNO): For operator learning, mixtures of neural operators are shown to be universal with respect to nonlinear operator targets, under explicit parameter scaling constraints (Kratsios et al., 13 Apr 2024).
• Mixed MoE for Multilevel Data (MMoE): These architectures extend gating and expert mechanisms to handle random effects and nested/hierarchical dependency patterns, ensuring weak-dense approximation capabilities for mixed-effects models (Fung et al., 2022).

3. Theoretical Guarantees: Main Theorems and Proof Strategies

Universal approximation for MoE models relies on two key technical ingredients:

Theorem (as numbered in source)	Statement	Reference
Theorem 2.1 (Univariate Wang)	$\mathcal{M}_1^*(X)$ is dense in $C(X)$ under sup-norm.	(Nguyen et al., 2017)
Theorem 2.2 (Univariate Norets-Pelenis)	For any $\epsilon>0$, there exists an MoE density $f(y|x;\theta)$ (constant experts, Gaussian gates) with $$\int_{X\times Y}\log[g_{Y|X}/f(y|x;\theta)] < \epsilon$$.	(Nguyen et al., 2017)
Theorem 3.1 (Multivariate Extension)	For each marginal conditional density $g_{Y_j|X}$, approximate in KL divergence by a joint MoE, controlling all marginals within $\epsilon$.	(Nguyen et al., 2017)
Theorem 3.2 (Mean-Function Denseness)	Both $\mathcal{M}_q^*(X)$ and $\mathcal{M}_q(X)$ are dense in $C_q(X)$ under $\|\cdot\|_{q,\infty}$.	(Nguyen et al., 2017)
Theorem 2.1 (General MoE UAT)	For MoE mean functions on compact $X$, if the gating class admits soft partitions of unity, and expert class is dense, then the MoE class is dense in $C(X)$.	(Nguyen et al., 2016)
Theorems 3.1/4.3 (Structured MoE)	For functions on smooth manifolds with regular atlases or compositional sparsity, (deep) MoE achieve intrinsic-dimension optimal approximation and exponential region partitioning.	(Wang et al., 30 May 2025)
Thm 4.1 (MoNO)	Any Lipschitz nonlinear operator $G^+: H^{s_1}([0,1]^d)\to H^{s_2}([0,1]^d)$ can be uniformly approximated on Sobolev balls by an MoNO while each expert NO has $O(\epsilon^{-1})$ parameter size.	(Kratsios et al., 13 Apr 2024)
Theorem 4.1/5.1 (MMoE for Multilevel Data)	For multilevel regression, MMoE with appropriate random-effect gating/expert structure is dense in the class of continuous mixed-effects laws in weak topology.	(Fung et al., 2022)

Proofs are structured around constructing a locally accurate collection of experts on a finite cover of $X$, with gating networks implementing soft or hard partitions of unity, and then assembling the mixture with uniform error control. Multivariate and operator-valued generalizations are established via closure properties and coordinate-wise constructions.

4. Approximation Rates, Curse of Dimensionality, and Memory Scalings

While universal approximation guarantees the existence of MoE approximants, explicit error rates and memory/parameter scalings are of central practical significance:

• For MoLEs and shallow MoEs, the number of experts required to achieve sup-norm error $\epsilon$ on $X\subset\mathbb{R}^d$ grows as $O(\epsilon^{-d})$, reflecting the classic curse of dimensionality (Nguyen et al., 2016, Nguyen et al., 2017).
• If the target is Lipschitz or $\alpha$-Hölder smooth, MoE models with PReLU-MLP experts and hard (nearest-prototype) gating achieve error $\epsilon$ with $O(\epsilon^{-1})$ active parameters per forward pass (for $\alpha=1$) (Kratsios et al., 5 Feb 2024), outperforming monolithic MLPs by reducing the active-memory load.
• In operator learning, distributed MoNO architectures can achieve $\epsilon$-accurate operator approximation with each expert operator of $O(\epsilon^{-1})$ size, the number of experts increasing exponentially but only one active per input (Kratsios et al., 13 Apr 2024).
• Deep MoEs with $L$ layers and $E$ experts per layer can represent up to $E^L$ distinct regions, supporting exponential representational efficiency for structured, compositional tasks (Wang et al., 30 May 2025).

Explicit error bound formulas are reported for rates in terms of target smoothness, dimension, and expert/gating complexity (Nguyen et al., 2017, Kratsios et al., 5 Feb 2024, Wang et al., 30 May 2025).

5. Generalizations: Multivariate Outputs, Densities, Hierarchical and Structured Data

Universal approximation for MoE extends to several advanced settings:

• Multiple-output (vector-valued) regression: All theorems remain valid with appropriate modifications to norms and error metrics (Nguyen et al., 2017).
• Arbitrary conditional densities: MoE models with Gaussian-gated linear experts can approximate any collection of continuous, locally log-bounded conditional densities in marginal KL, under mild assumptions (Nguyen et al., 2017).
• Hierarchical/multilevel data: MMoE and its nested extensions (handling random effects across multiple levels) are proven dense in the space of mixed-effects models in the sense of weak convergence of laws, capturing complex dependence and regression structures (Fung et al., 2022).
• Functions on manifolds and compositional structures: For targets supported on low-dimensional manifolds or exhibiting compositional sparsity, shallow and deep MoEs supply optimal or exponential approximation rates relative to intrinsic structure (Wang et al., 30 May 2025).
• Operators on infinite-dimensional spaces: MoNO can universally approximate nonlinear continuous operators between Sobolev-ball-constrained function spaces, provided appropriately scaled per-expert architectures (Kratsios et al., 13 Apr 2024).

6. Practical Implications and Design Considerations

MoE universal approximation theorems have direct ramifications for the architecture and training of scalable neural and statistical models:

• Gating networks must admit partitions of unity; in practice, sufficiently expressive softmax gating or prototype/tree-based hard partitioning are adequate (Nguyen et al., 2016, Kratsios et al., 5 Feb 2024).
• The number and structure of experts are crucial for balancing expressivity, memory usage, and trainability, particularly as target accuracy increases (Kratsios et al., 5 Feb 2024, Kratsios et al., 13 Apr 2024).
• Deep, stacked, or nested MoE variants can factorize the partitioning, greatly reducing total expert count required for highly structured functions or operators (Wang et al., 30 May 2025).
• For high-dimensional or functional input spaces, MoE architectures enable the “distribution” of expressive complexity over multiple compact experts, arming practitioners against practical memory and parallelization bottlenecks (Kratsios et al., 13 Apr 2024).
• In multilevel, hierarchical, or mixed-effects settings relevant in statistics and applied fields, MMoE architectures theoretically recover all continuous models of interest, motivating their use for complex dependence modeling (Fung et al., 2022).

Rigorous error bounds and explicit mapping between approximation error, expert/gate complexity, and scaling laws underpin practical choices in architecture selection and tuning for target applications, ensuring broad applicability across deep learning and nonparametric statistics.