---
title: Universal Approximation Theorem for MoE
url: https://www.emergentmind.com/topics/universal-approximation-theorem-for-moe
type: topic
---

# Universal Approximation Theorem for MoE

A mixture of experts (MoE) model is a partitioned neural architecture combining multiple “expert” subnetworks, each specialized to a region of the input space, coordinated by a gating function. The universal approximation theorem for MoE asserts that, under general conditions, MoE architectures can approximate a wide range of functions or conditional distributions to arbitrary accuracy. This result is of foundational importance, both theoretically and practically, in informing the design and scope of MoE networks in machine learning and statistics.

## 1. Precise Formulations of Universal Approximation for MoE

Universal approximation for MoE is formalized in relation to the denseness of the MoE hypothesis class in various function spaces. Consider functions $f:X\to\mathbb{R}^q$, with $X\subset\mathbb{R}^p$ compact. The mean function of a generic MoE takes the form
\[
m(x) = \sum_{k=1}^n \text{Gate}_k(x; \alpha)\, h_k(x; \psi_k)
\]
where $\text{Gate}_k(x; \alpha)\geq 0$, $\sum_{k} \text{Gate}_k(x; \alpha)=1$ are gating functions (softmax, Gaussian/RBF, or nearest-neighbor), and $h_k$ are expert functions (linear, polynomial, MLP, operator, etc.).

For real-valued outputs ($q=1$), it is established that MoE mean functions are dense in $C(X)$, the space of continuous functions on $X$ with the supremum norm, provided gating functions can generate soft partitions of unity and expert functions are dense in $C(X)$ [1602.03683]. For multivariate outputs, the class is dense in $C_q(X)$ under the sum-of-max-norms [1704.00946]. For conditional density modeling, Gaussian-gated MoE models can approximate arbitrary continuous conditional densities in marginal KL divergence.

These theorems extend immediately to multi-output settings and conditional densities for multivariate responses. In hierarchical or multilevel data settings, MMoE (Mixture of Experts with Mixed Effects) are shown to be dense in the space of continuous mixed effects models in the weak topology [2209.15207].

## 2. Model Classes and Architectural Variants

Several distinct MoE model classes and architectural variants are encompassed by these results:

- **Mixture of Linear Experts (MoLE):** Experts $h_k(x; \psi_k) = a_k + B_k^Tx$, gating networks are either softmax (affine in $x$) or Gaussian-RBF [1704.00946]. For multivariate output spaces, both mean functions and conditional densities satisfy universal approximation properties.
- **General MoE with Nonlinear Experts:** Experts can be any dense function class (e.g., polynomial, shallow/deep MLPs). If experts are ReLU/MLP subnetworks and gating admits partitions of unity, universal approximation holds over $C(X)$ [1602.03683].
- **Deep and Multilayer MoE:** When layers of MoE are stacked, the expressive power increases exponentially in the number of layers; $L$ layers with $E$ experts per layer yield models capable of representing $E^L$ compositional pieces, supporting efficient approximation for structured targets [2505.24205].
- **MoE with Neural Operators (MoNO):** For operator learning, mixtures of neural operators are shown to be universal with respect to nonlinear operator targets, under explicit parameter scaling constraints [2404.09101].
- **Mixed MoE for Multilevel Data (MMoE):** These architectures extend gating and expert mechanisms to handle random effects and nested/hierarchical dependency patterns, ensuring weak-dense approximation capabilities for mixed-effects models [2209.15207].

## 3. Theoretical Guarantees: Main Theorems and Proof Strategies

Universal approximation for MoE models relies on two key technical ingredients:

| Theorem (as numbered in source)             | Statement                                                                                                                                                                                          | Reference     |
|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
| Theorem 2.1 (Univariate Wang)               | $\mathcal{M}_1^*(X)$ is dense in $C(X)$ under sup-norm.                                                                                                                                            | [1704.00946]  |
| Theorem 2.2 (Univariate Norets-Pelenis)     | For any $\epsilon>0$, there exists an MoE density $f(y|x;\theta)$ (constant experts, Gaussian gates) with $\int_{X\times Y}\log[g_{Y|X}/f(y|x;\theta)] < \epsilon$.                               | [1704.00946]  |
| Theorem 3.1 (Multivariate Extension)        | For each marginal conditional density $g_{Y_j|X}$, approximate in KL divergence by a joint MoE, controlling all marginals within $\epsilon$.                                                      | [1704.00946]  |
| Theorem 3.2 (Mean-Function Denseness)       | Both $\mathcal{M}_q^*(X)$ and $\mathcal{M}_q(X)$ are dense in $C_q(X)$ under $\|\cdot\|_{q,\infty}$.                                                                                              | [1704.00946]  |
| Theorem 2.1 (General MoE UAT)               | For MoE mean functions on compact $X$, if the gating class admits soft partitions of unity, and expert class is dense, then the MoE class is dense in $C(X)$.                                     | [1602.03683]  |
| Theorems 3.1/4.3 (Structured MoE)           | For functions on smooth manifolds with regular atlases or compositional sparsity, (deep) MoE achieve intrinsic-dimension optimal approximation and exponential region partitioning.                 | [2505.24205]  |
| Thm 4.1 (MoNO)                              | Any Lipschitz nonlinear operator $G^+: H^{s_1}([0,1]^d)\to H^{s_2}([0,1]^d)$ can be uniformly approximated on Sobolev balls by an MoNO while each expert NO has $O(\epsilon^{-1})$ parameter size. | [2404.09101]  |
| Theorem 4.1/5.1 (MMoE for Multilevel Data)  | For multilevel regression, MMoE with appropriate random-effect gating/expert structure is dense in the class of continuous mixed-effects laws in weak topology.                                   | [2209.15207]  |

Proofs are structured around constructing a locally accurate collection of experts on a finite cover of $X$, with gating networks implementing soft or hard partitions of unity, and then assembling the mixture with uniform error control. Multivariate and operator-valued generalizations are established via closure properties and coordinate-wise constructions.

## 4. Approximation Rates, Curse of Dimensionality, and Memory Scalings

While universal approximation guarantees the existence of MoE approximants, explicit error rates and memory/parameter scalings are of central practical significance:

- For MoLEs and shallow MoEs, the number of experts required to achieve sup-norm error $\epsilon$ on $X\subset\mathbb{R}^d$ grows as $O(\epsilon^{-d})$, reflecting the classic curse of dimensionality [1602.03683], [1704.00946].
- If the target is Lipschitz or $\alpha$-Hölder smooth, MoE models with PReLU-MLP experts and hard (nearest-prototype) gating achieve error $\epsilon$ with $O(\epsilon^{-1})$ active parameters per forward pass (for $\alpha=1$) [2402.03460], outperforming monolithic MLPs by reducing the active-memory load.
- In operator learning, distributed MoNO architectures can achieve $\epsilon$-accurate operator approximation with each expert operator of $O(\epsilon^{-1})$ size, the number of experts increasing exponentially but only one active per input [2404.09101].
- Deep MoEs with $L$ layers and $E$ experts per layer can represent up to $E^L$ distinct regions, supporting exponential representational efficiency for structured, compositional tasks [2505.24205].

Explicit error bound formulas are reported for rates in terms of target smoothness, dimension, and expert/gating complexity [1704.00946], [2402.03460], [2505.24205].

## 5. Generalizations: Multivariate Outputs, Densities, Hierarchical and Structured Data

Universal approximation for MoE extends to several advanced settings:

- **Multiple-output (vector-valued) regression**: All theorems remain valid with appropriate modifications to norms and error metrics [1704.00946].
- **Arbitrary conditional densities**: MoE models with Gaussian-gated linear experts can approximate any collection of continuous, locally log-bounded conditional densities in marginal KL, under mild assumptions [1704.00946].
- **Hierarchical/multilevel data**: MMoE and its nested extensions (handling random effects across multiple levels) are proven dense in the space of mixed-effects models in the sense of weak convergence of laws, capturing complex dependence and regression structures [2209.15207].
- **Functions on manifolds and compositional structures**: For targets supported on low-dimensional manifolds or exhibiting compositional sparsity, shallow and deep MoEs supply optimal or exponential approximation rates relative to intrinsic structure [2505.24205].
- **Operators on infinite-dimensional spaces**: MoNO can universally approximate nonlinear continuous operators between Sobolev-ball-constrained function spaces, provided appropriately scaled per-expert architectures [2404.09101].

## 6. Practical Implications and Design Considerations

MoE universal approximation theorems have direct ramifications for the architecture and training of scalable neural and statistical models:

- Gating networks must admit partitions of unity; in practice, sufficiently expressive softmax gating or prototype/tree-based hard partitioning are adequate [1602.03683], [2402.03460].
- The number and structure of experts are crucial for balancing expressivity, memory usage, and trainability, particularly as target accuracy increases [2402.03460], [2404.09101].
- Deep, stacked, or nested MoE variants can factorize the partitioning, greatly reducing total expert count required for highly structured functions or operators [2505.24205].
- For high-dimensional or functional input spaces, MoE architectures enable the “distribution” of expressive complexity over multiple compact experts, arming practitioners against practical memory and parallelization bottlenecks [2404.09101].
- In multilevel, hierarchical, or mixed-effects settings relevant in statistics and applied fields, MMoE architectures theoretically recover all continuous models of interest, motivating their use for complex dependence modeling [2209.15207].

Rigorous error bounds and explicit mapping between approximation error, expert/gate complexity, and scaling laws underpin practical choices in architecture selection and tuning for target applications, ensuring broad applicability across deep learning and nonparametric statistics.

Source: https://www.emergentmind.com/topics/universal-approximation-theorem-for-moe