Mixture-of-Experts Model Structure

• Mixture-of-Experts (MoE) models are ensemble architectures that use a dynamic gating network to route inputs to specialized expert sub-networks.
• The gating mechanism employs dense (softmax) or sparse (Top-K) routing to balance computational efficiency and specialization through load-balancing losses.
• Training integrates mutual distillation and EM-based optimization to ensure robust cross-expert feature sharing and prevent expert collapse.

A Mixture-of-Experts (MoE) model is a structured ensemble in which a gating function dynamically routes inputs to a set of specialized sub-networks (“experts”), producing output as a weighted aggregation of expert predictions. The design enables partitioning of complex tasks, targeted specialization, and highly efficient scaling via sparsity, underpinning many of the most performant architectures in deep learning and statistical modeling.

1. Formal Model Structure

An MoE comprises two primary components: a gating network and several expert networks. For an input $x \in \mathbb{R}^d$, the conditional model output is

$h(x) = \sum_{i \in \psi(x)} g_i(x)\,e_i(x),$

where:

• $e_i(\cdot)$ is the output of expert $i$,
• $g_i(x)$ is the gating weight assigning the relative contribution for expert $i$ at input $x$,
• $\psi(x)$ is the set of active experts (all $N$ for dense routing, or the top $K \ll N$ for sparse routing),
• $h(x)$ is evaluated either directly or via a downstream classifier/regressor.

The gating weights $g_i(x)$ are required to be nonnegative and sum to one within $\psi(x)$. For categorical gating, a Top-K operator is used to enforce sparsity by selecting only the $K$ largest-weighted experts per sample.

2. Gating Mechanisms: Dense and Sparse Routing

Two principal paradigms are observed:

• Dense MoE (DMoE):

$g(x) = \mathrm{Softmax}(f(x)),$

where $f(x)$ is a small MLP or linear layer. All experts participate via their (normalized) weights, i.e., $|\psi(x)|=N$.

• Sparse MoE (SMoE):

$$g(x) = \mathrm{TopK}\bigl(\mathrm{Softmax}(f(x))+\epsilon\bigr),$$

with $\epsilon \sim \mathcal{N}(0, 1/E^2)$ sampled independently per input to induce exploration. Only $K$ experts are active for a given $x$.

The Top-K operation zeros out all but the $K$ largest entries of the softmax output. This dynamic “hard” routing sharply reduces per-sample computational cost and induces strong specialization, as the gradient $$\frac{\partial h}{\partial \theta_i} = g_i(x)\,\frac{\partial e_i}{\partial \theta_i}$$ implies little-to-no update for experts with $g_i(x) \approx 0$ (Xie et al., 2024, Han et al., 2024).

To target stability and avoid degenerate “expert-collapse” regimes, auxiliary load-balancing losses are standard, encouraging even utilization of all experts (Han et al., 2024, Shu et al., 17 Nov 2025). For example,

$$\mathcal{L}_{\text{aux}} = \alpha N \sum_{i=1}^{N} f_i P_i,$$

where $f_i$ is the empirical frequency of assigning to expert $i$, and $P_i$ is the average gating weight for expert $i$.

3. Expert Network Architectures

The specific form of each expert $e_i(\cdot)$ is modality-dependent:

• Tabular data: Typically a two-layer MLP, e.g., dimension–16–output.
• NLP: Standard Transformer blocks with FFN replaced by an MoE layer.
• Vision: Small CNNs, ResNet-18 derivatives, or ViT-FFNs as MoE blocks (Han et al., 2024).
• Statistical MoE (regression/classification): GLMs or logistic regressions (Nguyen et al., 2023, Nguyen et al., 2017).

All experts process the full input $x$ in parallel; their outputs are aggregated by the gate for the final mixture output.

4. Mutual Distillation and Specialization

A key challenge in MoE training is “narrow vision,” where each expert only improves on its own dominating subset $DS_i = \{x \mid g_i(x) > 0.5\}$, failing to incorporate feature representations uncovered by other experts and hence under-exploiting shared signals. To mitigate this, mutual distillation regularization is employed: $$L_{\mathrm{KD}}(x) = \frac{1}{K} \sum_{i=1}^{K} \mathrm{mean} \left((e_i(x) - e_{\mathrm{avg}}(x))^2\right), ~~ e_{\mathrm{avg}}(x) = \frac{1}{K} \sum_{i=1}^{K} e_i(x)$$ or, for $K=2$, a symmetric MSE loss between expert outputs.

The total loss is then

$$L(x, y) = L_{\text{task}}(x, y) + \alpha L_{\mathrm{KD}}(x),$$

where $\alpha$ balances task and distillation losses. Appropriate tuning of $\alpha$ prevents collapse to homogeneous experts (over-distillation) while ensuring enough cross-expert feature sharing to improve generalization (Xie et al., 2024).

5. Training and Inference Workflow

MoE models are trained either by direct gradient optimization (most modern deep variants), or, in the classical and statistical setting, by an Expectation-Maximization (EM) procedure:

• E-Step: Compute responsibilities $\tau_{ij}$ (probability sample $i$ is explained by expert $j$).
• M-Step: Update gating and expert parameters via weighted maximization (Fruytier et al., 2024, Makkuva et al., 2018).

The EM procedure is closely connected to mirror descent with a KL-divergence regularizer, admitting stationarity, sublinear, and (under strong convexity) linear convergence guarantees (Fruytier et al., 2024). Recent work demonstrates globally consistent and efficient parameter recovery for nonlinear MoE using tensor decompositions of cross-moment statistics (Makkuva et al., 2018).

At inference, routing is performed identically to training—inputs are run through the gating network to yield active experts, whose predictions are aggregated as in the forward pass of training.

6. Design Principles, Expressivity, and Specializations

Model Capacity and Scaling

Efficiency is governed by two budgets: total model size (memory) and the number of parameters activated per inference (latency). Empirically, overall MoE validation loss $L$ follows a scaling law: $$L \propto N_{\text{total}}^{-0.052}~ s^{+0.018}~ n_{\text{exp}}^{+0.005},$$ where $N_{\text{total}}$ is parameter count, $s = n_{\text{exp}}/n_{\text{topk}}$ is the sparsity ratio, and $n_{\text{exp}}$, $n_{\text{topk}}$ are expert and routed-expert counts, respectively. This relation implies that increasing model width (parameters), decreasing sparsity ratio, and limiting the total number of experts (to maximize per-expert capacity) is optimal, subject to deployment constraints (Liew et al., 13 Jan 2026).

Expressiveness

MoEs achieve a dramatic increase in expressivity. In particular, a deep $L$-layer MoE with $E$ experts per layer can realize up to $E^L$ piecewise function segments via hierarchical composition, modeling exponentially many structured tasks without a corresponding explosion in parameter count (Wang et al., 30 May 2025). Shallow MoEs efficiently approximate functions on low-dimensional manifolds, circumventing the curse of ambient dimensionality.

Shared and Specialized Experts

Practical MoE systems in vision and LLMs often include a “shared” expert—receiving all inputs—to stabilize training and prevent the catastrophic failure of pure specialization (which can occur if routed experts are poorly utilized or fail to capture shared patterns). Empirical analysis (e.g., routing heatmaps) reveals that only the deepest layers in Transformer-based MoEs exhibit high specialization; early layers can remain dense (Han et al., 2024, Shu et al., 17 Nov 2025).

Table: Key Design Choices in MoE Architectures

Component	Typical Choices	Impact
Gating	Linear/MLP + Softmax/Top-K	Specialization, sparsity
Expert Architecture	MLP, CNN, Transformer FFN	Modality adaptation
Routing Sparsity	Dense (all), Top-K (sparse)	Computation/memory efficiency
Distillation	None, symmetric MSE/Cross-Ent	Generalization, vision coverage
Shared Experts	Absent, single per layer	Stability, common knowledge

7. Theoretical Properties and Generalizations

Identifiability of MoE models holds under standard conditions: distinct experts and gating functions, richness in $x$, and non-degenerate parameterizations (Nguyen et al., 2017). Classical MoEs can be trained with EM, with stationary points and sometimes local linear convergence (in high signal-to-noise) (Fruytier et al., 2024). Structural generalizations include:

• Varying-coefficient MoE: experts and gates are indexed by covariates or time, leading to locally-adaptive models with provable consistency, asymptotic normality, and uniform confidence bands (Zhao et al., 5 Jan 2026).
• Bayesian MoE: horseshoe priors over gating coefficients induce adaptive sparsity in expert usage, with particle learning amortizing online inference (Polson et al., 14 Jan 2026).
• Multi-head and hierarchical routing: stacking multi-head gating layers enables modeling of richer inter-expert relationships without increased FLOPs (Huang et al., 2024).

These results—spanning both practical deep learning and rigorous statistical theory—demonstrate the depth and versatility of the Mixture-of-Experts model structure across contemporary machine learning (Xie et al., 2024, Han et al., 2024, Nguyen et al., 2023, Liew et al., 13 Jan 2026, Fruytier et al., 2024, Nguyen et al., 2017, Wang et al., 30 May 2025, Zhao et al., 5 Jan 2026, Huang et al., 2024, Polson et al., 14 Jan 2026).