Mixture-of-Experts Language Models

• MoE language models are conditional computation frameworks that sparsely activate expert subnetworks via top-k gating, enabling vast parameter scaling with efficient per-token processing.
• They decouple total model parameters from active computation, allowing significant compute savings and improved domain-specific performance compared to dense models.
• Advanced techniques such as load balancing, expert divergence, and shared-expert design stabilize training and enhance inter-expert diversity, driving superior generalization.

A Mixture-of-Experts (MoE) LLM is a LLM in which a routing mechanism activates only a sparse subset of expert subnetworks for each input, usually by replacing Transformer feed-forward sublayers with expert pools and top-$k$ gating. The defining principle is conditional computation: total parameter count can grow dramatically, while per-token computation depends mainly on the number of activated experts rather than the total number of experts. In LLMs, MoE is most commonly inserted in place of the Transformer feed-forward network because FFNs dominate parameter count and are more sparse or domain-specific than self-attention (Cai et al., 2024). Recent reviews describe MoE as a modular alternative to the dense “one-size-fits-all” network, with selective activation, expert routing, hierarchical variants, multimodal and multitask extensions, and a strong dependence on routing quality, expert diversity, calibration, and deployment efficiency (Zhang et al., 15 Jul 2025).

1. Architectural form and routing semantics

The canonical MoE layer computes

$y=\sum_{i=1}^{N} g_i(x)E_i(x),$

where $E_i$ is expert $i$ and only $k \ll N$ experts receive nonzero mass for a given input (Zhang et al., 15 Jul 2025). In sparse LLM MoE, the router produces scores, applies a Top-$K$ operator, and then normalizes the surviving scores with a softmax. The survey literature distinguishes token-choice routing, in which each token selects its experts, from expert-choice routing, in which each expert selects a budgeted set of tokens; it also distinguishes sparse, dense, hierarchical, heterogeneous, and adaptive MoE variants (Zhang et al., 15 Jul 2025).

In autoregressive language modeling, a concrete and influential instantiation replaces every other feed-forward network in a Transformer decoder with an MoE layer while keeping the rest of the model dense (Artetxe et al., 2021). In that configuration, each MoE layer contains $E=512$ experts, uses top-2 expert selection, assigns each expert capacity $\frac{C\cdot B}{E}$ with $C=2$, and overflows excess tokens through the residual connection if an expert is full (Artetxe et al., 2021). The same work uses pre-normalization Transformer blocks, GELU activations, dense attention only, and sinusoidal positional embeddings following Shortformer, making the MoE substitution highly localized to the FFN pathway rather than a full architectural redesign (Artetxe et al., 2021).

Later open MoE platforms and model families show that this core pattern is not unique. FLAME-MoE, for example, uses a decoder-only Transformer in which all FFN sublayers except the first are replaced with MoE layers, with 64 experts per layer, top-8 routing, and 2 shared experts, so that each token activates 6 routed experts plus 2 shared experts (Kang et al., 26 May 2025). This shared-expert design formalizes a recurring architectural compromise: routed experts provide specialization, while shared experts provide a stable common computation path.

2. Scaling laws, compute efficiency, and design constraints

The central attraction of MoE is that parameter count and active computation are only weakly coupled. In the large-scale empirical study of autoregressive MoE language modeling, the FLOP accounting adds an extra $32Tlh^2$ term to dense-model compute for top-2 MoE routing, and this additional cost is independent of the number of experts (Artetxe et al., 2021). The implication is direct: expert count can increase total parameters dramatically without proportional growth in per-token FLOPs, because only a small fraction of experts is active for each token.

Empirically, this scaling regime is favorable but not uniform. Across in-domain and out-of-domain language modeling, zero-shot and few-shot priming, and full supervised fine-tuning, MoE models are substantially more compute efficient than dense models except in fine-tuning (Artetxe et al., 2021). At more modest training budgets, MoEs can match dense models using $y=\sum_{i=1}^{N} g_i(x)E_i(x),$04 times less compute; the gap narrows at scale, but a 1.1T-parameter MoE still outperforms a compute-equivalent 6.7B dense model, and the advantage is largest for in-domain language modeling, smaller for The Pile, and highly task-dependent (Artetxe et al., 2021). This establishes the basic encyclopedia-level trade-off: MoE offers superior parameter scaling and often better compute efficiency, but the quality margin depends on domain and evaluation regime.

Subsequent work argues that total parameters and active parameters alone are not sufficient to describe an optimal MoE architecture. A more principled design account identifies performance as being governed primarily by total non-embedding parameters,

$y=\sum_{i=1}^{N} g_i(x)E_i(x),$1

and expert sparsity,

$y=\sum_{i=1}^{N} g_i(x)E_i(x),$2

with the fitted law

$y=\sum_{i=1}^{N} g_i(x)E_i(x),$3

This analysis shows that lower sparsity is better, but also that $y=\sum_{i=1}^{N} g_i(x)E_i(x),$4 and $y=\sum_{i=1}^{N} g_i(x)E_i(x),$5 do not cancel cleanly: larger raw expert counts slightly penalize performance because, under a memory constraint, they force reductions in depth and width of the dense backbone (Liew et al., 13 Jan 2026). The resulting design rule is to maximize $y=\sum_{i=1}^{N} g_i(x)E_i(x),$6, maximize $y=\sum_{i=1}^{N} g_i(x)E_i(x),$7, and minimize $y=\sum_{i=1}^{N} g_i(x)E_i(x),$8 subject to memory and inference constraints, while keeping granularity $y=\sum_{i=1}^{N} g_i(x)E_i(x),$9 in a reasonable range around $E_i$0–$E_i$1 and the width-to-depth ratio $E_i$2 roughly $E_i$3–$E_i$4 (Liew et al., 13 Jan 2026).

An open compute-scaling study reaches a compatible systems-level conclusion. FLAME-MoE releases seven decoder-only MoE models from 38M active / 100M total parameters up to 1.7B active / 10.3B total parameters, and reports gains of 1.8 to 3.4 points in average accuracy over dense baselines trained with identical FLOPs (Kang et al., 26 May 2025). Its compute-optimal analysis uses IsoFLOP profiles and parametric loss fitting, and finds that validation loss correlates strongly with downstream performance, with Spearman correlation 0.89 on HellaSwag (Kang et al., 26 May 2025). Taken together, these results indicate that MoE scaling is neither arbitrary nor exhausted by headline parameter counts: expert count, top-$E_i$5, dense-core geometry, and training budget all materially shape the effective frontier.

3. Optimization, load balancing, and expert collapse

MoE training adds optimization problems that are weak or absent in dense models. The first is routing imbalance: without regularization, routers can collapse onto a small subset of experts. Standard practice therefore adds a load-balancing term, often alongside a router stabilization penalty. FLAME-MoE trains with

$E_i$6

using $E_i$7 and $E_i$8, where the z-loss penalizes large router logits for stability (Kang et al., 26 May 2025). Earlier large-scale training also reports a specific stabilization trick for experts: because expert parameters effectively see a smaller per-expert batch, gradients are rescaled by $E_i$9 (Artetxe et al., 2021).

Load balance, however, is not the only failure mode. A distinct line of work argues that many MoE LLMs suffer from homogeneous representation even when routing is not severely imbalanced: experts learn nearly the same parameters and the same subspace, with similarities reported up to 99% in a well-performed MoE model (Liu et al., 2023). OMoE addresses this by adapting Orthogonal Weight Modification to MoE fine-tuning, alternating between a standard optimizer step and an orthogonal projection step so that each expert is updated in a direction orthogonal to the subspace spanned by other experts. The method improves GLUE, SuperGLUE, SQuAD v1.1, and CoNLL-2003 relative to AdamW-based MoE, while adding about 1.38$i$0 memory and 1.82 GMacs of compute overhead; it also reports that more diversity does not always mean better performance, and that $i$1 is a good skipping step in its alternating schedule (Liu et al., 2023).

A complementary pretraining remedy is Expert Divergence Learning. Rather than only balancing expert usage globally, EDL uses domain labels already present in the pretraining corpus and maximizes Jensen–Shannon divergence between domain-average routing distributions, yielding a final objective

$i$2

with $i$3 and $i$4 (Li et al., 10 Feb 2026). This method lowers language modeling loss across 3B-A0.3B, 8B-A0.8B, and 15B-A1.5B models, improves downstream averages most clearly under the 49-class domain-label scheme, and reports negligible training and inference overhead because the auxiliary term operates only on router probabilities (Li et al., 10 Feb 2026).

Open training traces further clarify the temporal dynamics of these mechanisms. In FLAME-MoE, expert specialization increases during training, co-activation matrices remain sparse, and routing stabilizes early: most layers exceed 70% agreement with final top-$i$5 choices by the midpoint of training (Kang et al., 26 May 2025). This suggests that routing policies often solidify well before convergence of all model weights, a point with direct implications for curriculum design, router freezing, and interpretability.

4. Generalization behavior relative to dense models

The strongest broad empirical comparison between MoE and dense LLMs remains the large autoregressive study spanning perplexity, priming, and fine-tuning (Artetxe et al., 2021). On language modeling, MoE models consistently outperform matched-compute dense counterparts across validation and most Pile subsets, but the magnitude of the gain depends sharply on domain. In-domain validation is the most favorable regime, where MoEs can often match dense models trained with 8–16$i$6 more compute; on The Pile, the advantage is smaller, around 2–4$i$7 compute savings, and on distant domains such as DM Mathematics the advantage nearly disappears (Artetxe et al., 2021).

Zero-shot and few-shot behavior is similarly nonuniform. On six prompt-based tasks including ReCoRD, HellaSwag, PIQA, WinoGrande, StoryCloze, and OpenBookQA, MoEs consistently beat dense models at every matched compute point, but the improvement is task-dependent: the gap is large on HellaSwag and PIQA, smaller on ReCoRD and WinoGrande (Artetxe et al., 2021). Few-shot results preserve the same ordering, yet the increment from zero-shot to few-shot is smaller for MoEs than for dense models, suggesting that sparse and dense models may encode and exploit in-context task information differently (Artetxe et al., 2021).

Full supervised fine-tuning complicates the picture. Dense models benefit strongly across the fine-tuned task suite, whereas MoE models improve on StoryCloze, BoolQ, SST-2, MNLI, and some OpenBookQA settings but can become worse than their own zero-shot performance on HellaSwag, PIQA, and WinoGrande (Artetxe et al., 2021). The general conclusion is not that MoE is uniformly better or worse, but that MoE and dense models generalize differently. The cited study explicitly treats them as complementary model families whose differences remain worthy of further study (Artetxe et al., 2021).

A separate open benchmark family reaches a compatible conclusion under matched-FLOP training. FLAME-MoE improves average accuracy over dense baselines across six tasks—ARC-E, ARC-C, OBQA, HellaSwag, PIQA, and WinoGrande—by 1.8 to 3.4 points, and can match or exceed dense models trained with about $i$8 the FLOPs (Kang et al., 26 May 2025). The convergence between these independent results strengthens the encyclopedic claim that MoE’s main empirical signature is not only larger nominal capacity, but a distinct speed–quality frontier whose benefits vary by task and adaptation protocol.

5. Specialization, multilingual structure, and expert collaboration

A common simplification treats experts as isolated specialists. Recent analyses instead show that expert behavior is structured across language, layer, and co-activation pattern. In multilingual MoE models, routing is family-structured rather than arbitrary: routing similarity measured by Jensen–Shannon divergence aligns with linguistic relatedness, shared experts are used differently across high-resource and low-resource languages, and language-exclusive experts follow a U-shaped pattern over depth, being most prominent in early and late layers and least prominent in the middle (Chen et al., 20 Jan 2026). Causal masking shows that early layers support language-specific understanding, late layers support language identity control and generation fidelity, and middle layers behave as language-agnostic capacity hubs (Chen et al., 20 Jan 2026). Based on this structure, routing-guided steering of middle-layer routing toward shared experts associated with dominant languages improves PolyMath, with average gains of +1.9% under English steering and +0.7% under Chinese steering, while steering early or late layers reduces performance (Chen et al., 20 Jan 2026).

The collaboration perspective extends beyond multilingual routing. A study of hidden collaboration in MoE LLMs introduces hierarchical sparse dictionary learning to mine latent co-activation motifs from expert activations and argues that experts often form collaborative groups, sometimes across layers, rather than functioning as purely independent units (Tang et al., 16 Apr 2025). On MMLU-pro, the learned dictionary atoms correspond to semantically meaningful structures, from broad categories such as “date, symbol, and mathematical calculation” to finer patterns such as “mathematical calculation” and “key verbs,” and 60% of the patterns discovered by HSDL correspond to the top 10% most frequent expert combinations found by exhaustive search (Tang et al., 16 Apr 2025). The same work derives Contribution-Aware Expert Pruning from these collaboration patterns and reports average score 0.612 after pruning 25% of experts, outperforming SEER-MoE and GEM by about 2.5% on average (Tang et al., 16 Apr 2025).

These analyses support a more precise understanding of specialization. Experts do not merely partition tokens; they participate in higher-order computational organization. A plausible implication is that MoE quality depends as much on the geometry of expert interaction as on nominal expert count or routing sparsity.

6. Compression, adaptation, and specialized MoE variants

Because total parameter count remains a memory burden even when active compute is sparse, many recent variants treat MoE itself as the object of optimization. Expert Tying shares expert FFN weights across consecutive layers while keeping attention, routers, and normalization gains layer-specific. With tying group size $i$9, this reduces total parameters by roughly 43%–52% on scaled-down OLMoE, DeepSeekMoE, and Qwen3-MoE, and on full OLMoE-1B-7B reduces total parameters from 7.12B to 3.50B while keeping accuracy essentially unchanged at 47.5 versus 47.4; throughput gains reach up to 23.7% (Jaggi, 15 Jun 2026). Training-free compression targets a different bottleneck: Mixture Compressor combines mixed-precision expert quantization and online dynamic pruning, achieving 76.6% model compression at 2.54 bits with only a 3.8% average accuracy loss, and then reducing activated parameters by 15% with less than 0.6% additional drop (Huang et al., 2024). MoSE modifies the experts themselves, making each expert slimmable and executable at variable widths; in GPT-style pretraining on OpenWebText, it matches or improves standard MoE at full width and shifts the accuracy–FLOPs Pareto frontier downward, especially when a lightweight test-time procedure learns how to map router probabilities to expert widths under a fixed budget (Tastan et al., 5 Feb 2026).

A second line of work treats MoE as an adaptation scaffold. ToMoE converts dense decoder-only LLMs into sparse MoE-style models after training, leaving original weights frozen and learning routing and structural masks that expose latent expert-like substructures in MLPs and MHA; it uses top-1 routing for MLP experts, default expert count 8, and consistently outperforms structural pruning baselines across Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5 even without fine-tuning dense weights (Gao et al., 25 Jan 2025). MoE-LPR applies upcycling to multilingual post-pretraining, freezing original parameters, adding new experts, and then using a review stage with Language Priors Routing; on Qwen1.5-1.8B, it reaches 45.07 average on expanded languages while retaining 96.6% of original-language performance, using original-language replay data totaling less than 1% of the post-pretraining corpus (Zhou et al., 2024). Mix-MoE specializes multilingual translation by splitting expert pools into LM Experts and MT Experts and augmenting routers with FFT-derived features, while CL-MoE uses LoRA adapters as experts in continual multimodal visual question answering and reports $k \ll N$0 and $k \ll N$1 when combining its routing and momentum mechanisms (Li et al., 23 May 2026, Huai et al., 1 Mar 2025).

Multimodal and deployment-oriented variants expose additional pressures. EvoMoE argues that existing sparse multimodal LLMs suffer from expert uniformity and router rigidity, replacing replicated-expert initialization with expert evolution and static linear gating with a Dynamic Token-aware Router; in its ablations on Qwen-1.8B, the dense baseline scores 65.4 average, the MoE-tuning baseline 65.5, and the full EvoMoE 66.5, with top-1 routing outperforming top-2 in that setting (Jing et al., 28 May 2025). WDMoE distributes MoE inference across a base-station edge server and mobile devices, keeping attention and routing at the base station and placing experts on devices; its latency-aware expert selection policy reduces end-to-end latency by 16.2% to 37.54% across evaluated tasks while outperforming Llama 2 70B on all evaluated benchmarks in the reported setup (Xue et al., 2024). Bayesian-MoE addresses a different deployment concern—reliability—by applying a structured Laplace approximation to the second linear layer of each expert in fine-tuned Qwen1.5-MoE and DeepSeek-MoE, improving both expected calibration error and negative log-likelihood without modifying the original training procedure (Dialameh et al., 12 Nov 2025).

Across these variants, the underlying pattern is consistent. MoE LLMs are no longer defined only by sparse top-$k \ll N$2 routing inside a Transformer FFN. They have become a broader architectural family in which expert identity, expert width, expert sharing, expert calibration, expert quantization, expert placement, and expert role specialization are all independently manipulated to trade off capability, memory, latency, and transfer. The field’s central unresolved questions therefore concern not only how many experts to add, but how to make them diverse, how to route stably, how to preserve useful shared structure, and how to align sparse computation with real deployment constraints.