Mixture-of-Experts (MoE) Routing

• MoE routing is a paradigm that partitions model capacity among specialized expert sub-networks, dynamically selecting experts via top-K and confidence-adaptive mechanisms.
• Advanced techniques integrate geometric, statistical, and retrieval-based methods to improve load balancing, expert specialization, and overall performance.
• Innovative strategies like cross-layer sharing and parameter-efficient fine-tuning enhance scalability, efficiency, and robustness under distribution shifts.

A Mixture-of-Experts (MoE) architecture is a neural network paradigm that partitions model capacity into multiple specialized expert sub-networks, each dynamically controlled by a routing mechanism—the “router”—which determines, for each input (typically each token), which subset of experts is activated. MoE routing is central to the architecture’s efficiency, scalability, specialization, and overall performance. Modern MoE research has advanced routing from simple fixed top-k selectors to sophisticated dynamic, geometric, and retrieval-based schemes, integrating statistical, combinatorial, and geometric principles.

1. Foundational MoE Routing: Top-K and Dynamic Hierarchies

In canonical sparsely-gated MoE layers, the router is a linear or shallow feed-forward module projecting token representations $x\in\mathbb{R}^d$ to $E$ routing logits. A normalized probability vector $P(x)$ is produced via softmax, and a Top-K operator selects the $K$ experts with largest $P_i(x)$: $$P(x) = \operatorname{Softmax}(W_rx), \qquad g^{(Top-K)}_i(x) = \begin{cases} P_i(x) / \sum_{j\in TopK(x)} P_j(x) & \text{if } i\in TopK(x);\ 0 & \text{otherwise}. \end{cases}$$ The MoE output is $\MoE(x)=\sum_{i=1}^E g_i(x) e_i(x)$, where $e_i$ is the $i$-th expert’s output. This top-k framework underpins most MoE models, but has well-known drawbacks—namely, static $K$ can under-utilize hard-to-route tokens and over-allocate easy ones (Huang et al., 2024).

To address these limitations, (Huang et al., 2024) introduced dynamic top-K routing: For each input, the number of active experts $E$0 is the smallest $E$1 satisfying the cumulative sum of $E$2 over the $E$3 largest $E$4 exceeds a threshold $E$5. This "confidence-adaptive" method allows $E$6 experts per token at inference, lowering parameter and FLOP usage (by ~12%) while yielding higher accuracy than fixed-K, notably for hard tasks requiring complex reasoning (BBH accuracy +2.3%).

2. Routing Beyond Layer Locality and Parameterization

Standard MoE architectures restrict the router at each layer to its own expert pool. This design constrains routing diversity and expert expressiveness, as increasing expert width (capacity) necessitates lowering expert count. ReXMoE (Tan et al., 20 Oct 2025) decouples layer-local expert selection by forming "reuse groups"—sets of adjacent layers whose routers share an enlarged common expert pool. During training, Progressive Scaling Routing (PSR) schedules the gradual introduction of more experts to maintain stability. ReXMoE’s expert reuse multiplies routing flexibility with negligible router parameter overhead and boosts perplexity and zero-shot accuracy by up to 1–2% at fixed compute.

Further, Cross-layer sharing has been generalized to the "Omni-Router" (Gu et al., 8 Jul 2025), which utilizes a single shared router across all MoE layers. Here, every layer’s expert weights are independent, but expert selection is driven by a global decision boundary, promoting cross-layer specialization consistency. This approach yields improved load balancing, greater expert specialization, and consistently lower word error rates for ASR models.

3. Geometric and Statistical Routing: Latent Spaces and Manifold Alignment

Multiple works have reframed routing as a geometric or clustering problem. L2R (Yang et al., 29 Jan 2026) projects token states into a low-rank latent space and scores expert selection by saturated, norm-controlled inner products (SIPS), providing explicit Lipschitz constraints on routing sensitivity and breaking high-dimensional angular concentration. Multi-anchor routing further extends expressiveness within the low-rank subspace, and the resulting architecture improves both language and vision model performance compared to standard linear gating.

Latent Prototype Routing (LPR) (Yang, 26 Jun 2025) generalizes this approach by clustering token embeddings in a learned, low-dimensional latent space. Each expert is identified with a prototype vector, initialized hyperspherically, with alignment and diversity losses preventing collapse. LPR achieves near-perfect load balance (Gini ≈ 0.035, Min–Max ratio ≈ 0.7) while incurring no more than 1–2% modeling loss penalty. EMoE (Cheng et al., 17 Jan 2026) replaces pairwise gating with routing via projection on a learned orthonormal eigenbasis, partitioning tokens geometrically along directions of maximal variance, leading to both uniform load and maximally decorrelated expert specialization—without explicit load balancing loss.

Routing Manifold Alignment (RoMA) (Li et al., 10 Nov 2025) targets the discrepancy between learned routing weights and semantic structure. By aligning routing manifolds to those of task embeddings using manifold regularization, it substantially closes the performance gap to oracle routing, improving MMLU and other downstream scores by 7–15% absolute with negligible computational cost.

4. Advanced Routing: Retrieval, Differentiability, and Optimization

To overcome the brittleness of frozen parametric routers under distribution shift, retrieval-augmented routing (kNN-MoE, (Lyu et al., 5 Jan 2026)) introduces a memory store of optimal router assignments constructed offline. At inference, a query retrieves expert configs from similar prior cases, with a confidence-adaptive blend of memory and parametric outputs. This enables reliable performance gains, especially on high-perplexity (distribution-shifted) data, while maintaining inference efficiency.

Dirichlet-Routed MoE (DirMoE, (Vahidi et al., 9 Feb 2026)) fundamentally decomposes routing into two separately-differentiable stages: Bernoulli expert selection (via Gumbel-Sigmoid) and Dirichlet-weighted allocation over activated experts (via implicit Gamma reparameterization). This fully differentiable router allows precise control of both sparsity and mass allocation, yielding improved expert specialization and a tight match to the targeted number of actives.

For constrained assignments (e.g., fixed expert batch capacities), Maximum Score Routing (MaxScore, (Dong et al., 18 Aug 2025)) formulates routing as differentiable minimum-cost maximum-flow (MCMF) over a bipartite graph—with a SoftTopk operator ensuring gradient flow. This method eliminates token dropping and load imbalance at fixed budget, securing both hardware efficiency and higher accuracy than unconstrained or greedy rerouting methods.

5. Routing Robustness, Specialization, Stability, and Load Balancing

MoE routers are also tasked with maximizing expert specialization and balanced utilization. Similarity-preserving routers (SimBal, (Omi et al., 16 Jun 2025)) add an orthonormality-based penalty to the router matrix, ensuring that similar inputs yield similar expert choices. This induces faster convergence (–36%), lowers expert redundancy (min pairwise cosine similarity reduction), and is computationally lighter than conventional balancing losses.

StableMoE (Dai et al., 2022) addresses routing fluctuation—where dynamic learning-to-route oscillates token assignments among experts during training by distilling a teacher router into a frozen, fast student router. This two-stage strategy eliminates drift, speeds up convergence (~15% relative to BASE Layer), and improves downstream language and multilingual translation scores.

MaskMoE (Su et al., 2024) introduces per-token expert visibility masks, set pre-training based on frequency: infrequent tokens are restricted to one expert (preventing underfitting), while frequent tokens may access several (encouraging diversity). This static mask mechanism consistently improves perplexity and accuracy over both pure dynamic and fixed routing, especially as expert count grows.

6. Routing in Specialized and Multimodal Scenarios

Multilingual routing in MoEs (Bandarkar et al., 6 Oct 2025) demonstrates that expert selection is highly language-specific in lower/upper decoder layers but converges in the middle (“semantic hub”) layers. The degree of alignment with English expert usage in middle layers predicts cross-language generalization accuracy ($E$7 for Qwen-30B). Test-time steering that boosts middle-layer “task experts” (identified from English) for other languages yields consistent 1–2% performance gains across 15+ languages; attempts outside those layers or biasing toward explicitly multilingual-specialized experts consistently degrade accuracy.

Combinatorial knowledge sharing is addressed by CartesianMoE (Su et al., 2024), which employs a two-stage routing over orthogonally partitioned "sub-experts," yielding a Cartesian product of expert compositions. This enables group-wise sharing, making MoE outputs robust to routing perturbations and insensitive to minor expert-selection errors.

SMILE (He et al., 2022) addresses routing in distributed hardware environments, splitting global expert routing into efficient bi-level intra-node and inter-node stages, substantially reducing communication overhead (2.47× speedup over Switch Transformer on AWS clusters).

7. Safety, Adaptation, and Fine-Tuning via Routing

Fine-grained alignment of MoE models with safety or new downstream objectives requires precise routing-aware adaptation. RASA (Liang et al., 4 Feb 2026) detects experts disproportionately triggered by adversarial (“jailbreak”) inputs and selectively repairs only these via targeted fine-tuning under frozen routing, then enforces routing consistency between benign and adversarial contexts to prevent adversarial bypass. This mechanism achieves near-perfect robustness without over-refusal or loss of general-domain capability.

Parameter-Efficient Routed Fine-Tuning (Perft) (Liu et al., 4 Aug 2025) extends PEFT approaches by attaching a parallel, routed set of adapters to each FFN/MoE layer, with their own router, loss, and balancing terms. Experimentally, routed adapters significantly outperform vanilla LoRA under active-parameter constraints, with routing central to efficient capacity use.

---

Routing in Mixture-of-Experts architectures has thus developed into a rich area integrating confidence-adaptive, geometric, combinatorial, retrieval-augmented, and fine-tuning-aware methodologies. The design and analysis of routers is central to scaling, specialization, robustness, and efficiency of large neural networks across domains (Huang et al., 2024, Tan et al., 20 Oct 2025, Gu et al., 8 Jul 2025, Yang et al., 29 Jan 2026, Vahidi et al., 9 Feb 2026, Dong et al., 18 Aug 2025, Omi et al., 16 Jun 2025, Dai et al., 2022, Lyu et al., 5 Jan 2026, Su et al., 2024, Li et al., 10 Nov 2025, Bandarkar et al., 6 Oct 2025, Liang et al., 4 Feb 2026, Su et al., 2024). The continuing evolution of routing paradigms underpins advances in LLMs and multimodal sparse networks.