Dynamic Mixture-of-Experts Router

• Dynamic Mixture-of-Experts Routers are mechanisms that route tokens to a subset of experts based on input features and contextual signals, enabling adaptive and sparse computation.
• They employ techniques such as attention-derived importance, thresholding, and differentiable sparse projections to adjust both the identity and the number of activated experts.
• Empirical studies indicate that dynamic routers improve accuracy, reduce latency, and balance loads across tasks, while addressing challenges like load-imbalance and scalability.

Searching arXiv for the cited router papers to ground the article in current literature. A Dynamic Mixture-of-Experts Router is a routing mechanism in a Mixture-of-Experts (MoE) model that determines, as a function of the current input and sometimes additional context, which experts to activate and how strongly to weight them. In the literature summarized here, the term covers several distinct but related designs: routers that condition on both local and global context in speech recognition (You et al., 2021); routers that vary the number of experts per token using attention-derived importance (Aghdam et al., 2024) or cumulative-probability thresholds (Huang et al., 2024); routers that replace discrete TopK with differentiable sparse projections and token-dependent sparsity factors (Zhuang et al., 30 Sep 2025); routers that operate over multiple sub-routers (Zhang et al., 30 Mar 2025) or attention-derived query–key matching (Ran et al., 31 Aug 2025); routers decoupled from the MoE backbone for system-aware pre-gating (Cai et al., 2024); and routers whose dynamics are analyzed as adaptive softmax systems exhibiting bifurcation and load-imbalance regimes (Kiselev, 27 May 2026). Across these formulations, the central objective is adaptive expert allocation: routing should reflect token complexity, task structure, domain or accent context, temporal regime, or system constraints while preserving sparse computation.

1. Formal role of the router in sparse MoE models

In transformer-based MoE models, a standard dense FFN block is replaced by a set of experts, and a router maps a token representation to expert scores. A standard formulation uses router logits

$\text{router\_logits} = W_r \cdot X$

followed by softmax probabilities

$$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$

after which a sparse subset of experts is selected (Aghdam et al., 2024). In the fixed-top-$K$ paradigm, all tokens receive the same number of experts, as in top-1 or top-2 routing (Aghdam et al., 2024). The main limitation identified across recent work is that fixed-$K$ routing neglects token- and context-dependent variability in required computation (Aghdam et al., 2024, Huang et al., 2024).

Several dynamic routers retain the same basic decomposition—score computation, expert selection, sparse expert execution—but alter one or more of its components. Some modify the router input. In SpeechMoE2, the router receives a concatenation of frame-level grapheme embedding, utterance-level accent embedding, utterance-level domain embedding, and the previous-layer output: $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$ with top-1 routing and probability-gated expert output (You et al., 2021). Others modify the selection rule itself. DA-MoE computes token importance from self-attention and sets the number of experts per token as

$$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$

after which routing remains top-$K_k$ over standard softmax probabilities (Aghdam et al., 2024). In "Harder Tasks Need More Experts," selection is Top-$P$-style: experts are sorted by probability and the smallest prefix whose cumulative mass exceeds a threshold $p$ is selected (Huang et al., 2024).

A separate line of work reframes routing itself as a probability distribution over expert subsets. ProbMoE defines Bernoulli parameters $p_i = \sigma(r_i)$, constructs a normalized distribution over subsets $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$0 under exact-$$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$1 or range-constrained cardinality, samples $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$2 in the forward pass, and uses exact marginal probabilities as a differentiable surrogate in the backward pass (Zhao et al., 1 Jun 2026). This replaces deterministic top-$$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$3 with probabilistic inference over subset space.

This suggests a useful organizing distinction between fixed-cardinality routers, dynamic-cardinality routers, context-augmented routers, multi-router systems, and probabilistic subset routers. The underlying MoE abstraction is shared, but the mechanism that decides which experts and how many can vary substantially.

2. Context-aware routing signals

A recurring design question is what information the router should see. SpeechMoE2 is an explicit example of enriching routing inputs with context unavailable to local frame-level features alone. Its motivation is that routers that only observe local, frame-level information fail to specialize experts according to higher-level factors such as domain and accent, which harms robustness in multi-domain and multi-accent conditions (You et al., 2021). The model therefore augments local grapheme embedding $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$4 with utterance-level accent and domain embeddings $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$5, obtained by average-pooling frame-level grapheme embeddings and applying learned projections: $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$6 This yields a router that is dynamic with respect to current acoustic context, local graphemic content, and global accent/domain priors (You et al., 2021).

In DA-MoE, the contextual signal comes instead from the transformer's own self-attention. Token importance is computed from the last attention layer as the average across heads of each token’s strongest attention weight: $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$7 This importance is not used to select which experts—softmax routing still does that—but to determine how many experts should be activated for that token (Aghdam et al., 2024). The resulting computation budget becomes token-dependent.

Read-ME makes context dependence autoregressive and sequence-level. It replaces per-layer routers with a single transformer router $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$8 that observes the token prefix $$\text{router\_probs}_i = \frac{\exp(\text{router\_logits}_i)}{\sum_{j=1}^{E} \exp(\text{router\_logits}_j)},$$9 and emits expert scores reused across all MoE layers: $K$0 This design is justified by empirical redundancy among adjacent per-layer routers and enables pre-computed routing, expert-aware batching, and Belady-style caching (Cai et al., 2024). The router is still dynamic per token, but its decisions are decoupled from layer-local hidden states.

Dynamic TMoE introduces temporal memory as routing context in non-stationary time-series forecasting. Its router maintains a recurrent hidden state

$K$1

and computes sparse top-$K$2 dispatch from $K$3 (Zhu et al., 20 May 2026). It also fuses $K$4 with a retrieved anomaly-state prototype from an external repository: $K$5 The result is explicitly history-aware routing, designed to stabilize expert selection under drift (Zhu et al., 20 May 2026).

These designs share a common premise: routing quality depends critically on what constitutes the router state. Local token embeddings, attention-derived token importance, utterance-level embeddings, autoregressive prefix summaries, recurrent hidden states, and anomaly memories each define different notions of context, and the chosen notion shapes expert specialization.

3. Dynamic expert cardinality

A defining property of many recent dynamic routers is that they do not allocate a fixed number of experts to every token. DA-MoE is the most direct formulation. It maps token importance to cardinality through

$K$6

then selects the top-$K$7 experts under the router probabilities (Aghdam et al., 2024). This creates a token-dependent compute pattern in which highly informative tokens receive more experts and less informative tokens receive fewer.

"Harder Tasks Need More Experts" implements dynamic selection through a cumulative-probability threshold. Given router probabilities $K$8, experts are sorted descending, and the selected set is the smallest prefix whose cumulative probability exceeds a threshold $K$9 (Huang et al., 2024). The paper explicitly interprets flatter distributions as lower confidence and sharper distributions as higher confidence. Training uses a fixed threshold $K$0, and inference additionally caps activation at two experts, yielding an average of 1.76 activated experts across evaluated tasks versus exactly two for Top-2 routing (Huang et al., 2024). The same study reports that harder reasoning tasks such as BBH activate more experts on average than easier tasks such as PIQA or HellaSwag (Huang et al., 2024).

ProbMoE generalizes this idea into a full probabilistic model over subsets. In Exact-$K$1, support is restricted to subsets of size $K$2; in Dynamic-$K$3, support is restricted to a range

$K$4

The partition function becomes

$K$5

and the induced probability over cardinalities is

$K$6

Thus the number of active experts is itself inferred from router logits through exact dynamic programming over feasible subset sizes (Zhao et al., 1 Jun 2026). This is dynamic-$K$7 in a stronger sense than heuristic thresholding: cardinality is an explicit latent variable in the routing distribution.

LD-MoLE removes fixed $K$8 altogether by replacing TopK with Sparsegen. Given logits $K$9 and learned sparsity factor $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$0, routing probabilities are the solution of a constrained quadratic program over the simplex: $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$1 The closed-form solution

$$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$2

induces a support size $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$3 that depends jointly on token-specific logits and token-specific $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$4, thereby producing token-dependent and layer-wise expert allocation without a discrete TopK operator (Zhuang et al., 30 Sep 2025).

A plausible implication is that dynamic-cardinality routers can be ordered by how explicitly they model cardinality: heuristic thresholding (Huang et al., 2024), deterministic importance-to-$$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$5 mappings (Aghdam et al., 2024), support-size control through sparse simplex projections (Zhuang et al., 30 Sep 2025), and subset-space inference (Zhao et al., 1 Jun 2026) represent progressively stronger commitments to cardinality as a first-class routing variable.

4. Differentiability, sparsity control, and balancing

Sparse routing creates a tension between efficient hard selection and stable optimization. Several router designs address this by changing how gradients flow through selection or by introducing auxiliary objectives.

SpeechMoE2 uses top-1 routing but retains differentiability through softmax probabilities and auxiliary losses. The forward pass executes only the top-1 expert, but router parameters receive gradients through the selected expert’s gating weight and through sparsity and load-balancing losses: $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$6 Here $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$7 encourages sparse routing and $$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$8 encourages balanced expert usage across the batch (You et al., 2021).

DA-MoE retains standard softmax routing and top-$$r^{l} = W_{r}^{l} \cdot \text{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}),$$9 masking. Its novelty lies in dynamic $$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$0, not in a new differentiable estimator. The paper explicitly notes that top-K selection and masking are discrete, while the softmax router itself remains differentiable, and it relies on standard Switch-style practice for gradient propagation (Aghdam et al., 2024).

LD-MoLE replaces non-differentiable TopK with Sparsegen specifically to obtain a fully differentiable router. The support size is controlled by a learnable $$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$1, and a sparsity-control loss pushes $$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$2 into a regime associated with at most $$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$3 active experts: $$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$4 This is combined with a standard load-balancing loss

$$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$5

and the main language-modeling loss (Zhuang et al., 30 Sep 2025). The architecture thereby makes the sparsity mechanism itself differentiable and analytically controllable.

ProbMoE solves the same problem differently. It samples a subset $$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$6 in the forward pass but uses exact subset marginals in the backward pass. For Exact-$$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$7, the marginal probability that expert $$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$8 is active is

$$\text{num\_experts\_to\_route}_k = \left\lceil \text{token\_importance}_k \times E \right\rceil,$$9

and for Dynamic-$K_k$0,

$K_k$1

A straight-through construction

$K_k$2

or its Dynamic-$K_k$3 analogue makes the forward pass sparse and sampled while the backward pass follows exact marginals (Zhao et al., 1 Jun 2026). This yields a probabilistic router that is sparse at execution time and differentiable in training through tractable subset probabilities.

At a more theoretical level, "A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router" studies the adaptive dynamics of a two-expert softmax router with reinforcement and decay: $K_k$4 In the symmetric case $K_k$5, it exhibits a supercritical pitchfork bifurcation at

$K_k$6

beyond which two stable asymmetric load-imbalance states appear (Kiselev, 27 May 2026). This provides a minimal mechanism for abrupt transitions from balanced to imbalanced routing and formalizes why load-balancing interventions can be understood as negative feedback on router dynamics.

This suggests a broader pattern: differentiability and balancing are not merely implementation details but constitutive aspects of router design. They determine whether sparse allocation remains trainable, whether experts collapse, and whether routing trajectories are stable or bifurcate into imbalanced regimes.

5. Multi-router, attention-like, and decoupled router architectures

Dynamic routing is not restricted to single linear routers. Several recent systems expand the router itself into a structured module.

Mixture of Routers (MoR) replaces a single router with multiple sub-routers plus a main router. Each sub-router $K_k$7 produces a distribution $K_k$8 over experts, the main router produces weights

$K_k$9

and the final routing distribution is a mixture

$P$0

A Consistent Routing Weighting variant fixes $P$1, which was found to be more stable in transfer settings (Zhang et al., 30 Mar 2025). The stated motivation is redundancy and fault tolerance at the router level: multiple routing functions reduce the impact of individual misrouting decisions (Zhang et al., 30 Mar 2025).

Router Upcycling similarly increases router expressivity, but through an attention-like formulation initialized from a dense model’s attention heads. Router head $P$2 computes a query

$P$3

expert $P$4 has a key $P$5, and the per-router score is

$P$6

Scores are summed across routers,

$P$7

softmaxed, and then top-$P$8 selected (Ran et al., 31 Aug 2025). This yields token-dependent, multi-view routing while reusing pretrained attention structure.

Read-ME changes router placement rather than router multiplicity. It argues that layer-wise routers are redundant and introduces a pre-gating router decoupled from the MoE backbone, allowing expert decisions to be computed ahead of expert execution (Cai et al., 2024). The system-level consequences are central: expert-aware batching, prefetching, and Belady-style caching all become feasible only because routing is no longer entangled with per-layer hidden states.

RoE applies dynamic routing over depth rather than over parallel experts. Each original transformer layer $P$9 becomes a heavy expert paired with a lightweight adapter $p$0, and the router selects between them using routing tokens and a binary softmax: $p$1 A structure-sparsity loss

$p$2

encourages skipping heavy layers in favor of adapters, producing example-dependent expert paths through depth (Wu et al., 2024).

These architectures show that “dynamic router” can refer not only to how experts are selected, but to the internal architecture of the routing module itself, the granularity of what is considered an expert, and the point in the network where routing is performed.

6. Empirical patterns, specialization, and open issues

Empirically, dynamic routers are repeatedly associated with improved specialization or better compute allocation, though the underlying mechanisms differ by domain. SpeechMoE2 reports 1.6%–4.8% relative CER reduction on a multi-domain task and 1.9%–17.7% relative CER reduction on a multi-accent task relative to SpeechMoE, with comparable parameters and constant FLOPs under top-1 routing (You et al., 2021). DA-MoE reports that it beats a Switch Transformer baseline in 7/8 GLUE tasks in one setup and improves 6/8 tasks in fine-tuning from a pretrained Switch model (Aghdam et al., 2024). "Harder Tasks Need More Experts" reports an average improvement of 0.7% over Top-2 routing while using less than 90% of Top-2’s activated parameters (Huang et al., 2024).

Read-ME reports that its router-decoupled pre-gating design improves mean end-to-end latency up to 6.1% and mean end-to-end latency up to 30% lower than on-demand loading under constrained cache capacities, while preserving or improving accuracy relative to similar-scale dense alternatives (Cai et al., 2024). RoE reports an average performance gain of 3.3% on 5 benchmarks while being faster than MoE-LLaVA, reflecting the effectiveness of dynamic routing over depth in MLLMs (Wu et al., 2024).

Dynamic TMoE extends the notion of dynamic routing into continual structural adaptation: expert pools are added and pruned under MMD-detected drift, while a GRU-based temporal memory router stabilizes expert selection. Across nine benchmarks, it reports average reductions of 10.4% in MSE and 7.8% in MAE (Zhu et al., 20 May 2026). This is dynamic routing in the strongest sense: both the routing policy and the expert pool evolve during training.

Several works also highlight misconceptions. One is that load balancing alone suffices for good routing. "Routers Learn the Geometry of Their Experts" argues instead that effective routing relies on geometric coupling between router and expert directions, and that auxiliary load balancing can break this coupling by spreading input-directed gradients across router weights, making router directions nearly three times more similar to each other (Ahrac et al., 12 May 2026). Another is that dynamic routing must always be discrete. LD-MoLE shows a differentiable sparse alternative, and ProbMoE shows that discrete subset sampling can still be trained through exact marginals (Zhuang et al., 30 Sep 2025, Zhao et al., 1 Jun 2026).

A second misconception is that multilingual MoE routing simply localizes languages to separate experts. "Multilingual Routing in Mixture-of-Experts" instead shows a layer-wise phenomenon: early and late layers route in language-specific ways, but middle layers exhibit substantial cross-lingual routing alignment, and increased alignment to English in these middle layers correlates strongly with better performance (Bandarkar et al., 6 Oct 2025). Inference-time interventions that promote middle-layer task experts frequently activated in English yield 1–2% gains across tasks, models, and 15+ languages (Bandarkar et al., 6 Oct 2025). This suggests that the dynamic router mediates both specialization and universal sharing, depending on depth.

A plausible implication is that future router design will increasingly separate three objectives that are often conflated: cardinality control, expert identity selection, and structural or temporal consistency. The surveyed literature already distributes these across different mechanisms—importance measures, thresholding, sparse projections, subset inference, memory states, clustering, and multi-router ensembling—rather than treating routing as a single linear top-$p$3 decision.