Dynamic Mixture-of-Experts (DyMoE)

• Dynamic Mixture-of-Experts (DyMoE) is a design principle for conditional computation that adjusts expert allocation based on input difficulty, temporal shifts, and system constraints.
• It leverages mechanisms like dynamic routing, adaptive data mixing, and token-level expert selection to optimize model performance and control resource usage.
• DyMoE supports continual learning and efficient inference by dynamically evolving expert pools, reducing computational costs, and tailoring expert activation to task demands.

Dynamic Mixture-of-Experts (DyMoE) denotes a family of mixture-of-experts formulations in which some part of the mixture is allowed to change with the input, the training state, the time index, the task sequence, or the deployment budget. Across the literature, “dynamic” has been attached to several distinct but related mechanisms: time-varying expert and gate parameters in online prediction, input-dependent expert counts, adaptive expert population growth, dynamic data mixing for instruction tuning, scale-aware expert selection in visual autoregression, and runtime orchestration of precision, caching, and prefetching for edge inference (Munezero et al., 2021, Huang et al., 2024, Zhu et al., 2024, Vincenti et al., 8 Oct 2025, Huang et al., 19 Mar 2026). This suggests that DyMoE is best understood as a design principle for conditional computation rather than a single canonical architecture.

1. Conceptual scope and historical development

The term emerged in both statistical and neural contexts. In the statistical formulation of “Dynamic Mixture of Experts Models for Online Prediction,” the conditional density at time $j$ is

$$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$

and both the expert parameters and the mixture weights evolve in time through random-walk processes (Munezero et al., 2021). In later neural MoE work, the same dynamicity is reinterpreted as changing which experts run, how many experts run, how much data each dataset contributes, or how expert resources are provisioned at inference (Huang et al., 2024, Zhu et al., 2024, Huang et al., 19 Mar 2026).

Usage of dynamicity	Dynamic object	Representative source
Online prediction	Expert and gate parameters evolve in time	(Munezero et al., 2021)
Token-adaptive routing	Number of active experts varies by input	(Huang et al., 2024, Guo et al., 2024)
Dense-to-sparse conversion	Number of executed experts varies per token	(Szatkowski et al., 2023)
Instruction tuning	Dataset sampling weights change during SFT	(Zhu et al., 2024)
Visual autoregression	Thresholds vary by generation scale	(Vincenti et al., 8 Oct 2025)
Continual learning	Experts are added, pruned, or isolated over tasks	(Kim, 24 Nov 2025, Zhao et al., 29 Mar 2026, Kong et al., 13 Aug 2025)
Edge deployment	Precision, caching, and prefetching are decided online	(Huang et al., 19 Mar 2026)

A recurring motivation is that fixed-top-$K$, fixed expert counts, fixed dataset mixtures, and fixed deployment policies impose rigid capacity allocation. The DyMoE literature replaces those fixed design choices with mechanisms that respond to input difficulty, task boundaries, routing uncertainty, temporal drift, or systems constraints (Huang et al., 2024, Guo et al., 2024, Huang et al., 19 Mar 2026).

2. Dynamic routing and token-adaptive computation

At the architectural level, DyMoE usually starts from the standard MoE layer

$$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$

with router probabilities

$$\mathbf{P}=Softmax(\mathbf{W_r}\cdot \mathbf{x}^{T}).$$

The dynamic step is to abandon the assumption that every token must activate the same number of experts (Huang et al., 2024).

In “Harder Tasks Need More Experts: Dynamic Routing in MoE Models,” the router distribution is treated as a confidence signal. Experts are sorted by probability, and experts are added until the cumulative probability exceeds a threshold $p$. The method adds an entropy-based dynamic loss,

$Loss_{d}=- \sum_{i=1}^N {P_i * log (P_i)},$

alongside a standard load-balancing term. Empirically, it achieves an average improvement of 0.7% over Top-2 routing with less than 90% activated parameters, and the average activated experts per token at inference is 1.76, with BBH: 1.87 experts versus 1.72–1.76 on other tasks, supporting the claim that harder tasks need more experts (Huang et al., 2024).

A related dense-to-sparse conversion line appears in “Exploiting Activation Sparsity with Dense to Dynamic-$k$ Mixture-of-Experts Conversion.” There, a pre-trained dense Transformer is converted into experts, routers are trained to predict the $\ell_2$-norm of each expert’s output, and an expert is activated if

$$G(z)_i= \begin{cases} 1 & \text{if } R(z)_i \ge \tau \cdot \max R(z) \ 0 & \text{if } R(z)_i < \tau \cdot \max R(z) \end{cases}.$$

The number of executed experts is therefore chosen dynamically per token, and the reported savings reach up to 60% of inference cost without significantly impacting performance (Szatkowski et al., 2023).

“Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models” extends the idea further with “top-any gating.” For token embedding $$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$0, the gate is

$$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$1

so each expert is activated independently and the active expert count becomes

$$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$2

The same work also adjusts the number of experts during training by adding experts when some tokens activate none and removing experts when some experts are never selected. On GLUE, DynMoE reaches 81.64, while the best conventional fixed setting is around 81.77, and in vision-language evaluation throughput improves from 19 to 26 tokens/s, 18 to 23 tokens/s, and 14 to 18 tokens/s for different backbones (Guo et al., 2024).

Inference-time routing can also be made dynamic without retraining. “Dynamic Experts Search” treats the number of active experts $$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$3 as an explicit inference-time variable,

$$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$4

and combines it with Expert Configuration Inheritance, which keeps $$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$5 constant within one reasoning path while varying it across runs. With $$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$6 rollouts, $$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$7, $$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$8, and initial expert counts $$f_{j}\left(y_{j}\mid \tilde{\mathbf{x}}_{j},\boldsymbol{\omega}_{j},\boldsymbol{\lambda}_{j}\right)=\sum_{k=1}^{K}\omega_{jk}\left(\mathbf{z}_{j}\right)f_{jk}\left(y_{j}\mid \lambda_{jk}\left(\mathbf{x}_{j}\right)\right),$$9, DES reliably outperforms Best-of-$K$0, Beam Search, and DVTS with similar computational cost (Han et al., 26 Sep 2025).

3. Dynamic control during training: data mixtures, curricula, and specialization

A distinct DyMoE line moves the dynamic mechanism from routing to the data pipeline. “Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts” observes that standard MoE SFT typically merges all instruction datasets and samples them with static weights, even though the usefulness of a dataset changes as the model trains. The method constructs a dataset-level representation from last-layer token routing: $K$1 normalizes it, measures inter-dataset redundancy by pairwise $K$2 distance,

$K$3

and averages those distances into $K$4. Sampling weights are then updated by

$K$5

The weights are updated every $K$6 training steps, the gates are frozen, and the last MoE layer is used because deeper layers have better specialization (Zhu et al., 2024).

The reported setup uses MoLM 700M-4E and LLaMA-MoE 3.5B-2E, with four instruction datasets: ShareGPT, OpenOrca, Math-Instruct, and Code Instructions. On MoLM, Dynamic improves the averaged knowledge/reasoning score to 22.00 and MT-Bench to 2.73. On LLaMA-MoE, Dynamic reaches 30.78 average K&R and 5.22 MT-Bench, with 11.90 on GSM8K and 16.88 on MBPP. The work also reports that RefLoss is strong but doubles training computation because it needs an extra reference-loss estimate, while Dynamic performs better without that overhead (Zhu et al., 2024).

Several ablations define the practical regime of the method. With $K$7, weights change too slowly and fail to stabilize; with $K$8, training becomes noisy and performance drops; $K$9 is the best compromise for LLaMA-MoE. The method requires at least three datasets, because with only two datasets the distance vector $$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$0 offers no meaningful variation, so the weights would remain unchanged. Uniform initialization is beneficial; non-uniform initialization from Sentence-BERT hurts performance relative to uniform initialization (Zhu et al., 2024).

Taken together, these results generalize the DyMoE idea from “which experts run?” to “which data should the experts see next?” A plausible implication is that MoE internal routing can serve as a feedback signal not only for token dispatch, but also for data curriculum design.

4. Continual learning, severe distribution shift, and lifelong expert evolution

Under continual learning and nonstationary streams, DyMoE usually means that the expert pool itself changes over time. In “Dynamic Mixture of Experts Against Severe Distribution Shifts,” the model periodically adds new experts to MoE layers to preserve plasticity under changing distributions. It does not require task indices, uses a bottleneck expert design, and evaluates growth granularities 1, 2, and 4. On sequential Tiny ImageNet chunks, g2 and g4 maintain initial performance better than static baselines, and in Craftax RL a 1-to-2 grow policy can surpass No MoE and 2 experts, because full capacity may be wasted on early distributions while dynamic growth reserves capacity for later shifts (Kim, 24 Nov 2025).

For multimodal continual instruction tuning, “On Token’s Dilemma” identifies routing-drift: old-task tokens become mistakenly attracted to newly added experts even when old experts are frozen. LLaVA-DyMoE addresses this with Drift-aware Token Assignment (TAG) and Routing Score Regularization (RSR). Tokens are categorized through old-group and new-group routing scores, ambiguous and old tokens are steered away from new experts, and exclusivity and specialization losses are added. On CoIN, LLaVA-DyMoE achieves MFN = 57.03, MAA = 57.70, and BWT = -4.67, compared with IncMoELoRA at 49.68, 49.50, and -16.67. The paper summarizes this as over a 7% gain in mean final accuracy and a 12% reduction in forgetting (Zhao et al., 29 Mar 2026).

DIMoE-Adapters brings the same logic to VLM adapters. It keeps a frozen CLIP ViT-B/16 backbone, uses LoRA-based experts inside residual MoE adapters, and introduces Self-Calibrated Expert Evolution (SCEE) and Prototype-Guided Expert Selection (PGES). SCEE expands or prunes experts based on Expert Optimization Contribution and Expert Optimization Instability; PGES uses task prototypes and Mahalanobis scoring to select the appropriate task router or fall back to frozen CLIP. On MTIL Order I, compared with MoE-Adapters, DIMoE-Adapters improves Transfer by +0.4%, Avg. by +2.4%, and Last by +4.0%; trainable parameters drop by 33.9%, GPU memory by 26.9%, and training time per iteration by 21.5% (Qin et al., 8 May 2026).

Graph incremental learning introduces a different form of temporal structure. “Dynamic Mixture-of-Experts for Incremental Graph Learning” assigns one expert to each incoming data block, trains only the new expert and the gates when a new block arrives, and adds a block-guided loss and a graph block-guided loss so that older experts do not process future nodes as if they had been present earlier. A sparse top-$$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$1 variant reduces computation time as the number of blocks grows. The reported headline is a 4.92\% relative accuracy increase compared to the best baselines on class incremental learning (Kong et al., 13 Aug 2025).

Related lifelong formulations extend the same dynamic-MoE pattern beyond standard supervised prediction. MoLEM keeps the base reasoner entirely frozen and routes through latent-memory experts, improving the average accuracy by 10.40% over the Vanilla pretrained baseline after the full Math/Science/Code continual-learning sequence (Yu et al., 21 May 2026). LiMoDE uses a dynamic MoE structure in robot manipulation pre-training and a lifelong MoE adaptation mechanism for new tasks; on LIBERO-OBJECT it reports FWT 81.0, BWT 5.0, and AUC 76.0 (Gu et al., 24 Jun 2026).

5. Inference-time efficiency, systems support, and deployment

A major branch of DyMoE is explicitly about budgeted inference. In “Dynamic Mixture-of-Experts for Visual Autoregressive Model,” dense FFNs are replaced by experts and the router predicts expert utility; expert $$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$2 is selected at scale $$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$3 when

$$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$4

The thresholds are scale-dependent, so coarse scales admit more experts and fine scales become sparser. Starting from the pre-trained VAR-d16 baseline, fine-tuning for two epochs, using 32 experts of size 128, applying MoE only on the last three scales, and setting

$$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$5

DyMoE-VAR achieves about 19% fewer FLOPs, about 11% faster inference time, and FID within 1% of the dense VAR d16 baseline (Vincenti et al., 8 Oct 2025).

At the systems level, “DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge” treats expert precision and offloading as runtime decisions. It combines importance-aware prioritization, depth-adaptive scheduling, and look-ahead prefetching. During prefill, token importance is estimated from attention; during decode, expert importance is the gate weight; critical experts are kept at higher precision, sub-critical experts are quantized more aggressively or skipped in the 0-bit setting. On commercial edge hardware, DyMoE reports TTFT reduction: 3.44x to 22.7x and TPOT speedup: up to 14.58x compared to offloading baselines. It also reports 68.07% MMLU for Mixtral-8×7B with 4/0 at $$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$6, nearly identical to uniform Int4 at 67.95%, and 91.74% on GSM8K for Qwen3-30B-A3B, exceeding standard Int4 at 89.08% (Huang et al., 19 Mar 2026).

These deployment-oriented DyMoE systems depend on runtime support. “Optimizing Mixture of Experts using Dynamic Recompilations” introduces DynaMoE, a DNN library that changes the computation graph and tensor sizes during training with low overhead. Its headline result is a 1.8x speedup and support for 2.3x larger model sizes compared to existing MoE systems, even without recompilations; dynamic capacity factors and sample assignment caching then yield an additional 1.7x speedup while reducing memory pressure and improving model quality (Kossmann et al., 2022).

Dense-to-MoE conversion papers form a nearby deployment line. SADMoE converts dense Transformers into dynamic-$$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$7 expertized models and reports up to 60% of inference cost reduction (Szatkowski et al., 2023). ToMoE converts dense LLM MLP layers into MoE experts through differentiable dynamic pruning and, even without fine-tuning, consistently outperforms previous structural pruning techniques across Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5 (Gao et al., 25 Jan 2025).

6. Theory, interpretation, and recurring limitations

Theoretical work treats DyMoE as a model of adaptive specialization rather than merely an engineering heuristic. In the statistical DyMoE for online prediction, the dynamic parameter vector follows a random walk,

$$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$8

and inference is performed with a sequential Monte Carlo algorithm whose tailored proposal combines linear Bayes ideas with EM-style responsibilities. Dynamic models strongly outperform static models in the software-fault prediction study, and the framework gives a unified treatment of static and dynamic mixtures through the discount factor $$MoE(\mathbf{x})=\sum_{i=1}^{N}g_i(\mathbf{x}) * e_i(\mathbf{x}),$$9 (Munezero et al., 2021).

A separate theory line studies joint optimization in soft-routed MoE. “Guided by the Experts” proves convergence guarantees for student-teacher training with a soft-routed MoE, showing that the student network undergoes a feature learning phase in which the router’s learning process is “guided” by the experts, then admits a post-training pruning stage and a provably convergent fine-tuning stage that reaches global optimality (Liao et al., 8 Oct 2025). A more architectural theory appears in “DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity,” where the number of reachable activation patterns expands from fixed-top-$$\mathbf{P}=Softmax(\mathbf{W_r}\cdot \mathbf{x}^{T}).$$0

$$\mathbf{P}=Softmax(\mathbf{W_r}\cdot \mathbf{x}^{T}).$$1

to

$$\mathbf{P}=Softmax(\mathbf{W_r}\cdot \mathbf{x}^{T}).$$2

and the empirical conclusion is that optimal expert schedules are task- and scale-dependent: descending schedules outperform uniform baselines on image classification, while language modeling prefers descending for Tiny, ascending for Small, and uniform for Medium (Gülmez, 2 Mar 2026).

Several misconceptions are corrected by the recent literature. Frozen old experts do not, by themselves, prevent forgetting, because routing-drift can still redirect old-task tokens to new experts (Zhao et al., 29 Mar 2026). Dynamic MoE is also not synonymous with variable top-$$\mathbf{P}=Softmax(\mathbf{W_r}\cdot \mathbf{x}^{T}).$$3: the same term is used for dynamic data mixing, dynamic expert growth, dynamic runtime orchestration, and time-varying statistical mixtures (Zhu et al., 2024, Huang et al., 19 Mar 2026). Taken together, the literature suggests that “dynamic” refers to adaptive allocation of compute or capacity, but the allocation target differs across papers.

Limitations are similarly heterogeneous. Dynamic Data Mixing has only been tested on two decoder-style MoE models, not larger models such as Mixtral, and it requires at least three datasets (Zhu et al., 2024). DyMoE-VAR notes that very fine details can be missed, and that the optimal batch size and expert size are hardware-specific (Vincenti et al., 8 Oct 2025). Edge DyMoE depends on expert importance skewness, depth-dependent sensitivity, and adjacent-layer predictability (Huang et al., 19 Mar 2026). Continual-learning DyMoE variants must manage the stability–plasticity trade-off not only in expert weights but also in routers, memory selection, and task inference (Qin et al., 8 May 2026, Kim, 24 Nov 2025).

The unifying theme is therefore structural adaptability. Whether the object being adapted is the expert count, the expert pool, the dataset mixture, the routing threshold, the precision format, or the mixture parameters over time, DyMoE methods replace static allocation rules with mechanisms that respond to changing evidence. That shared principle explains why the same label now spans instruction tuning, reasoning-time search, continual learning, visual generation, graph learning, robotics, and edge inference.