Dynamic TMoE: Adaptive Expert Models

Updated 5 July 2026

Dynamic TMoE is a family of adaptive mixture-of-experts models that dynamically adjust expert selection, capacity, and execution based on input characteristics.
It incorporates methods such as percentile-threshold routing, cumulative top-p selection, and thresholded sigmoid gating to customize computation at token and layer levels.
Dynamic TMoE also addresses system-level challenges by managing irregular batch shapes, communication imbalances, and dynamic resource allocation for enhanced efficiency.

Searching arXiv for papers on Dynamic TMoE and closely related dynamic MoE work. {"query":"\"Dynamic TMoE\" mixture of experts dynamic routing arXiv", "max_results": 10} arXiv search query: Dynamic TMoE mixture of experts dynamic routing Dynamic TMoE denotes a family of mixture-of-experts formulations in which expert selection, expert capacity, or expert execution policy is adapted dynamically rather than fixed a priori. In recent arXiv usage, the term has referred most often to dynamic Transformer Mixture-of-Experts designs with input-dependent routing, such as DynaMoE’s dynamic token-level activation and layer-wise adaptive capacity allocation, but it has also been used for drift-aware temporal forecasting architectures, system-level runtime specialization, and month-conditioned temporal expert routing in weather models (Gülmez, 2 Mar 2026, Zhu et al., 20 May 2026, Chen et al., 21 Sep 2025). The common thread is the relaxation of static assumptions—especially fixed Top- $K$ routing, uniform expert allocation, or static execution plans—in favor of mechanisms that adapt to token difficulty, layer role, temporal regime, topology, or hardware state.

1. Terminology and scope

In the cited literature, TMoE is not a single standardized acronym. It denotes Transformer Mixture-of-Experts in work on dynamic routing and systems optimization (Kossmann et al., 2022, Huang et al., 2024); Temporal Mixture-of-Experts in seasonal weather forecasting (Chen et al., 21 Sep 2025); Task-aware Mixture-of-Experts in multi-objective soft sensing (Pan et al., 2023); and, in an earlier statistical tradition, $t$ Mixture-of-Experts, where experts follow a Student- $t$ distribution for robustness to heavy tails and atypical observations (Chamroukhi, 2016). This suggests that “Dynamic TMoE” is best understood as a contextual umbrella rather than a single architecture family.

Within neural sequence modeling, the phrase usually refers to MoE systems that make at least one of three quantities adaptive: the number of active experts per token, the number or composition of experts available at a layer, or the execution strategy used to realize sparse routing efficiently. DynaMoE, DynMoE, DSMoE, DOT-MoE, and ToMoE all instantiate this adaptivity, but they do so through different control variables: percentile-threshold routing, thresholded sigmoid gating, dense-to-MoE conversion, differentiable structural pruning, or adaptive expert-count adjustment (Gülmez, 2 Mar 2026, Guo et al., 2024, Lv et al., 18 Feb 2025, Bamba et al., 1 Jun 2026, Gao et al., 25 Jan 2025).

2. Dynamic routing as the core mechanism

The most direct form of Dynamic TMoE replaces fixed Top- $K$ routing with an input-dependent expert count. In DynaMoE, the router computes gating logits $g(x)=W_gx$ and probabilities $p_i(x)=\mathrm{softmax}(g(x))_i$ , then defines the active set by percentile thresholding,

$S_\tau(x)=\{i: p_i(x)>\mathrm{percentile}_\tau(p(x))\},$

with fallback to $\arg\max_i p_i(x)$ if the set is empty. The resulting active count satisfies $1\le K(x)\le \lceil (1-\tau)N_\ell\rceil$ , and the expert outputs are combined with a temperature-scaled aggregation using $T=0.5$ ; during training, Gaussian gating noise with $t$ 0 is added for exploration (Gülmez, 2 Mar 2026). The same work formalizes the expressivity gain: fixed Top- $t$ 1 admits $t$ 2 routing patterns, whereas dynamic routing admits $t$ 3.

A closely related but distinct mechanism appears in “Harder Tasks Need More Experts,” where the number of experts is determined by routing confidence rather than a percentile threshold. For token $t$ 4, experts are sorted by probability, and the selected count $t$ 5 is the smallest $t$ 6 such that the cumulative probability exceeds a threshold $t$ 7. In experiments, $t$ 8 during training and inference, while inference additionally caps activation at $t$ 9. The model uses raw probabilities for the selected experts, adds an entropy regularizer to discourage diffuse routing, and reports an average improvement of $t$ 0 over Top-2 with less than $t$ 1 activated parameters; complex reasoning tasks such as BBH receive more experts on average (Huang et al., 2024).

Dynamic routing can also be implemented without softmax Top- $t$ 2 selection. DSMoE partitions each dense FFN into matrix-level experts and applies thresholded sigmoid gates,

$t$ 3

with $t$ 4 and a straight-through estimator to preserve gradients through hard selection. The number of active experts varies per token and layer, and output magnitude is normalized by the ratio of total experts to active experts (Lv et al., 18 Feb 2025). By contrast, the dense-to-dynamic DynaMoE framework for post-training LLM adaptation retains Top-1 inference but makes expert width dynamic: each MLP is sliced into nested experts of widths $t$ 5, $t$ 6, $t$ 7, and $t$ 8, and a token-difficulty-aware router is supervised with labels derived from similarity to the full-width expert output (Nishu et al., 17 Feb 2025).

The literature therefore does not equate Dynamic TMoE with a single router. It includes percentile-threshold routing, cumulative top- $t$ 9 routing, thresholded sigmoid gating, and token-difficulty-driven expert-width selection. What unifies these formulations is that compute is matched to the token or input state rather than prescribed uniformly.

3. Structural adaptation across depth, expert pools, and dense-to-MoE conversion

A second axis of dynamism concerns where capacity is placed. DynaMoE makes the expert count $K$ 0 a function of depth and evaluates six schedules: descending, ascending, pyramid-up, pyramid-down, wave-down, and wave-up. Under a total expert budget constraint, these schedules replace the uniform-per-layer assumption. The empirical finding is explicitly task- and scale-dependent: descending schedules outperform uniform baselines on image classification, while in language modeling the best schedule changes with model size—descending for Tiny, ascending for Small, and uniform for Medium (Gülmez, 2 Mar 2026).

Other methods make the expert pool itself adaptive during training. DynMoE records per-expert activation counts $K$ 1 and a statistic $K$ 2 for tokens whose dynamic gating activates no experts. It then removes experts with zero activation and adds a new expert initialized from $K$ 3, yielding an auto-tuned expert count per layer rather than a fixed one (Guo et al., 2024). Dynamic TMoE for non-stationary forecasting extends this idea further: it uses MMD-based drift detection to trigger instantiation of heterogeneous experts and prunes redundant ones using usage statistics and redundancy checks. The router maintains recurrent hidden states and an anomaly repository, but no test-time parameter updates are performed (Zhu et al., 20 May 2026).

Dense-to-MoE conversion introduces a different kind of structural adaptivity. DOT-MoE formulates neuron-to-expert assignment as a balanced optimal transport problem with Sinkhorn-Knopp iterations and strict expert capacities, then jointly learns discrete neuron assignment and token routing with straight-through estimators (Bamba et al., 1 Jun 2026). ToMoE, by contrast, converts dense LLM MLPs into top-1 routed experts via differentiable dynamic structural pruning. Expert masks are learned with Straight-Through Gumbel-Sigmoid, the base model weights remain frozen, and a fixed active parameter budget per forward pass is enforced through regularizers and equalized expert widths per layer (Gao et al., 25 Jan 2025). In these formulations, the routing may remain fixed-top-1 or fixed-top- $K$ 4, yet the structure exposed to routing is learned dynamically.

4. System-level Dynamic TMoE

Dynamic TMoE is also a systems problem, because sparse routing creates irregular batches, dynamic tensor shapes, communication imbalance, and memory pressure. Several papers treat the runtime as part of the architecture.

System paper	Dynamic signal	Reported effect
DynaMoE (Kossmann et al., 2022)	Runtime routing statistics, dynamic recompilation, capacity resizing, sample-assignment caching	1.8x speedup, 2.3x larger models, plus an additional 1.7x speedup
TA-MoE (Chen et al., 2023)	Network topology and communication cost	1.01x–1.61x over DeepSpeed-MoE; 1.01x–4.77x over FastMoE
D $K$ 5MoE (Wang et al., 17 Apr 2025)	Per-token bit-width routing and hottest-expert-bit-first scheduling	up to 1.39x throughput, up to 53% peak memory reduction
DyMoE (Huang et al., 19 Mar 2026)	Runtime expert importance, depth-aware precision, look-ahead prefetch	TTFT reduced by 3.44x–22.7x; up to 14.58x TPOT speedup
MoECa (Li et al., 14 Jun 2026)	Cross-timestep branch reuse in DiT-MoE	up to 2.83x inference speedup

The 2022 DynaMoE systems paper addresses static-graph inefficiency by recompiling only the routing-dependent subgraph when capacities change and by overlapping CPU-side recompilation decisions with GPU execution through a launch frontier. It also introduces routing-aware operators such as TopK, GroupBy, Aggregate, AggregateSpec, and Cache (Kossmann et al., 2022). TA-MoE instead makes the router topology-aware: it models communication cost over hierarchical interconnects and adds a topology-aware auxiliary loss to bias routing toward low-cost source-to-destination pairs without sacrificing model accuracy (Chen et al., 2023).

On-device and edge inference extends the same principle to storage, quantization, and scheduling. D $K$ 6MoE performs dual routing—expert identity plus bit-width—and uses matryoshka weight quantization together with a hottest-expert-bit-first heuristic to overlap I/O and compute under a memory budget (Wang et al., 17 Apr 2025). DyMoE similarly treats expert importance as dynamic, assigns mixed precision according to depth and runtime importance, and uses look-ahead prefetching to reduce wait-for-weight stalls on PCIe-limited hardware (Huang et al., 19 Mar 2026). In diffusion Transformers, MoECa argues that cross-timestep redundancy is better characterized at the expert-branch level than at the token level, and it performs branch-level caching with synchronized updates across attention and MoE paths (Li et al., 14 Jun 2026).

5. Empirical regularities across domains

Several empirical regularities recur across otherwise different Dynamic TMoE formulations. First, harder or more heterogeneous inputs tend to recruit more capacity. The confidence-based routing study reports that BBH receives the highest average number of activated experts among evaluated tasks and that lower layers often activate more experts than upper layers (Huang et al., 2024). DynaMoE reports that descending schedules concentrate early compute and that average active experts per token decline with depth, with an illustrative drop from roughly $K$ 7 at the first layer to roughly $K$ 8 at the fourth layer in a small image-classification setting (Gülmez, 2 Mar 2026).

Second, dynamic routing is most beneficial when domain heterogeneity or regime shift is intrinsic to the task. In Tra-MoE, sparse Top-1 MoE combined with out-of-domain data raises the LIBERO average success rate to $K$ 9, whereas dense training with the same OOD augmentation drops to $g(x)=W_gx$ 0 (Yang et al., 2024). In TMoW, routing is updated at test time over world-model adapters, and the paper reports an average unseen-domain success-rate improvement of $g(x)=W_gx$ 1 in zero-shot adaptation (Jang et al., 30 Jan 2026). These results suggest that dynamic routing can function as a mechanism for reconciling specialization with broad data coverage.

Third, temporal non-stationarity consistently motivates dynamic expert management. Dynamic TMoE for time-series forecasting reports state-of-the-art performance on nine benchmarks, with average reductions of $g(x)=W_gx$ 2 in MSE and $g(x)=W_gx$ 3 in MAE (Zhu et al., 20 May 2026). STCast’s Temporal Mixture-of-Experts routes inputs according to a month-conditioned discrete Gaussian prior; on global forecasting, removing TMoE or removing month embedding degrades long-horizon skill, while the full model reaches a 10-day RMSE of $g(x)=W_gx$ 4 and ACC of $g(x)=W_gx$ 5 (Chen et al., 21 Sep 2025).

Fourth, post-training dynamic MoEfication can approach competitive accuracy without full retraining. The dense-to-dynamic DynaMoE framework adapts a pretrained dense LLM into a token-difficulty-driven MoE with a single fine-tuning stage on $g(x)=W_gx$ 6 tokens and reports similar aggregated accuracy to Flextron while using only $g(x)=W_gx$ 7 of its fine-tuning cost (Nishu et al., 17 Feb 2025). DOT-MoE similarly reports retention of about $g(x)=W_gx$ 8 of the original dense model’s performance while reducing active parameters by $g(x)=W_gx$ 9 (Bamba et al., 1 Jun 2026).

6. Limitations, misconceptions, and open problems

A recurring misconception is that Dynamic TMoE always means “variable $p_i(x)=\mathrm{softmax}(g(x))_i$ 0 per token.” The literature is broader. Some models do vary $p_i(x)=\mathrm{softmax}(g(x))_i$ 1 explicitly (Gülmez, 2 Mar 2026, Huang et al., 2024); others keep Top-1 or fixed Top- $p_i(x)=\mathrm{softmax}(g(x))_i$ 2 routing but make expert widths, neuron partitions, expert pools, capacities, bit-widths, or execution plans dynamic (Nishu et al., 17 Feb 2025, Bamba et al., 1 Jun 2026, Kossmann et al., 2022). Another misconception is that dynamic routing automatically yields lower end-to-end cost. Several papers emphasize that routing adaptivity shifts complexity into dispatch, caching, recompilation, communication, or offloading, and that these costs must be managed explicitly (Kossmann et al., 2022, Wang et al., 17 Apr 2025).

Scalability remains an open issue. DynaMoE’s architectural formulation does not include capacity-factor caps, token dropping, or auxiliary balancing losses, and the absence of such stabilizers is identified as a limitation for very large-scale settings; its language-modeling experiments are also explicitly pilot-scale, using Recycling-the-Web-1k (Gülmez, 2 Mar 2026). Dynamic recompilation can lose effectiveness if routing remains highly volatile, because frequent recompiles erode the benefit of specialization (Kossmann et al., 2022). Topology-aware routing depends on accurate topology estimation, and stale or noisy bandwidth estimates can mis-bias the gate (Chen et al., 2023). DOT-MoE avoids heuristic expert partitioning, but its hard capacity applies to neuron counts rather than token counts, so transient token overload is still handled through balancing loss rather than a hard routing cap (Bamba et al., 1 Jun 2026).

The most general unresolved question is therefore not whether routing should be dynamic, but which aspect should be dynamic for a given regime: token-level expert count, layer-wise capacity, expert structure, world-model composition, bit-width, cache state, or communication pattern. The surveyed work collectively indicates that “Dynamic TMoE” is less a single algorithm than a design principle: sparse expert models become substantially more effective when the source of heterogeneity—input difficulty, depth, temporal drift, topology, or memory pressure—is itself made explicit in the routing or execution policy.