---
title: 'Dynamic TMoE: Adaptive Expert Models'
url: https://www.emergentmind.com/topics/dynamic-tmoe
type: topic
---

# Dynamic TMoE: Adaptive Expert Models

Searching arXiv for recent papers on Dynamic TMoE and closely related dynamic MoE work.
{"query":"\"Dynamic TMoE\" mixture of experts dynamic routing arXiv", "max_results": 10}
arXiv search query: Dynamic TMoE mixture of experts dynamic routing
Dynamic TMoE denotes a family of mixture-of-experts formulations in which expert selection, expert capacity, or expert execution policy is adapted dynamically rather than fixed a priori. In recent arXiv usage, the term has referred most often to dynamic Transformer Mixture-of-Experts designs with input-dependent routing, such as DynaMoE’s dynamic token-level activation and layer-wise adaptive capacity allocation, but it has also been used for drift-aware temporal forecasting architectures, system-level runtime specialization, and month-conditioned temporal expert routing in weather models [2603.01697], [2605.20678], [2509.25210]. The common thread is the relaxation of static assumptions—especially fixed Top-$K$ routing, uniform expert allocation, or static execution plans—in favor of mechanisms that adapt to token difficulty, layer role, temporal regime, topology, or hardware state.

## 1. Terminology and scope

In the cited literature, **TMoE** is not a single standardized acronym. It denotes **Transformer Mixture-of-Experts** in work on dynamic routing and systems optimization [2205.01848], [2403.07652]; **Temporal Mixture-of-Experts** in seasonal weather forecasting [2509.25210]; **Task-aware Mixture-of-Experts** in multi-objective soft sensing [2302.10477]; and, in an earlier statistical tradition, **$t$ Mixture-of-Experts**, where experts follow a Student-$t$ distribution for robustness to heavy tails and atypical observations [1701.07429]. This suggests that “Dynamic TMoE” is best understood as a contextual umbrella rather than a single architecture family.

Within neural sequence modeling, the phrase usually refers to MoE systems that make at least one of three quantities adaptive: the number of active experts per token, the number or composition of experts available at a layer, or the execution strategy used to realize sparse routing efficiently. DynaMoE, DynMoE, DSMoE, DOT-MoE, and ToMoE all instantiate this adaptivity, but they do so through different control variables: percentile-threshold routing, thresholded sigmoid gating, dense-to-MoE conversion, differentiable structural pruning, or adaptive expert-count adjustment [2603.01697], [2405.14297], [2502.12455], [2606.01666], [2501.15316].

## 2. Dynamic routing as the core mechanism

The most direct form of Dynamic TMoE replaces fixed Top-$K$ routing with an input-dependent expert count. In DynaMoE, the router computes gating logits $g(x)=W_gx$ and probabilities $p_i(x)=\mathrm{softmax}(g(x))_i$, then defines the active set by percentile thresholding,
$$
S_\tau(x)=\{i: p_i(x)>\mathrm{percentile}_\tau(p(x))\},
$$
with fallback to $\arg\max_i p_i(x)$ if the set is empty. The resulting active count satisfies $1\le K(x)\le \lceil (1-\tau)N_\ell\rceil$, and the expert outputs are combined with a temperature-scaled aggregation using $T=0.5$; during training, Gaussian gating noise with $\sigma=0.1$ is added for exploration [2603.01697]. The same work formalizes the expressivity gain: fixed Top-$K$ admits $\binom{N}{K}$ routing patterns, whereas dynamic routing admits $\sum_{k=1}^{K_{\max}}\binom{N}{k}$.

A closely related but distinct mechanism appears in “Harder Tasks Need More Experts,” where the number of experts is determined by routing confidence rather than a percentile threshold. For token $i$, experts are sorted by probability, and the selected count $K_i$ is the smallest $k$ such that the cumulative probability exceeds a threshold $p$. In experiments, $p=0.4$ during training and inference, while inference additionally caps activation at $K_{\max}=2$. The model uses raw probabilities for the selected experts, adds an entropy regularizer to discourage diffuse routing, and reports an average improvement of $0.7\%$ over Top-2 with less than $90\%$ activated parameters; complex reasoning tasks such as BBH receive more experts on average [2403.07652].

Dynamic routing can also be implemented without softmax Top-$K$ selection. DSMoE partitions each dense FFN into matrix-level experts and applies thresholded sigmoid gates,
$$
g_i=\sigma(\hat h_t^lY_i), \qquad
G(x)=\begin{cases}
x & x>\tau\\
0 & \text{otherwise},
\end{cases}
$$
with $\tau=0.5$ and a straight-through estimator to preserve gradients through hard selection. The number of active experts varies per token and layer, and output magnitude is normalized by the ratio of total experts to active experts [2502.12455]. By contrast, the dense-to-dynamic DynaMoE framework for post-training LLM adaptation retains Top-1 inference but makes expert *width* dynamic: each MLP is sliced into nested experts of widths $0.25H$, $0.5H$, $0.75H$, and $H$, and a token-difficulty-aware router is supervised with labels derived from similarity to the full-width expert output [2502.12325].

The literature therefore does not equate Dynamic TMoE with a single router. It includes percentile-threshold routing, cumulative top-$p$ routing, thresholded sigmoid gating, and token-difficulty-driven expert-width selection. What unifies these formulations is that compute is matched to the token or input state rather than prescribed uniformly.

## 3. Structural adaptation across depth, expert pools, and dense-to-MoE conversion

A second axis of dynamism concerns *where* capacity is placed. DynaMoE makes the expert count $N_\ell$ a function of depth and evaluates six schedules: descending, ascending, pyramid-up, pyramid-down, wave-down, and wave-up. Under a total expert budget constraint, these schedules replace the uniform-per-layer assumption. The empirical finding is explicitly task- and scale-dependent: descending schedules outperform uniform baselines on image classification, while in language modeling the best schedule changes with model size—descending for Tiny, ascending for Small, and uniform for Medium [2603.01697].

Other methods make the expert pool itself adaptive during training. DynMoE records per-expert activation counts $R_E$ and a statistic $R_S$ for tokens whose dynamic gating activates no experts. It then removes experts with zero activation and adds a new expert initialized from $R_S/\|R_S\|$, yielding an auto-tuned expert count per layer rather than a fixed one [2405.14297]. Dynamic TMoE for non-stationary forecasting extends this idea further: it uses MMD-based drift detection to trigger instantiation of heterogeneous experts and prunes redundant ones using usage statistics and redundancy checks. The router maintains recurrent hidden states and an anomaly repository, but no test-time parameter updates are performed [2605.20678].

Dense-to-MoE conversion introduces a different kind of structural adaptivity. DOT-MoE formulates neuron-to-expert assignment as a balanced optimal transport problem with Sinkhorn-Knopp iterations and strict expert capacities, then jointly learns discrete neuron assignment and token routing with straight-through estimators [2606.01666]. ToMoE, by contrast, converts dense LLM MLPs into top-1 routed experts via differentiable dynamic structural pruning. Expert masks are learned with Straight-Through Gumbel-Sigmoid, the base model weights remain frozen, and a fixed active parameter budget per forward pass is enforced through regularizers and equalized expert widths per layer [2501.15316]. In these formulations, the routing may remain fixed-top-1 or fixed-top-$k$, yet the *structure* exposed to routing is learned dynamically.

## 4. System-level Dynamic TMoE

Dynamic TMoE is also a systems problem, because sparse routing creates irregular batches, dynamic tensor shapes, communication imbalance, and memory pressure. Several papers treat the runtime as part of the architecture.

| System paper | Dynamic signal | Reported effect |
|---|---|---|
| DynaMoE [2205.01848] | Runtime routing statistics, dynamic recompilation, capacity resizing, sample-assignment caching | 1.8x speedup, 2.3x larger models, plus an additional 1.7x speedup |
| TA-MoE [2302.09915] | Network topology and communication cost | 1.01x–1.61x over DeepSpeed-MoE; 1.01x–4.77x over FastMoE |
| D$^2$MoE [2504.15299] | Per-token bit-width routing and hottest-expert-bit-first scheduling | up to 1.39x throughput, up to 53% peak memory reduction |
| DyMoE [2603.19172] | Runtime expert importance, depth-aware precision, look-ahead prefetch | TTFT reduced by 3.44x–22.7x; up to 14.58x TPOT speedup |
| MoECa [2606.15615] | Cross-timestep branch reuse in DiT-MoE | up to 2.83x inference speedup |

The 2022 DynaMoE systems paper addresses static-graph inefficiency by recompiling only the routing-dependent subgraph when capacities change and by overlapping CPU-side recompilation decisions with GPU execution through a launch frontier. It also introduces routing-aware operators such as TopK, GroupBy, Aggregate, AggregateSpec, and Cache [2205.01848]. TA-MoE instead makes the router topology-aware: it models communication cost over hierarchical interconnects and adds a topology-aware auxiliary loss to bias routing toward low-cost source-to-destination pairs without sacrificing model accuracy [2302.09915].

On-device and edge inference extends the same principle to storage, quantization, and scheduling. D$^2$MoE performs dual routing—expert identity plus bit-width—and uses matryoshka weight quantization together with a hottest-expert-bit-first heuristic to overlap I/O and compute under a memory budget [2504.15299]. DyMoE similarly treats expert importance as dynamic, assigns mixed precision according to depth and runtime importance, and uses look-ahead prefetching to reduce wait-for-weight stalls on PCIe-limited hardware [2603.19172]. In diffusion Transformers, MoECa argues that cross-timestep redundancy is better characterized at the expert-branch level than at the token level, and it performs branch-level caching with synchronized updates across attention and MoE paths [2606.15615].

## 5. Empirical regularities across domains

Several empirical regularities recur across otherwise different Dynamic TMoE formulations. First, harder or more heterogeneous inputs tend to recruit more capacity. The confidence-based routing study reports that BBH receives the highest average number of activated experts among evaluated tasks and that lower layers often activate more experts than upper layers [2403.07652]. DynaMoE reports that descending schedules concentrate early compute and that average active experts per token decline with depth, with an illustrative drop from roughly $3.2$ at the first layer to roughly $1.2$ at the fourth layer in a small image-classification setting [2603.01697].

Second, dynamic routing is most beneficial when domain heterogeneity or regime shift is intrinsic to the task. In Tra-MoE, sparse Top-1 MoE combined with out-of-domain data raises the LIBERO average success rate to $61.4$, whereas dense training with the same OOD augmentation drops to $52.0$ [2411.14519]. In TMoW, routing is updated at test time over world-model adapters, and the paper reports an average unseen-domain success-rate improvement of $+27.21\%$ in zero-shot adaptation [2601.22647]. These results suggest that dynamic routing can function as a mechanism for reconciling specialization with broad data coverage.

Third, temporal non-stationarity consistently motivates dynamic expert management. Dynamic TMoE for time-series forecasting reports state-of-the-art performance on nine benchmarks, with average reductions of $10.4\%$ in MSE and $7.8\%$ in MAE [2605.20678]. STCast’s Temporal Mixture-of-Experts routes inputs according to a month-conditioned discrete Gaussian prior; on global forecasting, removing TMoE or removing month embedding degrades long-horizon skill, while the full model reaches a 10-day RMSE of $0.5763$ and ACC of $0.4715$ [2509.25210].

Fourth, post-training dynamic MoEfication can approach competitive accuracy without full retraining. The dense-to-dynamic DynaMoE framework adapts a pretrained dense LLM into a token-difficulty-driven MoE with a single fine-tuning stage on $10B$ tokens and reports similar aggregated accuracy to Flextron while using only $\frac{1}{9}\text{th}$ of its fine-tuning cost [2502.12325]. DOT-MoE similarly reports retention of about $90\%$ of the original dense model’s performance while reducing active parameters by $50\%$ [2606.01666].

## 6. Limitations, misconceptions, and open problems

A recurring misconception is that Dynamic TMoE always means “variable $K$ per token.” The literature is broader. Some models do vary $K(x)$ explicitly [2603.01697], [2403.07652]; others keep Top-1 or fixed Top-$k$ routing but make expert widths, neuron partitions, expert pools, capacities, bit-widths, or execution plans dynamic [2502.12325], [2606.01666], [2205.01848]. Another misconception is that dynamic routing automatically yields lower end-to-end cost. Several papers emphasize that routing adaptivity shifts complexity into dispatch, caching, recompilation, communication, or offloading, and that these costs must be managed explicitly [2205.01848], [2504.15299].

Scalability remains an open issue. DynaMoE’s architectural formulation does not include capacity-factor caps, token dropping, or auxiliary balancing losses, and the absence of such stabilizers is identified as a limitation for very large-scale settings; its language-modeling experiments are also explicitly pilot-scale, using Recycling-the-Web-1k [2603.01697]. Dynamic recompilation can lose effectiveness if routing remains highly volatile, because frequent recompiles erode the benefit of specialization [2205.01848]. Topology-aware routing depends on accurate topology estimation, and stale or noisy bandwidth estimates can mis-bias the gate [2302.09915]. DOT-MoE avoids heuristic expert partitioning, but its hard capacity applies to neuron counts rather than token counts, so transient token overload is still handled through balancing loss rather than a hard routing cap [2606.01666].

The most general unresolved question is therefore not whether routing should be dynamic, but **which aspect** should be dynamic for a given regime: token-level expert count, layer-wise capacity, expert structure, world-model composition, bit-width, cache state, or communication pattern. The surveyed work collectively indicates that “Dynamic TMoE” is less a single algorithm than a design principle: sparse expert models become substantially more effective when the source of heterogeneity—input difficulty, depth, temporal drift, topology, or memory pressure—is itself made explicit in the routing or execution policy.

Source: https://www.emergentmind.com/topics/dynamic-tmoe