Temporal Mixture of Experts (T-MoE)

• Temporal Mixture of Experts (T-MoE) is a family of sparse architectures that conditionally activates specialized neural experts based on temporal data patterns.
• It employs diverse routing mechanisms—including token-wise, segment-wise, and dynamic gating—to balance efficiency, scalability, and task-specific performance.
• T-MoE models enhance forecasting, anomaly detection, and multimodal integration by leveraging temporal hierarchies, structural priors, and adaptive expert pools.

Temporal Mixture of Experts (T-MoE) denotes a family of sparse conditional-computation architectures in which expert selection is adapted to temporal structure rather than treated as a generic token-routing problem. In the time-series literature, the label is used for several related but non-identical designs: decoder-only forecasting foundation models with sparse expert FFNs, encoder-only Transformers with segment-wise routing, task-aware routers that use hierarchical temporal representations, structure-guided routers regularized by interpretable series descriptors, drift-aware systems with dynamic expert pools, and multimodal variants in which routing or expert computation is conditioned on text or quantified cross-modal interaction dynamics (Shi et al., 2024, Liu et al., 2024, Wu et al., 26 Sep 2025, Ortigossa et al., 29 Jan 2026, Wang et al., 24 May 2026, Zhu et al., 20 May 2026, Zhang et al., 29 Jan 2026, Han et al., 30 Sep 2025). Across these variants, the shared premise is that temporal data are heterogeneous, non-stationary, and often multi-scale; sparse expert activation is therefore used to increase capacity and specialization without incurring dense computation on every token, patch, segment, or series.

1. Terminological scope and representative formulations

Within recent arXiv usage, T-MoE is not restricted to a single backbone or routing rule. Time-MoE scales a decoder-only forecasting foundation model to 2.4 billion parameters using sparse MoE sublayers and a Time-300B corpus of approximately 309 billion time points across 9 broad domains (Shi et al., 2024). Moirai-MoE likewise inserts sparse experts into a decoder-only Transformer, but emphasizes token-level specialization with a single shared input/output projection and a cluster-based gate derived from $k$-means on pretrained representations (Liu et al., 2024). PatchMoE uses a Transformer whose FFN is replaced by a T-MoE block with shared experts, routed experts, a Recurrent Noisy Gating router shared across layers, and dual temporal-channel load balancing for forecasting, anomaly detection, imputation, and classification (Wu et al., 26 Sep 2025).

Other formulations change the routing unit or the conditioning signal. Seg-MoE routes contiguous segments rather than individual tokens, using segment lengths $\omega_b$ that may vary by layer in a multi-resolution design (Ortigossa et al., 29 Jan 2026). AME-TS guides token-level routing by a series-level structural prior over experts derived from forecastability, seasonality, trend, and sparsity descriptors (Wang et al., 24 May 2026). Dynamic TMoE augments sparse routing with Maximum Mean Discrepancy drift detection, a GRU-based temporal memory router, heterogeneous experts, and expert instantiation and pruning during training (Zhu et al., 20 May 2026). In multimodal settings, MoME can be viewed as a T-MoE in which both routing and expert computation are conditioned directly on auxiliary textual signals (Zhang et al., 29 Jan 2026), while the temporally guided multimodal Time-MoE routes according to redundancy, uniqueness, and synergy trajectories derived from directed-information decompositions (Han et al., 30 Sep 2025).

System	Distinguishing routing unit or conditioning	Reported scope
Time-MoE	token-level Top-$k$ sparse routing in a decoder-only Transformer	forecasting foundation model
Moirai-MoE	token-level sparse routing with linear or cluster-based gating	zero-shot and in-distribution forecasting
PatchMoE	recurrent noisy gating with temporal-channel balancing	forecasting, anomaly detection, imputation, classification
Seg-MoE	contiguous segment-wise routing, possibly multi-resolution	multivariate long-term forecasting
AME-TS	token routing guided by a series-level structural prior	sparse time series foundation modeling
Dynamic TMoE	GRU memory router with drift-triggered expert evolution	non-stationary forecasting
MoME	router modulation and expert modulation by text	multi-modal time series prediction
Time-MoE (multimodal interactions)	routing guided by temporal RUS interaction signals	multimodal classification and regression

This usage suggests that “temporal” in T-MoE has at least four distinct meanings in the current literature: temporal tokens or patches as the routed objects, temporal continuity in the routing decision, temporal scale as the routing granularity, and temporally varying side-information that modulates routing.

2. Shared architectural template

A common template replaces the dense FFN sublayer in a Transformer block by a sparse mixture: $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$ where $z_t$ is a token, patch, or segment representation, $g_i$ is a gate, $\mathcal S(z_t)$ is typically a Top-$k$ subset, and $E_i$ is an expert network. In Time-MoE, the base model is a decoder-only Transformer for univariate autoregressive forecasting; each scalar $x_t\in\mathbb R$ is embedded by a pointwise SwiGLU into a $\omega_b$0-dimensional hidden state, then passed through stacked causal self-attention layers interleaved with sparse MoE sublayers (Shi et al., 2024). Moirai-MoE adopts the same high-level decoder-only pattern, dividing a raw series of length $\omega_b$1 into $\omega_b$2 non-overlapping patches of size $\omega_b$3, normalizing each patch with a causal normalizer, and mapping it to a $\omega_b$4-dimensional token via a residual MLP projection (Liu et al., 2024).

Encoder-style T-MoE designs preserve the same substitution pattern while changing the upstream temporal representation. Seg-MoE uses an encoder-only Transformer for multivariate long-term forecasting; each look-back window $\omega_b$5 is embedded, split into $\omega_b$6 non-overlapping patches of length $\omega_b$7 per channel, and processed by stacked blocks in which the dense FFN is replaced by a sparse MoE layer (Ortigossa et al., 29 Jan 2026). PatchMoE similarly starts from $\omega_b$8, applies reversible instance-norm, splits the normalized input into contiguous patches per channel, linearly embeds them into $\omega_b$9, and feeds them through $k$0 Transformer layers whose feed-forward modules are replaced by T-MoE blocks (Wu et al., 26 Sep 2025).

Expert architectures differ by design objective. Time-MoE and Moirai-MoE use standard FFN-style experts (Shi et al., 2024, Liu et al., 2024). PatchMoE uses ordinary two-layer ReLU MLP shared experts together with routed experts (Wu et al., 26 Sep 2025). Seg-MoE defines each expert as an FFN mapping $k$1, because the routed object is a segment rather than a token (Ortigossa et al., 29 Jan 2026). Dynamic TMoE departs most strongly from homogeneous expert pools by defining Identity, Trend, Seasonality, and Fluctuation experts with distinct inductive biases, including FFT/iFFT processing for seasonality and 1D causal convolution plus GLU for fluctuation (Zhu et al., 20 May 2026). MoME is explicitly backbone-agnostic: in its primary instantiation each expert is a small MLP,

$k$2

but alternate backbones include MMLinear and MiTransformer (Zhang et al., 29 Jan 2026).

3. Routing granularity and conditioning mechanisms

The principal technical variation across T-MoE systems lies in what is routed and what information the router is allowed to use. Standard token-wise Top-$k$3 sparse routing remains the baseline. Time-MoE computes

$k$4

so that only $k$5 experts are activated per token (Shi et al., 2024). Moirai-MoE uses an analogous token-level mechanism, but compares a standard linear gate,

$k$6

with a cluster-based gate,

$k$7

where $k$8 and $k$9 are centroids obtained by $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$0-means on pretrained dense representations (Liu et al., 2024).

A recurrent misconception is to equate T-MoE with this token-wise formulation alone. Seg-MoE instead groups contiguous time-step patches into non-overlapping segments of length $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$1, flattens each segment $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$2 into $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$3, and routes the segment as one unit through a linear gate with Softmax scores followed by hard Top-$$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$4 selection (Ortigossa et al., 29 Jan 2026). The same paper also introduces an always-active shared expert with gate $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$5. When $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$6 everywhere, Seg-MoE reduces to standard token-wise MoE (Ortigossa et al., 29 Jan 2026).

Other systems preserve token-level routing but condition it on temporal hierarchy, series structure, or side-information. PatchMoE’s RNG-Router is shared across all $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$7 MoE layers and conditions routing at layer $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$8 on the entire hierarchy of previous layers through a GRU: $$y_t=\sum_{i\in \mathcal S(z_t)} g_i(z_t)\,E_i(z_t),$$9 followed by Gaussian heads, differentiable noisy gating during training, KeepTopK, and Softmax (Wu et al., 26 Sep 2025). AME-TS augments the standard token-level router $z_t$0 with a series-level prior $z_t$1 over specialized and shared experts; the prior is constructed from forecastability, seasonality strength, trend strength, and sparsity estimated by four separate CNN-plus-attention regressors of approximately 450 K parameters each (Wang et al., 24 May 2026). This prior is used only during training as a structural guide; at inference AME-TS uses the learned router alone (Wang et al., 24 May 2026).

Dynamic TMoE further adds temporal continuity and architectural plasticity. Its temporal memory router updates a GRU state,

$z_t$2

projects $z_t$3 to logits over the current expert pool, applies Top-$z_t$4 sparse gating, and fuses the state with a prototype retrieved from an anomaly-state repository by soft attention (Zhu et al., 20 May 2026). On detected drift, a Drift Pattern Profiler chooses one of three expert types—Trend, Seasonality, or Fluctuation—to instantiate, while persistently underused experts are pruned (Zhu et al., 20 May 2026).

Multimodal T-MoE variants extend routing beyond temporal content alone. In MoME, a pooled textual embedding $z_t$5 is produced from raw tokens $z_t$6 by a frozen QwenMoE_A2.7B plus a small cross-attention QueryPool and then injected directly into routing: $z_t$7 with Top-$z_t$8 masks applied to the resulting unnormalized energies (Zhang et al., 29 Jan 2026). In the multimodal interaction-guided Time-MoE, the router incorporates a temporal context built from redundancy and synergy trajectories through an attention module and uniqueness trajectories through a GRU: $z_t$9 after which an MLP produces routing logits over experts (Han et al., 30 Sep 2025).

4. Objectives, regularization, and theoretical accounts

Training objectives in T-MoE systems reflect both the downstream task and the need to prevent routing collapse. Time-MoE uses an autoregressive forecasting objective and in practice adopts the Huber loss for robustness to outliers, combined with an auxiliary gating/load-balancing term $g_i$0, where $g_i$1 is the fraction of tokens for which expert $g_i$2 is active and $g_i$3 is the average router probability assigned to that expert (Shi et al., 2024). Seg-MoE uses Huber prediction loss over the forecast horizon and a load-balancing loss

$g_i$4

with $g_i$5 defined from average Softmax scores and $g_i$6 from Top-$g_i$7 frequencies over routed segments (Ortigossa et al., 29 Jan 2026). Moirai-MoE adopts the standard MoE load loss from GShard/Switch,

$g_i$8

to align each expert’s hard-assignment share with its soft-probability share (Liu et al., 2024).

PatchMoE makes the regularization explicitly two-dimensional. Its temporal load balancing and channel load balancing terms are defined on the $g_i$9 token grid, and combined as

$\mathcal S(z_t)$0

with the total objective given by the task loss plus $\mathcal S(z_t)$1 (Wu et al., 26 Sep 2025). AME-TS adds two other regularizers: a forward KL prior-alignment term between token-level router distributions and the structural prior $\mathcal S(z_t)$2, with layer weights $\mathcal S(z_t)$3, and an orthogonality loss among co-activated experts in the same group (Wang et al., 24 May 2026). Dynamic TMoE, by contrast, uses only the standard forecasting loss and an optional weight-decay term; the drift mechanism is not implemented as a differentiable penalty but as a training-time control process that activates instantiation and pruning when MMD exceeds a dynamic threshold (Zhu et al., 20 May 2026). The multimodal interaction-guided Time-MoE combines the task loss with three auxiliary objectives: a redundancy loss encouraging similar router distributions for highly redundant modalities, a uniqueness loss encouraging diverse distributions for highly unique modalities, and a synergy loss pushing high-synergy tokens toward dedicated synergy experts (Han et al., 30 Sep 2025).

Theoretical accounts in this area are still heterogeneous rather than unified. Time-MoE reports that forecasting error follows power-law scaling with model size and data size, and its ablations indicate that doubling model size from 113 M to 453 M to 2.4 B or increasing data size consistently yields approximately 5–10% relative MSE reductions (Shi et al., 2024). MoME provides the most explicit theorem-level analysis. It states that a dense MLP layer with $\mathcal S(z_t)$4 hidden units can be algebraically decomposed into $\mathcal S(z_t)$5 disjoint sub-MLPs, so that $\mathcal S(z_t)$6 is equivalent to an MoE with fixed routing $\mathcal S(z_t)$7 (Zhang et al., 29 Jan 2026). It also proves a sparse-routing bound resembling truncated PCA: if $\mathcal S(z_t)$8 is the Top-$\mathcal S(z_t)$9 active set, $k$0, and the mutual coherence satisfies $k$1, then

$k$2

and the bound decreases monotonically as $k$3 grows (Zhang et al., 29 Jan 2026). This theorem is used there to interpret sparse MoE as an energy-based denoising mechanism.

5. Empirical performance and observed specialization

The empirical record for T-MoE is broad rather than uniform, because different papers optimize different tasks and scales. On large-scale foundation-model forecasting, Time-MoE reports average zero-shot MSE at horizon 96 of approximately 0.262 for Time-MoE ultra, compared with approximately 0.336 for dense baselines including Moirai, TimesFM, and Chronos$k$4; after one epoch of fine-tuning, Time-MoE ultra reports average MSE 0.301 versus 0.356 for strong dense models (Shi et al., 2024). The same work reports approximately 78% lower training FLOPs and approximately 39% lower inference FLOPs than a dense Transformer with the same number of activated parameters (Shi et al., 2024). Moirai-MoE reports a 17% relative MAE reduction for its Small model versus Moirai Small and an 8% reduction for its Base model versus Moirai Base on the Monash Benchmark, and states that T-MoE Base outperforms larger dense Moirai-Large and Chronos-Large while activating up to 65× fewer parameters; on 10 unseen datasets it reports the best average normalized CRPS of 0.478 and MASE of 0.651 for T-MoE Small (Liu et al., 2024).

On multivariate long-term forecasting, Seg-MoE reports that it consistently achieves the lowest average MSE across all datasets. The paper gives ETTh1 average MSE over four horizons as 0.381 versus 0.449 for the best dense or token-wise MoE baseline, and Weather average MSE as 0.223 versus 0.255 (Ortigossa et al., 29 Jan 2026). Ablations show that routing at $k$5 or $k$6 outperforms token-wise MoE by 3–4% in MSE on most datasets, while multi-resolution routing adds approximately 1–2% further MSE gains over the best single-$k$7 model (Ortigossa et al., 29 Jan 2026). On multi-task time-series analytics, PatchMoE reports univariate forecasting msMAPE $k$8 versus 20.77% for the best prior method and MASE $k$9 versus 2.02 on 8,068 TFB series; in multivariate forecasting it reports, among other results, Electricity MSE $E_i$0 versus 0.171 and Solar MSE $E_i$1 versus 0.200; it also reports average anomaly-detection F1 $E_i$2, AUC $E_i$3, average imputation MSE $E_i$4, and average classification accuracy $E_i$5 (Wu et al., 26 Sep 2025).

Structure-guided and dynamic variants emphasize robustness and interpretability under heterogeneity. AME-TS reports on GIFT-Eval normalized MASE that Chronos-Small scores 0.892, AME-TS Tiny scores 0.856, Chronos-2 scores 0.698, AME-TS Large scores 0.700, and AME-TS Ultra scores 0.692; on M5 it reports WRMSSE 0.613 in zero-shot mode and 0.506 after fine-tuning, compared with 0.520 for the rank 1 competition system (Wang et al., 24 May 2026). Dynamic TMoE reports average MSE reduction of 10.4% and MAE reduction of 7.8% across nine datasets and four horizons relative to nine state-of-the-art baselines, together with a 5.9% MSE and 3.6% MAE improvement over the strongest MoE baseline TFPS (Zhu et al., 20 May 2026). It also reports top-2 ranking on 16 out of 18 benchmark/horizon settings and first place on 11 of them (Zhu et al., 20 May 2026).

Multimodal T-MoE results are similarly task-specific. MoME reports that removing expert modulation $E_i$6 leads to a 2–5% average performance drop, removing router modulation causes a minor and more task-dependent drop, sparse activation with $E_i$7 consistently outperforms the dense variant $E_i$8, and early or late fusion underperform while using 2–10× more memory or slower training (Zhang et al., 29 Jan 2026). The temporally guided multimodal Time-MoE reports, for example, PAMAP2 accuracy 91.4 and F1 90.4, MIMIC-IHM AUROC 85.4 and F1 85.0, and MOSI accuracy 72.0, outperforming Transformer, MulT, FuseMoE, and I2MoE on the reported metrics (Han et al., 30 Sep 2025).

Across these papers, expert specialization is not merely asserted but probed. Moirai-MoE reports t-SNE evidence that tokens from different frequencies but similar patterns overlap more strongly than under frequency-specific projections, that deeper layers converge toward a few common experts across all frequencies, and that Traffic-Hourly shows periodic expert-assignment patterns aligned with rush peaks and troughs (Liu et al., 2024). AME-TS reports clearer regime-aligned clusters in t-SNE and Calinski–Harabasz indices greater than $E_i$9 in encoder space, together with routing-consistency values that remain approximately $x_t\in\mathbb R$0 under fine-tuning, compared with approximately $x_t\in\mathbb R$1 for a standard MoE (Wang et al., 24 May 2026).

6. Interpretation, misconceptions, limitations, and future directions

The most common misconception is that T-MoE is simply standard sparsely gated MoE transplanted unchanged from NLP into time series. The recent literature points in a different direction. Seg-MoE argues that aligning routing granularity with contiguous temporal segments provides a powerful inductive bias (Ortigossa et al., 29 Jan 2026). PatchMoE treats the $x_t\in\mathbb R$2 tokens as an $x_t\in\mathbb R$3 grid and uses balancing losses along both temporal and channel axes because per-token routing is its only mechanism for channel fusion under channel-independent self-attention (Wu et al., 26 Sep 2025). AME-TS breaks expert symmetry with interpretable series-level priors, and Dynamic TMoE relaxes the assumption of a fixed expert pool by adding training-time expert growth and pruning (Wang et al., 24 May 2026, Zhu et al., 20 May 2026). In multimodal settings, MoME and interaction-guided Time-MoE further show that temporal side-information can modulate routing directly rather than being merged through token-level fusion alone (Zhang et al., 29 Jan 2026, Han et al., 30 Sep 2025).

A second misconception is that sparsity is valuable only for efficiency. The papers summarized here repeatedly attach functional significance to sparse activation. Time-MoE presents scaling-law and efficiency evidence that sparse experts enable larger-capacity forecasting foundation models at reduced activated cost (Shi et al., 2024). MoME explicitly interprets Top-$x_t\in\mathbb R$4 truncation as energy-based denoising through its error bound (Zhang et al., 29 Jan 2026). Moirai-MoE’s analysis of progressive routing concentration in deeper layers likewise suggests that sparsity may act as a temporal filtering mechanism rather than merely a compute-saving device (Liu et al., 2024). These accounts do not constitute a single unified theory, but together they suggest that expert sparsity can be tied to representation selection, denoising, and specialization.

The main limitations are equally clear in the current literature. Dynamic TMoE notes that MMD drift detection is $x_t\in\mathbb R$5 in window size, that the framework introduces many hyperparameters, and that new experts undergo a cold-start period (Zhu et al., 20 May 2026). AME-TS currently uses only historical series structure and explicitly identifies multimodal expert anchoring as future work (Wang et al., 24 May 2026). Seg-MoE identifies adaptive segment lengths, overlapping segments, heterogeneous expert architectures, and pre-training on billion-point corpora as natural extensions (Ortigossa et al., 29 Jan 2026). Dynamic TMoE likewise points to parameter-efficient fine-tuning, online adaptation at test time, and integration into large time-series foundation models (Zhu et al., 20 May 2026). MoME’s comparison with token-level fusion suggests a broader multimodal design question: whether cross-modal interaction should be performed in a shared token space or by direct control of routing and expert computation (Zhang et al., 29 Jan 2026).

Taken together, these works suggest that T-MoE is best understood as a design space organized around three axes: the routed temporal unit, the conditioning signal available to the router, and the degree of structural adaptation permitted in the expert pool. Current systems instantiate these axes in markedly different ways—token-wise versus segment-wise routing, static versus recurrent versus prior-guided gates, homogeneous versus heterogeneous or dynamically evolving experts, and unimodal versus text- or interaction-conditioned routing—yet all are attempts to match sparse conditional computation to the distinctive locality, continuity, heterogeneity, and non-stationarity of temporal data.