Temporal Mixture-of-Experts (TMoE)

• TMoE are architectures that allocate multiple expert networks to process temporal data, dynamically routing information based on time-specific signals.
• They utilize methods such as patch-level routing, sequence-context routing, and temporal-domain adaptation to enhance forecasting, mobility, and multimodal analysis.
• Empirical studies show improved prediction accuracy and dynamic expert specialization, though challenges remain in routing stability and interpretability.

Temporal Mixture-of-Experts (TMoE) denotes a family of architectures that allocate multiple experts to temporal or sequential data and use temporal context—either directly in the router, indirectly through sequence encoders, or through temporally defined domains or denoising steps—to determine which experts contribute to prediction. Recent literature does not use the term in a single narrow sense. Some models implement explicit patch- or token-level temporal routing, some route from global sequence summaries, some treat time periods as domains, and some make expert weights depend on diffusion timesteps rather than natural sequence time. As a result, TMoE is best understood as a spectrum of designs rather than a single canonical architecture (Wu et al., 26 Sep 2025, Zhu et al., 20 May 2026, He et al., 2024, Lee et al., 24 Jun 2026).

1. Definition and scope

In a narrow sense, a TMoE is a model in which experts specialize over temporal regimes or temporal units and routing varies over time. Patch-level routing across temporal patches with top-$k$ sparse activation is an explicit example of that interpretation, as in PatchMoE, where routing is performed on $(\text{channel},\text{time-patch})$ tokens and regularized separately along temporal and channel axes (Wu et al., 26 Sep 2025). Dynamic TMoE follows the same narrow reading while extending it to non-stationary forecasting through a GRU-based temporal router, MMD-triggered expert growth, and pruning (Zhu et al., 20 May 2026).

A broader usage includes systems in which temporal information conditions expert choice only after it has already been encoded into a global sequence representation. ST-MoE-BERT belongs to this class: it applies a top-2 MoE layer to a [CLS]-style summary produced by a BERT encoder over mobility histories, so routing is sequence-level rather than timestep-level (He et al., 2024). Other adjacent formulations include multilingual temporal-domain adaptation, where time is treated as a sequence of domains and experts are specialized to latent temporal clusters rather than to explicit timesteps (Liu et al., 12 Feb 2025).

The label also extends beyond ordinary time series. In ST-MoME, expert weights vary with diffusion denoising step $t$ and voxel location $u$, so the temporal axis is the diffusion trajectory rather than acquisition time (Lee et al., 24 Jun 2026). By contrast, some architectures that combine MoE with a temporal module are better regarded as neighboring designs rather than true TMoE. MOMENTA, for example, applies modality-specific MoE before a separate temporal aggregation stage; time does not drive expert routing itself (Abdollahinejad et al., 17 Apr 2026).

Formulation	Temporal unit or regime	Representative papers
Explicit temporal or patch routing	Timestamps, patches, or patch tokens	(Wu et al., 26 Sep 2025, Zhu et al., 20 May 2026)
Sequence-level temporal-context routing	Global sequence representation	(He et al., 2024)
Temporal-domain adaptation	Time periods treated as domains	(Liu et al., 12 Feb 2025)
Regime-switched routing	Recurrent vs non-recurrent conditions	(Ke et al., 2024)
Diffusion-time routing	Denoising timestep $t$	(Lee et al., 24 Jun 2026)

2. Architectural elements

Most TMoE systems share a standard MoE backbone in which expert outputs are combined by a gate. A representative dense formulation is

$$\operatorname{MoE}(\mathbf{x}) = \sum_{k=1}^{K} g_k(\mathbf{x}) f_k(\mathbf{x}),$$

with gate weights satisfying

$$\sum_{k=1}^K g_k(\mathbf{x})=1,\qquad g_k(\mathbf{x})\ge 0,$$

and softmax gating

$$g_k(\mathbf{x})=\frac{\exp(\mathbf{w}_k^\top \mathbf{x}+b_k)}{\sum_{j=1}^{K}\exp(\mathbf{w}_j^\top \mathbf{x}+b_j)}.$$

This formulation is stated explicitly in ST-MoE-BERT, where $\mathbf{x}$ is the post-BERT sequence representation and the experts are eight feed-forward networks (He et al., 2024).

Task-conditioned variants use multiple gates over a shared expert bank. TMMOE defines task-specific expert mixtures as

$$y_t^k = h^k \big( f^k(x_t) \big), \qquad f^k(x_t)= \sum_{i=1}^{n} g^k(x_t)_i \, f_i(x_t),$$

with a separate gate for each task. In that model, temporal structure appears throughout the stack: a TCN shared layer extracts temporal features, each expert is an LSTM, and each task gate is also implemented with LSTM equations (Yuan et al., 2023).

A second architectural axis concerns what constitutes an expert. Some papers use homogeneous FFN experts, as in ST-MoE-BERT and PatchMoE (He et al., 2024, Wu et al., 26 Sep 2025). Others use heterogeneous experts with different inductive biases. Dynamic TMoE explicitly defines identity, trend, seasonality, and fluctuation experts: $(\text{channel},\text{time-patch})$0

$(\text{channel},\text{time-patch})$1

$(\text{channel},\text{time-patch})$2

$(\text{channel},\text{time-patch})$3

This makes expert identity part of the model’s temporal bias rather than merely a routing consequence (Zhu et al., 20 May 2026).

A third axis is whether experts are full subnetworks, lightweight linear operators, or modality-bound branches. SPMTrack replaces attention and FFN linear layers with a TMoE module consisting of one frozen shared expert, one compression expert, and multiple routed linear experts, all combined by dense soft routing (Cai et al., 24 Mar 2025). ST-MoME uses one expert per MRI modality; these are modality-specific 3D CNN encoders rather than interchangeable experts sharing one input distribution (Lee et al., 24 Jun 2026).

3. Temporal routing mechanisms

The defining question for TMoE is not whether a model contains experts, but how temporal information influences routing. Existing work exhibits several distinct routing mechanisms.

Sequence-level routing uses a temporally encoded global representation as the router input. ST-MoE-BERT processes a 240-step mobility history with spatial and temporal embeddings, extracts the [CLS] token, and routes that sequence summary to the top two of eight FFN experts. Temporal information affects expert selection only indirectly through the BERT encoder, not through token-level or timestep-level dispatch (He et al., 2024).

Patch-level routing performs routing on explicit temporal units. PatchMoE operates on flattened $(\text{channel},\text{time-patch})$4 patch tokens, where $(\text{channel},\text{time-patch})$5 is the number of channels and $(\text{channel},\text{time-patch})$6 the number of temporal patches. Its Recurrent Noisy Gating shares a GRU-based router across layers: $(\text{channel},\text{time-patch})$7 followed by Gaussian noisy logits and top-$(\text{channel},\text{time-patch})$8 sparsification,

$(\text{channel},\text{time-patch})$9

Because routing is defined over patch tokens and regularized by both Temporal Load Balancing Loss and Channel Load Balancing Loss, PatchMoE is one of the clearest examples of a joint channel-temporal sparse MoE (Wu et al., 26 Sep 2025).

Interaction-aware routing extends temporal routing to multimodal settings. Time-MoE first computes lagged redundancy, uniqueness, and synergy signals from directed information, then uses them to build router context: $t$0

$t$1

Here routing depends on lagged temporal multimodal interaction structure rather than only on current token content (Han et al., 30 Sep 2025).

Temporal-domain routing treats time periods as domains rather than local positions. MoTE partitions source corpora into ordered time domains, clusters source representations, and routes each instance by a top-$t$2 gate,

$t$3

while also injecting centroid-relative shift vectors

$t$4

into expert computation. This makes expert assignment instance-specific but anchored to latent temporal-domain structure (Liu et al., 12 Feb 2025).

Hard regime routing uses externally defined temporal conditions. In traffic prediction under incident and non-incident settings, one expert is specialized to recurrent traffic and another to non-recurrent traffic, with routing effectively determined by the incident-defined regime indicator $t$5 in

$t$6

This is a temporal regime MoE without a learned soft gate (Ke et al., 2024).

Drift-aware routing adds sequential memory and expert evolution. Dynamic TMoE updates router state as

$t$7

then performs sparse routing

$t$8

When drift is detected, archived anomaly states can be fused into routing via

$t$9

This is an explicit temporal-memory router rather than a memoryless gate (Zhu et al., 20 May 2026).

A distinct temporal notion appears in diffusion. ST-MoME computes modality weights at every voxel and denoising step: $u$0 The temporal variable is diffusion step $u$1, not sequence time, but expert allocation is nonetheless explicitly time-dependent (Lee et al., 24 Jun 2026).

4. Representative model families and applications

TMoE research spans time-series forecasting, classification, traffic modeling, mobility analysis, multimodal learning, tracking, and diffusion-based generation. The diversity of application areas explains much of the variation in what different papers mean by “temporal.”

Model	Temporal unit or regime	Defining feature
PatchMoE (Wu et al., 26 Sep 2025)	Patch tokens over time and channels	Recurrent Noisy Gating with temporal/channel balancing
Dynamic TMoE (Zhu et al., 20 May 2026)	Patch-level routing with temporal memory	MMD-triggered expert growth and pruning
TMMOE (Yuan et al., 2023)	Shared temporal sequence for three tasks	TCN shared layer with LSTM experts and gates
MoTE (Liu et al., 12 Feb 2025)	Time domains	Cluster-based shift evaluator and top-$u$2 router
ST-MoME (Lee et al., 24 Jun 2026)	Diffusion timestep and voxel	Modality experts with masked softmax fusion

In general time-series analytics, PatchMoE is designed as a single cross-task framework for univariate forecasting, multivariate forecasting, anomaly detection, imputation, and classification, and argues that different tasks require different representational depths (Wu et al., 26 Sep 2025). Dynamic TMoE focuses on non-stationary multivariate forecasting and frames TMoE as a response to regime drift, evolving expert pools, and memory-aware dispatch (Zhu et al., 20 May 2026). TMMOE addresses joint vehicle trajectory prediction and driving intention classification, showing a multi-task temporal MoE in which temporal inductive bias appears in the shared encoder, the experts, and the gates (Yuan et al., 2023).

Traffic and mobility form a second major cluster. TESTAM uses three experts—temporal-only, static-graph spatio-temporal, and dynamic-graph spatio-temporal—and routes among them with pseudo-label classification losses to capture recurring and non-recurring traffic (Lee et al., 2024). A separate traffic-speed model hard-switches between recurrent and non-recurrent TFT experts based on incident-defined temporal regimes (Ke et al., 2024). TFMoE addresses continual traffic forecasting on evolving sensor networks by routing sensors to experts using reconstruction-based weekly traffic signatures rather than timestep-level temporal regimes (Lee et al., 2024). ST-MoE-BERT applies a spatial-temporal MoE-augmented BERT to long-horizon cross-city mobility prediction, but its experts are selected from a global sequence representation rather than from individual temporal tokens (He et al., 2024).

Multimodal and generative variants broaden the field further. Time-MoE uses lagged multimodal interaction dynamics—redundancy, uniqueness, and synergy—to guide routing in multimodal sequential prediction (Han et al., 30 Sep 2025). MoME keeps temporal experts primary while letting text modulate both router scores and expert outputs for multimodal forecasting (Zhang et al., 29 Jan 2026). SPMTrack uses a tracking-specific TMoE inside transformer linear layers while extending the input from image pairs to spatio-temporal reference context (Cai et al., 24 Mar 2025). VidPrism is a heterogeneous temporal MoE for image-to-video transfer, where experts are bound to temporally differentiated streams rather than selected by sparse token routing (Lin et al., 27 May 2026). By contrast, MOMENTA combines modality-specific MoE with a downstream temporal aggregation mechanism and is therefore better classified as a multimodal MoE with a separate temporal module (Abdollahinejad et al., 17 Apr 2026).

5. Objectives, evaluation, and empirical behavior

Training objectives vary with task, but a recurrent pattern is that TMoE models combine standard prediction losses with routing-specific or specialization-specific regularizers. ST-MoE-BERT formulates mobility forecasting as multi-step classification over a 40,000-class location vocabulary and uses cross-entropy over the future horizon. On cross-city prediction, it reports average improvements of 10.30% in GEO-BLEU, 15.50% in DTW, and 21.40% in accuracy across the three cities relative to benchmark methods; transfer learning yields 8.29% in GEO-BLEU, 9.90% in DTW, and 10.76% in accuracy in the ablation against the non-pretrained variant (He et al., 2024).

PatchMoE combines task loss with temporal and channel balancing losses rather than using only a standard batch-level load-balancing term. Its empirical range is broad: on univariate forecasting it reports best average results of msMAPE $u$3 and MASE $u$4; on 10 UEA datasets it reports average classification accuracy of $u$5; and its ablation attributes about 4.2% MSE reduction to the Recurrent Noisy Gating component, up to 6.9% MSE reduction on Traffic to shared experts, and consistent gains to the temporal/channel balancing loss (Wu et al., 26 Sep 2025).

Dynamic TMoE is trained with MSE and evaluates with MSE and MAE. Across nine benchmarks it reports average reductions of 10.4% in MSE and 7.8% in MAE, and an improvement over TFPS of 5.9% average MSE reduction and 3.6% average MAE reduction. The ablations attribute increased error to replacing the GRU router with non-temporal alternatives, to removing expert heterogeneity, to disabling MMD-based adaptation, and to dropping pruning or post-addition alignment (Zhu et al., 20 May 2026).

Other families show related patterns. TMMOE uses cross-entropy for intention classification, mean-squared loss for longitudinal and lateral regression, and homoscedastic uncertainty weighting across tasks; at a 6-second input length it reports classification accuracy 0.9544 versus 0.8796 for an LSTM baseline, longitudinal RMSE 2.323 versus 8.280, and lateral RMSE 0.2963 versus 0.6274 (Yuan et al., 2023). ST-MoME is trained end-to-end with the conditional DDPM loss only, without auxiliary MoE balancing, and reports average aggregated NMSE values of 1.39 for $u$6, 2.30 for $u$7, and 0.82 for $u$8, as well as tumor-region NMSE 3.982 versus 5.035 for ShaSpec under the full-modality setting (Lee et al., 24 Jun 2026). Time-MoE reports strong gains on PAMAP2, where it reaches 91.37 accuracy and 90.44 F1, and on MIMIC-IV in-hospital mortality, where it reaches AUROC 85.40 and F1 84.97, but it is not uniformly best on every dataset, as the MOSI table remains numerically stronger for FuseMoE (Han et al., 30 Sep 2025).

The empirical record therefore supports two cautious conclusions. First, expert specialization and temporally informed routing are often beneficial. Second, gains are not universal, and several papers report regime- or dataset-specific exceptions rather than uniform dominance.

6. Limitations, ambiguities, and research directions

A central ambiguity in the literature is definitional. Not every spatio-temporal MoE is a narrow TMoE. ST-MoE-BERT is a true MoE applied to a temporal prediction task, but its routing is sequence-level rather than timestep-level (He et al., 2024). MOMENTA contains MoE and temporal modeling, yet time affects only downstream aggregation, not the expert gate (Abdollahinejad et al., 17 Apr 2026). ST-MoME is explicitly temporal in diffusion time, but not in ordinary sequence time (Lee et al., 24 Jun 2026). This suggests that the term “temporal” may refer to natural chronology, temporal domains, patch order, regime labels, or denoising trajectories, depending on the paper.

A second limitation concerns expert specialization itself. Several models assert that experts capture distinct aspects of temporal structure, but the operational evidence is uneven. ST-MoE-BERT does not report routing-loss analysis, capacity control, or an expert utilization study (He et al., 2024). VidPrism identifies “expert homogenization” as a failure mode in video MoE and responds with multi-rate pathway specialization, diversity loss, and gate-balancing loss, which suggests that specialization rarely emerges automatically from expert multiplicity alone (Lin et al., 27 May 2026). MoTE gains from cluster-based temporal experts, but its time representation is discretized into domains and the alignment between clusters and genuine temporal regimes can remain imperfect (Liu et al., 12 Feb 2025).

A third issue is routing stability versus flexibility. PatchMoE introduces temporal/channel load balancing precisely because standard routers may collapse along one axis or fail to model cross-channel structure (Wu et al., 26 Sep 2025). Dynamic TMoE argues that memoryless routers are too unstable under non-stationarity, but it pays for temporal continuity with sequential routing, MMD overhead, and hyperparameter sensitivity around thresholds, pool size, and pruning (Zhu et al., 20 May 2026). ST-MoME avoids sparse routing altogether and instead uses dense mask-aware soft fusion, which is principled for missing modalities but does not address sparse conditional computation or expert-capacity balancing (Lee et al., 24 Jun 2026).

Open directions are correspondingly varied. PatchMoE explicitly suggests better capacity-aware temporal/channel routing, stronger theoretical grounding for hierarchical task-aware routing, extension beyond channel-independent backbones, and richer sequence-level routing decisions (Wu et al., 26 Sep 2025). Dynamic TMoE proposes online learning and test-time adaptation, parameter-efficient adaptation of newly created experts, and integration with time-series foundation models (Zhu et al., 20 May 2026). More broadly, the literature suggests three unresolved questions: how to define temporal expert identity in a way that is both interpretable and general; how to maintain stable expert usage without suppressing meaningful specialization; and how to unify temporal routing, structural adaptation, and multimodal context without reducing TMoE to either a static expert bank or a loosely coupled temporal add-on.

Taken together, these works establish TMoE less as a single architecture than as a research program: one centered on the proposition that temporal heterogeneity—whether expressed as patches, regimes, domains, interaction lags, or diffusion steps—should be matched by differentiated expert computation and by routing mechanisms that are themselves temporally informed.