Dynamic TMoE: A Drift-Aware Dynamic Mixture of Experts Framework for Non-Stationary Time Series Forecasting

Abstract: Non-stationary time series forecasting is challenged by evolving distribution shifts that static models struggle to capture. While Mixture-of-Experts (MoE) architectures offer a promising paradigm for decoupling complex drift patterns, existing approaches are limited by fixed expert pools and memoryless routing, hampering their ability to adapt to abrupt regime shifts. To address this, we propose Dynamic TMoE, a framework that unifies architectural evolution with temporal continuity during learning phase. By detecting distribution shifts via Maximum Mean Discrepancy (MMD), we dynamically instantiate heterogeneous experts and prune redundant ones to optimize capacity. Additionally, a temporal memory router leverages recurrent states and an anomaly repository to ensure stable, context-aware expert selection without requiring test-time updates. Experiments on nine benchmarks demonstrate state-of-the-art performance, reducing MSE by 10.4% and MAE by 7.8%. Code is available at https://github.com/andone-07/Dynamic-TMoE.

• The paper introduces a dynamic mixture-of-experts framework that employs an MMD-based drift detector and GRU routing to adapt to distribution shifts.
• It integrates specialized experts for identity, trend, seasonality, and fluctuation, alongside cyclic relation modeling, to capture heterogeneous temporal features.
• Empirical results across nine benchmarks demonstrate reduced MSE and MAE, highlighting the model's superior efficiency and adaptability over state-of-the-art baselines.

Dynamic TMoE: A Drift-Aware Dynamic Mixture of Experts for Non-Stationary Time Series Forecasting

Motivation and Problem Formulation

Non-stationary time series forecasting remains a central challenge due to distribution shifts and abrupt regime transitions in real-world data. Classical and deep architectures, even those equipped with external normalization (e.g., RevIN, FAN, IN-Flow) or model-internal adaptation (e.g., AdaRNN, TimeStacker, Non-stationary Transformers), primarily employ monolithic structures and lack fine-grained mechanisms to decouple heterogeneous temporal drifts. The Mixture-of-Experts (MoE) paradigm offers a modular solution, yet prior MoE methods deploy fixed expert pools and stateless routers, fundamentally restricting adaptability and limiting the ability to address dynamic, co-occurring distributions.

Methodology

Dynamic TMoE introduces a comprehensive architectural evolution approach for non-stationary time series via:

1. Distribution-Driven Expert Evolution:

A continuous Maximum Mean Discrepancy (MMD)-based detector monitors divergence between historical and current windows in a kernelized space, dynamically triggering expert pool expansion or pruning when distribution shifts surpass an adaptive k-sigma threshold. This ensures that model capacity scales with drift complexity.

2. Temporal Memory Routing:

Expert assignment is performed by a GRU-based sequential router that maintains a hidden state vector, enabling temporally coherent gating. Upon drift detection, the router integrates an anomaly state repository—storing historical router states associated with prior drifts—facilitating rapid adaptation to recurring regimes through gated fusion between the current and archived states. Routing adopts a scalable Top-k dispatch strategy, keeping the computation for activated experts efficient.

3. Heterogeneous Expert Pool Design:

Rather than homogeneous networks, Dynamic TMoE instantiates a set of specialized experts, each tailored to a canonical temporal feature:

• Identity expert: Linear pass-through for stable information propagation.
• Trend expert: Average pooling with MLP projection for low-frequency dynamics.
• Seasonality expert: Fast Fourier Transform (FFT)–based spectral modeling, encoding periodicity with explicit sinusoids.
• Fluctuation expert: 1D causal convolutions with Gated Linear Units to capture high-frequency volatility.

Cyclic relation modeling, inspired by CycleNet, is incorporated for multivariate dependency tracking. Experts leverage both static and learned periodic prototypes for dynamic adjacency matrix reconstruction.

4. Evolvable Expert Manager:

Lifecycle management ensures:

• Instantiation of new experts upon drift, with selection informed by a Drift Pattern Profiler decomposing error residuals into trend, seasonality, or fluctuation dominance.
• Post-addition alignment procedure, where only new expert and router head are fine-tuned on the concatenated reference/current drift windows, guaranteeing stable integration.
• An expert usage tracker prunes low-utility experts under a patience constraint, preventing unnecessary redundancy.

Empirical Results

Dynamic TMoE was validated across nine real-world benchmarks, including ETT, Electricity, Traffic, Weather, Exchange, and ILI datasets, reflecting a spectrum of non-stationarity and temporal complexity. Against nine SOTA baselines—including deep MoE models (TFPS), Transformers, CNNs, and efficient MLPs—Dynamic TMoE achieves the highest or second-highest performance in 16 of 18 metric comparisons, reducing MSE by 10.4% and MAE by 7.8% on average. In comparisons to recent advanced MoE architectures, the framework demonstrates substantial improvement, especially on benchmarks where distribution shifts are severe.

Ablation results highlight:

• Exclusion of MMD-based drift detection or the replacement of the GRU router with stateless alternatives leads to marked accuracy degradation, confirming their necessity.
• Removal of expert heterogeneity and cyclic relation modeling yields 4.5% and up to 2.1% increases in MSE and MAE, respectively.
• The specialized adaptation strategies (e.g., expert type selection by profile and post-addition expert alignment) outperform standard heuristics and direct fine-tuning approaches, avoiding catastrophic forgetting.

Efficiency analysis reveals Dynamic TMoE maintains a strong parameter-efficiency trade-off compared to other MoEs, balancing model capacity, inference latency, and memory consumption. Memory and compute remain bounded as the expert pool composition is dynamically regulated rather than statically overprovisioned.

Theoretical Analysis

A theoretical generalization bound derived in the RKHS setting shows that the MMD between source (historical) and target (current) distributions directly upper-bounds the target prediction risk. This substantiates the design of the adaptive drift detector, affirming that MMD-based triggering precisely tracks when the available experts become insufficient for maintaining generalization under shift.

Implications and Future Prospects

Dynamic TMoE advances the Mixture-of-Experts paradigm by decoupling architectural evolution from mere capacity scaling, shifting the focus to dynamic, context-aware adaptation. The model prioritizes core design principles—distribution-driven expansion, temporally consistent routing, expert specialization, and efficient pool management—making it especially suitable for deployment in real-world, drift-heavy forecasting applications.

Future research avenues include:

• Test-time expert evolution: Extending dynamic evolution to run-time adaptation for online, streaming, or indefinite-horizon inference, mitigating the need for offline retraining.
• Parameter-efficient adaptation: Leveraging PEFT methods such as LoRA or lightweight adapters to instantiate regime-specific specialization with minimal overhead.
• Foundation model integration: Embedding dynamic architectural adaptation within large-scale time series foundation models to enhance zero-shot or domain-agnostic generalization to OOD shifts.

Conclusion

Dynamic TMoE represents a substantive step forward in adaptive, modular time series forecasting for non-stationary environments. Its principled unification of distribution shift perception, memory-augmented routing, and heterogeneous expert pool evolution leads to SOTA accuracy, robust specialization, and efficient resource utilization. The paradigm provides a solid foundation for future work on lifelong, drift-aware time series modeling and for addressing the persistent challenges of real-world sequence prediction under non-stationarity.