WD-MoE: Dual-Domain Sparse Modeling for Time Series

• WD-MoE is a dual-domain sparse modeling framework that concurrently processes raw signal patches and wavelet coefficients using a unified mixture-of-experts routing.
• It aligns time-domain and wavelet-domain tokens through shared temporal patches, capturing multi-scale features via parallel transformer-based paths.
• Empirical results show superior forecasting accuracy across diverse benchmarks, particularly for periodic, multi-scale, and high-frequency signals.

Searching arXiv for the primary paper and named baselines to ground citations. Wavelet Domain Mixture-of-Experts (WD-MoE) denotes a dual-domain sparse modeling pattern for time-series forecasting in which time-domain tokens and wavelet-domain tokens are processed in parallel, aligned on a shared temporal axis, and coordinated by a unified mixture-of-experts (MoE) routing mechanism. In the formulation concretely instantiated by WaveMoE, a decoder-only time series foundation model (TSFM) jointly models raw signal patches and discrete wavelet transform (DWT) coefficients, uses shared expert assignment for corresponding temporal positions, and applies separate prediction heads in value space and wavelet-coefficient space under joint supervision (Wu et al., 12 Apr 2026). Multivariate series are handled through a channel-independent strategy that decomposes them into independent univariate series.

1. Architectural definition and model scope

In its instantiated form, WD-MoE consists of four coupled subsystems: a time-domain path operating on patches of the original series, a wavelet-domain path operating on DWT-derived coefficients from the same input patches, per-path Transformer-style sparse self-attention, and a unified MoE block that dispatches both domains through the same expert assignment mechanism (Wu et al., 12 Apr 2026). The output stage uses separate heads to predict future numerical values and future wavelet coefficients, with both heads jointly supervised.

The defining architectural statement is that both pathways process the same input time-series patch in parallel and produce temporally aligned token representations along a shared time axis, after which a unified router jointly determines expert assignments for time-domain and frequency-domain tokens at identical temporal locations. This makes WD-MoE neither a pure time-domain TSFM nor a standalone wavelet model. It is a coordinated dual-path system in which explicit frequency-domain structure is embedded directly into scalable MoE computation.

A central misconception is to treat WD-MoE as two independent expert systems, one per domain. The architecture is explicitly not organized that way. The expert pool is shared across domains, while domain-specificity is preserved inside each expert via separate branches. Another misconception is to assume that the wavelet path replaces the time path. In fact, the model is additive in representational scope: raw values and wavelet coefficients coexist throughout training and forecasting.

2. Wavelet-domain representation and temporal alignment

The wavelet-domain path applies a discrete wavelet transform to decompose the input sequence into multi-scale approximation and detail coefficients that retain localized temporal information (Wu et al., 12 Apr 2026). In WaveMoE, the basis is the bior2.2 wavelet and the decomposition level is $L=2$, yielding approximation coefficients at level 2, $A_2$, and detail coefficients at levels 1 and 2, $D_1$ and $D_2$. Conceptually, the decomposition is written as

$$x(t) \xrightarrow{\mathrm{DWT}_{\mathrm{bior2.2},\,L=2}} \{A_2(k), D_2(k), D_1(k)\},$$

where $k$ indexes time-localized coefficients at different scales.

Tokenization is defined by strict temporal compatibility between domains. If the time-domain path uses fixed-length patches of length $P$, then the $i$-th time patch is

$$\mathrm{Patch}^{\mathrm{time}}_i = x_{(i-1)P+1:iP}.$$

The same patch boundaries are used to group wavelet coefficients associated with that temporal support, conceptually forming

$$\mathrm{Patch}^{\mathrm{wavelet}}_i = \mathrm{flatten}\big(A_2^{(i)}, D_2^{(i)}, D_1^{(i)}\big).$$

The result is a one-to-one alignment: each patch index $A_2$0 corresponds to one time-domain token and one wavelet-domain token.

This alignment is the structural precondition for WD-MoE routing. Because both domains share unified temporal indexing, the router can treat corresponding time and wavelet tokens as a single decision unit for expert assignment. The excerpt does not state boundary conditions explicitly, although it situates the DWT within standard wavelet practice. It also fixes the decomposition level at 2 rather than introducing adaptive level selection. This suggests that the design prioritizes stable multi-scale structure and token compatibility over transform adaptivity.

3. Unified routing and dual-branch expert computation

The MoE block is the core mechanism that turns dual-domain representation into a WD-MoE system. Each expert contains a dual-branch feed-forward network: one branch processes time-domain tokens and the other processes wavelet-domain tokens (Wu et al., 12 Apr 2026). Routing, however, is shared. Let $A_2$1 and $A_2$2 denote the two token representations at temporal index $A_2$3. The routing input is their concatenation,

$A_2$4

which is passed through an MLP gating module to produce expert scores

$A_2$5

followed by a softmax over experts,

$A_2$6

This routing strategy means that expert selection is conditioned jointly on time-domain context and wavelet-domain structure. The two domains are therefore coordinated at the level of dispatch, not collapsed into a shared representation. The synthesis further writes the standard sparse MoE computation in top-$A_2$7 form,

$A_2$8

$A_2$9

The excerpt associates the design with Switch/MoE literature through its reference to Fedus et al. (2022), which suggests top-$D_1$0 routing, although the excerpt does not explicitly state the value of $D_1$1.

A second common misconception is that cross-domain interaction must be implemented by explicit cross-attention. In this design, it is not. Each path uses independent self-attention, and cross-domain coordination occurs through shared routing. The router sees both domains together; the experts then apply domain-specific transformations under the same assignment. The paper does not provide explicit routing heatmaps or direct measurements of expert frequency specialization. A plausible implication is that experts can become coupled time-frequency specialists because their assignment is conditioned on the joint signal state at each temporal location.

4. Sparse sequence modeling, supervision, and pretraining regime

Within each domain, token dependencies are modeled by independent Transformer-style self-attention layers equipped with sparse attention (Wu et al., 12 Apr 2026). The sparse mechanism selectively activates a subset of informative tokens: tokens with the top-$D_1$2 attention scores are retained for interaction, while less informative tokens are masked. This reduces computational overhead on long sequences and complements sparse expert activation in the MoE block.

At the output stage, the two pathways remain distinct. One head forecasts numerical values in the original domain, and the other forecasts corresponding wavelet components. The forecasting objective is optimized using the Huber loss, and the two heads are jointly supervised. The synthesis writes the total objective conceptually as

$D_1$3

while noting that the excerpt does not explicitly state the weighting $D_1$4. What is explicit is the use of Huber loss for forecasting and the presence of supervision in both domains.

The foundation-model regime is large scale. WaveMoE is pretrained on a corpus based on Time-300B, augmented with additional IoT datasets, and after domain balancing, filtering, and preprocessing the final pretraining dataset contains approximately 98 billion time steps across 9 domains: IoT, energy, finance, healthcare, nature, sales, transportation, network systems and synthetic data (Wu et al., 12 Apr 2026). The pretraining window length is 4096 time steps. The downstream evaluation setup uses a fixed context of 512 and horizon of 96.

The data pipeline includes domain-aware filtering and balanced sampling. The synthesis specifies thresholds that remove windows with more than 20% NaNs/Inf and more than 20% near-zero values in the raw series or first/second differences; missing values are replaced with 0 and excluded from loss via a loss mask. Optimization uses AdamW, a base learning rate of $D_1$5, batch size 128, 100{,}000 training steps, and a warmup ratio of 0.1. These details situate WD-MoE as a foundation-model design that couples representation enrichment with large-scale pretraining rather than as a small bespoke forecasting architecture.

5. Empirical results and comparison with time-only MoE forecasting

The main empirical evidence for WD-MoE comes from evaluation on 16 diverse benchmark datasets using Mean Squared Error (MSE) and Mean Absolute Error (MAE) as metrics (Wu et al., 12 Apr 2026). The comparison set includes Time-MoE, Chronos, Timer, and Sundial. According to the reported results, WaveMoE achieves the best MSE on 14 out of 16 datasets, the best MAE on 11 datasets, and the lowest average MSE and MAE overall.

Several dataset-level comparisons are explicitly reported. On ETT2, WaveMoE obtains MSE 0.0342 and MAE 0.1226, compared with MSE 0.0840 and MAE 0.1809 for Time-MoE. On M5, WaveMoE reports MSE 0.0688 and MAE 0.0150, compared with MSE 0.8408 and MAE 0.4251 for Time-MoE. On Traffic, WaveMoE reports MSE 0.1030 and MAE 0.0618, compared with MSE 0.1827 and MAE 0.2145 for Time-MoE. The extended table is described as consistently showing WaveMoE competitive or best.

The most informative ablation-like comparison is against Time-MoE. The two systems share a similar MoE framing, but Time-MoE uses only time-domain inputs, whereas WaveMoE adds the wavelet path, shared routing over aligned time-wavelet tokens, and wavelet-domain supervision. The reported gains are particularly associated with highly periodic data, complex multi-scale patterns, and noisy or high-frequency cases. Qualitative visualizations are described as showing better alignment with peaks, troughs, and sharp transitions, while time-only baselines smooth out high-frequency components. This supports, though does not by itself prove mechanistically, the claim that wavelet-domain tokens plus coordinated MoE routing improve modeling of multi-scale temporal dynamics.

6. Conceptual significance, interpretation, and stated limits

The motivating claim for WD-MoE is that frequency-domain representations, especially wavelet-domain representations, provide a complementary perspective for time-series analysis because wavelet transformations preserve structured time-frequency information and encode useful inductive biases for complex temporal signals (Wu et al., 12 Apr 2026). Within this framing, the DWT exposes scale-specific structure, while the MoE mechanism provides conditional capacity allocation. Their combination yields three intended effects: multi-scale inductive bias, localized high-frequency modeling, and efficient capacity scaling through sparse activation.

A plausible interpretation is that shared routing enables experts to act as frequency-scale specialists. Tokens with strong detail coefficients may tend toward experts that model bursts, oscillations, or sharp transitions, while tokens dominated by approximation coefficients may tend toward experts that model smooth trend and seasonality. The excerpt explicitly presents this as a conceptual reading rather than an empirically visualized routing analysis. It therefore supports the specialization hypothesis at the level of architectural intent, not direct interpretability evidence.

The design also has clearly stated boundaries. The excerpt does not present explicit routing heatmaps, does not state boundary conditions for the DWT, does not expose adaptive decomposition-level selection, and does not explicitly specify a load-balancing loss for expert usage. The synthesis notes that such a loss is standard in MoE literature and would be advisable to avoid expert collapse, but this remains an inference rather than a directly stated component of the excerpt. Likewise, the model does not use explicit cross-attention between time and wavelet pathways; coordination is exclusively through shared routing.

Taken together, WD-MoE is best understood as a general architectural template instantiated by WaveMoE: dual-domain tokenization, strict temporal alignment, parallel domain-specific sequence modeling, shared expert assignment from fused time-wavelet representations, dual-branch expert computation, and joint supervision in both original and wavelet domains (Wu et al., 12 Apr 2026). In that sense, it is a concrete mechanism for integrating wavelet-domain corpora and MoE scaling within a TSFM, rather than merely a time-series model with auxiliary spectral features.