- The paper introduces the PMD module that decouples patch means from residual shapes to reduce bias in attention computations.
- It proposes proximal variable attention (PVA) and trend restoration attention (TRA) to enhance dynamic cross-variable dependency modeling and trend recovery.
- PMDformer achieves state-of-the-art performance with a 5–11% MSE reduction across eight benchmarks compared to leading models.
Long-term time series forecasting (LTSF) for high-dimensional, non-stationary sequences underpins essential applications including energy management, quantitative finance, and transportation analytics. Recent LTSF architectures have increasingly adopted Transformer models with patch-based tokenization to encode long-range dependencies. However, patch-based representations in time series introduce substantial challenges in disentangling true shape similarity from confounding scale effects between patches and variables—contrasts with image-domain patching due to the inherent one-dimensional, non-stationary nature of time series.
The critical technical limitation in prior approaches lies in the failure of the self-attention mechanism to capture shape similarity when patches vary in mean or scale, sometimes generating erroneous attention scores. While approaches such as patch-wise normalization (e.g., Z-score normalization) seek to control for these discrepancies, they suffer from amplitude distortion, which fundamentally disrupts the temporal structure necessary for modeling. Additionally, variable-dependent and -independent models often suffer either from insufficient cross-variable modeling or from learning spurious dependencies over long temporal horizons.
Patch-Mean Decoupling (PMD)
PMDformer introduces the patch-mean decoupling (PMD) module, which subtracts the mean from each input patch without standard deviation normalization. This explicitly separates the deterministic trend (patch mean) from the residual shape, preserving the intrinsic amplitude and structure of local signals. This operation aligns the input representation with the natural objectives of attention—enabling shape-focused matching while cleanly decoupling scale-induced attention bias. Theoretical analysis in the paper rigorously demonstrates that ignoring patch means in attention computation is necessary to mitigate mean-dominated biases that obscure true similarity.
Proximal Variable Attention (PVA)
PVA is introduced to address the dynamic and non-stationary nature of cross-variable dependencies. Rather than attending across the entirety of historical data (which induces substantial noise and overfitting risks), PVA computes variable-wise self-attention only over the tokens corresponding to the most recent patch. Empirically, the restriction to the most recent patch yields lower average MSE and greater predictor stability, validating the premise that temporal proximity is paramount for inter-variable relationship discovery in non-stationary processes.
Trend Restoration Attention (TRA)
A major drawback of mean-subtractive procedures is the attenuation of long-range and global trend information. TRA solves this by explicitly injecting back the patch means (trend component) into the value pathway of the self-attention mechanism after shape-level token encoding. Since the trend statistics are introduced only in the projection pathway, shape matching remains governed by properly centered tokens, while ultimate output tokens regain the necessary trend information for accurate multi-step forecasting.
Overall Framework
PMDformer is organized as a sequence of four modules:
- PMD for patch partitioning and intra-patch mean separation.
- PVA for capturing immediate cross-variable dependencies.
- TRA for modeling patch-based shape similarity while restoring long-range trend information.
- Final projection which recombines trend and shape for calibrated prediction outputs.
Strong Numerical Results and Ablations
Across eight standard LTSF benchmarks, PMDformer outperforms contemporary SOTA time series models including but not limited to PatchTST, TimeBase, SOFTS, TQNet, and iTransformer. On average, the model demonstrates reductions of 5–11% in MSE and comparable improvements in MAE relative to leading baselines. PMDformer achieves the lowest error on 7 out of 8 benchmarks, indicating robust generalization across application domains and data characteristics.
Ablation studies show that:
- Removing PMD (including mean without restoration) or replacing it with scale normalization methods such as SAN results in significant error increases, validating the theoretical necessity of patch mean separation without amplitude distortion.
- Disabling TRA leads to a marked decrease in performance, as trend information is lost.
- Removing or generalizing PVA (e.g., to all historical patches) degrades performance by introducing spurious cross-variable linkages.
- The model is robust to patch size variation, though moderate-size patches occupy an optimal regime in terms of balancing locality and long-term dependency modeling.
- Memory and computational footprint analyses indicate that PMDformer maintains efficiency even in high-dimensional or long-sequence settings, requiring less GPU memory than comparable transformer approaches.
Implications and Directions for Future Research
PMDformer clarifies persistent modeling challenges in LTSF. By theoretically and empirically isolating the role of mean-induced bias in self-attention, the research demonstrates that true shape similarity—rather than simple amplitude-normalized correlation—drives multistep predictive accuracy. The architectural decoupling of trend and shape sets a precedent for more theoretically principled sequence model architectures in time series domains, suggesting the possibility of extending patch-mean separation to non-linear, multimodal, or hierarchical representations in other temporal domains (including multimodal sensor fusion and exogenous series conditioning).
Practically, PMDformer provides a robust foundation for real-world deployment in dynamic non-stationary environments (e.g., energy markets during demand shocks, financial asset correlations under crisis, or adaptive control in traffic management systems), where both the preservation of trend and the detection of subtle, shape-based analogies are critical to operational performance.
Anticipated future directions include scaling PMDformer to extremely high-dimensional (thousands of variables) multivariate settings, integrating with large time-series foundation models for zero-shot or few-shot transfer, and extending these normalization principles to cross-modal transformer integration (e.g., text or event streams in forecasting).
Conclusion
PMDformer provides a rigorous and effective solution to the LTSF problem by decoupling patch means to isolate shapes, focusing cross-variable modeling on temporally proximate data, and reinstating trends in the forecasting pipeline. The approach addresses core technical bottlenecks of previous Transformer-based methods for time series and establishes strong new performance baselines. Its methodological contributions have broad implications for the design of robust sequence models in non-stationary temporal domains and represent a significant refinement in shape-centric time series architecture engineering (2606.26549).