PMDformer: Transformer for LTSF

• The paper introduces patch-mean decoupling to separate trend and residual shape, ensuring attention focuses on shape rather than scale.
• It employs Trend Restoration Attention (TRA) and Proximal Variable Attention (PVA) to robustly capture temporal dynamics and cross-variable dependencies.
• Empirical evaluations show PMDformer achieves lower MSE and MAE on multiple benchmarks while enhancing stability and memory efficiency.

PMDformer is a Transformer architecture for long-term time series forecasting (LTSF) that targets a specific failure mode of patch-based forecasting models: attention scores can become dominated by patch level rather than patch shape when real-world series exhibit non-stationarity and scale variation. The model introduces patch-mean decoupling (PMD), which subtracts the mean of each patch to separate trend from residual shape while preserving intra-patch amplitude; Trend Restoration Attention (TRA), which restores the decoupled trend in the value pathway; and Proximal Variable Attention (PVA), which restricts cross-variable attention to the most recent patch. In the reported experiments, PMDformer improves stability, accuracy, and memory efficiency across standard multivariate LTSF benchmarks (Hu et al., 25 Jun 2026).

1. Problem formulation and motivation

PMDformer is defined on multivariate time series

$X = \{x_t \in \mathbb{R}^C \mid t = 1, \dots, L\},$

where $C$ is the number of variables and $L$ is the lookback length. The sequence is divided into $N = \lfloor L/S \rfloor$ non-overlapping patches of length $S$, with stride equal to $S$. Patch-based modeling is already standard in LTSF because it reduces effective sequence length and emphasizes local temporal semantics, but the PMDformer formulation begins from three limitations of existing patch-based Transformers (Hu et al., 25 Jun 2026).

The first limitation is non-stationarity and scale variation. Patch magnitudes fluctuate over time and across variables, so tokens with larger means can dominate attention scores even when their shapes are less similar. The second is the pitfall of patch normalization. Patch-level Z-score normalization reduces scale effects, but subtracting the mean and dividing by the standard deviation distorts amplitude and shape, which are treated as forecast-relevant. The third is cross-variable non-stationarity. Variable relationships drift over time, so modeling interactions across the full historical window can overfit to outdated correlations; the most predictive cross-variable dependencies are described as those closest to the prediction horizon.

This framing places PMDformer within the LTSF literature as a model that does not reject patching, but rather modifies how patch tokens are formed and how temporal and cross-variable attention are deployed. A plausible implication is that the design is especially targeted at datasets where level shifts and drifting inter-series dependence are central sources of error.

2. Patch-Mean Decoupling

For variable $i \in \{1,\dots,C\}$ and patch index $j \in \{1,\dots,N\}$, the raw patch is

$$P_i^{(j)} = (x_{i,(j-1)S+1}, x_{i,(j-1)S+2}, \dots, x_{i,jS}) \in \mathbb{R}^S.$$

PMD computes the per-patch mean

$$\mu_i^{(j)} = \frac{1}{S}\sum_{k=1}^{S} x_{i,(j-1)S+k},$$

and the mean-decoupled residual

$C$0

where $C$1 is the all-ones vector (Hu et al., 25 Jun 2026).

Each residual patch is then embedded with a shared linear projection across variables:

$C$2

with $C$3, $C$4, and positional embedding $C$5. The embedded tokens for a variable are stacked into $C$6, while the per-patch means are collected in $C$7. A broadcast map $C$8 produces $C$9 when the trend must be reinserted.

The rationale for mean-only decoupling is expressed directly at the attention-logit level. If a raw patch embedding is written as $L$0, then a bilinearized attention logit can be decomposed as

$L$1

into mean-mean, mean-residual, residual-mean, and residual-residual terms. The first three depend on patch means and can dominate the residual-residual term when means are large. PMD removes these mean-dependent components from query/key construction, so attention emphasizes shape similarity instead of scale (Hu et al., 25 Jun 2026).

This mean-only strategy is explicitly contrasted with standard deviation normalization and with SAN. The reported interpretation is that mean subtraction preserves intra-patch amplitude and shape, whereas normalization schemes that divide by the standard deviation can distort shape fidelity. The accompanying theoretical note states that level-dominated logits arise when the spectral norm $L$2 multiplied by the magnitude of the mean terms exceeds an upper bound on the residual and cross terms; this is presented as a sufficient condition motivating mean removal from the query/key pathway.

3. Trend Restoration Attention and Proximal Variable Attention

TRA and PVA are the two attention modules that operationalize the PMD representation. TRA performs per-variable temporal attention along the patch axis. For shape tokens $L$3, queries and keys are formed only from shape embeddings:

$L$4

Attention weights are

$L$5

while the value path restores the trend:

$L$6

The output is

$L$7

followed by LayerNorm, residual connections, and an FFN (Hu et al., 25 Jun 2026).

The stated purpose of TRA is to let attention scores remain shape-centric while still propagating trend information. The paper reports that replacing TRA with standard self-attention degrades performance, and that removing TRA likewise harms performance. The interpretation given is that standard self-attention either neglects trend or entangles it in query/key scoring, thereby reintroducing scale bias.

PVA addresses cross-variable dependence differently. Let $L$8 denote the most recent patch index. PVA extracts the $L$9 tokens at patch $N = \lfloor L/S \rfloor$0,

$N = \lfloor L/S \rfloor$1

and applies multi-head self-attention and an FFN only over these $N = \lfloor L/S \rfloor$2 tokens, with residual connections and LayerNorm. Tokens from earlier patches remain unchanged. The result is then concatenated back into the full tensor $N = \lfloor L/S \rfloor$3.

The restriction to the most proximal patch serves two roles. First, it reduces cross-variable attention complexity from $N = \lfloor L/S \rfloor$4 to $N = \lfloor L/S \rfloor$5 for this stage. Second, it is intended to avoid learning spurious long-range couplings under non-stationary drift. The reported sensitivity study states that using multiple recent patches ($N = \lfloor L/S \rfloor$6) produces higher MSE on ETTh1 and Solar across horizons, and that $N = \lfloor L/S \rfloor$7 is the most stable and accurate setting (Hu et al., 25 Jun 2026). A plausible implication is that PMDformer treats contemporaneous or horizon-adjacent inter-variable structure as more reliable than historically distant cross-series associations.

4. End-to-end architecture, complexity, and optimization

The forward pipeline is specified as follows: optional RevIN preprocessing; segmentation of the input of length $N = \lfloor L/S \rfloor$8 into $N = \lfloor L/S \rfloor$9 non-overlapping patches of size $S$0; PMD and residual-patch embedding; PVA on the last patch across variables; TRA along the patch axis for each variable; and final projection to the forecast horizon $S$1 (Hu et al., 25 Jun 2026).

Before forecasting, trend is added back again:

$S$2

where $S$3 and $S$4, after flattening over the $S$5 dimensions. If RevIN is used, inverse normalization is applied to return forecasts to the original scale. Inference is multi-step direct forecasting to horizon $S$6.

The reported dominant attention complexity for PMDformer is

$S$7

with PVA contributing $S$8 time over the last patch and TRA contributing $S$9 time for per-variable temporal attention. This is contrasted with full-sequence attention over all variables and timesteps, which is summarized as $S$0 time. Appendix results are said to show the lowest GPU memory among PatchTST, iTransformer, and ModernTCN as the number of variables $S$1 and the input length $S$2 vary.

Training uses mean squared error as the objective, with MSE and MAE as evaluation metrics. The reported optimization setup uses Adam with learning rate selected from $S$3 on a single NVIDIA A100 80GB GPU. The main experiments use input length $S$4 and forecast horizons $S$5. LayerNorm, residual connections, positional embeddings, and optional RevIN are described as stabilization components; dropout is not explicitly reported.

5. Empirical results and ablations

PMDformer is evaluated on eight standard LTSF benchmarks: ETTh1, ETTh2, ETTm1, ETTm2, Electricity (ECL), Traffic, Weather, and Solar. The ETT datasets use a 6:2:2 split, while ECL, Traffic, Weather, and Solar use a 7:1:2 split. Sampling intervals range from 10 minutes for Weather and Solar, to 15 minutes for ETTm, and 1 hour for ETT and ECL/Traffic (Hu et al., 25 Jun 2026).

The baseline set includes TQNet, TimeBase, SOFTS, SparseTSF, ModernTCN, iTransformer, TimeMixer, and PatchTST. The reported summary result is that PMDformer achieves the lowest MSE and MAE on 7 of 8 datasets across horizons. The paper also reports average relative reductions against several strong baselines.

Baseline	Average MSE reduction	Average MAE reduction
TimeBase	5.68%	6.61%
TQNet	8.62%	9.96%
iTransformer	11.44%	12.38%

Figure 1 is described as showing the best average MSE across all horizons, with especially large gains on non-stationary datasets. The ablation results further localize the source of these gains. In the PMD ablation, PMDformer outperforms variants using standard deviation normalization, SAN, and a version without PMD on ETTh2, ETTm1, Weather, Traffic, and Solar. In the TRA/PVA ablation, replacing TRA with standard self-attention or removing it degrades performance, while extending PVA to all tokens or removing it increases MSE and MAE. The paper also reports that swapping the order of PVA and TRA hurts performance, because compressing patch information through TRA first makes subsequent variable modeling less effective.

The sensitivity analysis reports that moderate patch sizes, such as 24, 48, and 72, provide the best trade-off, whereas very small or very large patch sizes degrade performance. This suggests that PMDformer depends not only on its decoupling and attention mechanisms, but also on an appropriate temporal granularity for patching.

6. Relation to prior work, limitations, and reproducibility

PMDformer is situated in the patch-based Transformer line of LTSF models represented in the paper by PatchTST and TimeBase. Relative to PatchTST, the distinctive claim is not the use of patches alone, but the explicit decoupling of patch mean from residual shape, followed by trend restoration in the value pathway and in the output projection (Nie et al., 2022). In the cross-variable modeling literature represented by iTransformer and Crossformer, PMDformer departs from full-history interaction by limiting cross-variable attention to the proximal patch, with the stated goals of improved robustness and efficiency (Liu et al., 2023).

The model’s handling of normalization also defines its relation to SAN and SIN. PMDformer does not divide by the patch standard deviation. Instead, it subtracts only the patch mean, with the explicit claim that this preserves amplitude while enabling shape-centric attention. A common misconception would be to treat PMD as interchangeable with generic patch normalization; the reported ablation argues against this equivalence, because the paper attributes performance differences to shape preservation rather than normalization alone (Hu et al., 25 Jun 2026).

The paper identifies several limitations. Performance is sensitive to patch size $S$6 and token dimension $S$7, and very small or very large $S$8 harms results. Proximal-only cross-variable attention, although empirically best at $S$9, may underutilize settings with strong persistent cross-variable lags unless tuned. The final projection from flattened $i \in \{1,\dots,C\}$0 to $i \in \{1,\dots,C\}$1 is described as potentially parameter-heavy when $i \in \{1,\dots,C\}$2 or $i \in \{1,\dots,C\}$3 is very large, in which case regularization or factorization can be necessary.

Reproducibility is supported by a public implementation at https://github.com/aohu1105/PMDformer, with PyTorch and the typical Python scientific stack noted in the implementation description. The reported configuration notes include non-overlapping patches, input length $i \in \{1,\dots,C\}$4, horizons $i \in \{1,\dots,C\}$5, tuning $i \in \{1,\dots,C\}$6 among $i \in \{1,\dots,C\}$7, Adam optimization, and optional RevIN around the Transformer. Empirical illustrations mentioned in the paper include attention visualizations showing reallocation toward shape-aligned patches after PMD and synthetic examples with alternating pulse and sine shapes of varying scales, where models without PMD are described as over-smoothed while PMDformer captures both shape and trend (Hu et al., 25 Jun 2026).