Patch-Mean Decoupling (PMD) in Forecasting

• Patch-Mean Decoupling (PMD) is a technique that subtracts the per-patch mean from time-series data to isolate residual shape information.
• It improves forecast accuracy by decoupling trends from high-frequency details and restoring trend information during critical model stages.
• PMD reduces level bias in attention mechanisms, and its extensions effectively separate stable semantics from fine-grained details in multimodal modeling.

Patch-Mean Decoupling (PMD) is a decoupling principle in which a patch-level low-frequency component is separated from residual shape or detail information so that downstream attention or decoding operates on the component most relevant to the task. In long-term time-series forecasting, PMD denotes subtracting the mean of each temporal patch before computing attention, thereby making attention shape-aware and scale-invariant, and then restoring the decoupled trend at designated stages so forecasts remain on the correct scale. In a broader PMD-style interpretation, the same logic can be applied to multimodal modeling by separating stable patch-level semantics from high-frequency details and reinjecting the latter later under semantic control (Hu et al., 25 Jun 2026, Zhang et al., 13 Mar 2026).

1. Motivation and core principle

PMD was introduced to address a persistent issue in patch-based Transformers for long-term time-series forecasting (LTSF): attention often prefers patches with similar levels rather than similar shapes, especially across variables with different scales. In the PMDformer formulation, Transformer LTSF models increasingly tokenize time series into temporal patches to capture long-range dependencies, but because time series are non-stationary, patches across time and across variables frequently differ in mean level and slow trend. Under these conditions, raw patch attention and patch-level z-score normalization exhibit distinct failure modes (Hu et al., 25 Jun 2026).

For raw patches, attention logits contain large level-level and level-residual cross terms. Two patches with dissimilar shapes but similar means can therefore receive high attention, while truly shape-similar patches at different scales are penalized. For patch-level z-score normalization, dividing by each patch’s standard deviation distorts amplitude proportions and thus shape, which harms shape matching. PMD resolves this by subtracting the mean of each patch only, decoupling a per-patch trend component from the residual shape while leaving amplitudes intact. Attention run on residuals becomes shape-focused; the removed means are explicitly reintroduced later, so forecasts retain trend and scale (Hu et al., 25 Jun 2026).

A common misconception is that PMD is merely another normalization heuristic. The PMDformer paper instead positions it as a structural modification of attention mechanics: queries and keys are constructed from mean-decoupled residuals so that attention scores reflect shape similarity rather than level similarity, while trend information is restored in the value pathway and again before projection to the forecast horizon. Another common confusion is to equate PMD with moving-average decomposition. The PMDformer description distinguishes the two by noting that PMD is tied to the attention token scale, removes only the per-patch mean, and does not blur high-frequency content or require additional filters (Hu et al., 25 Jun 2026).

2. Formalization in patch-based time-series modeling

In the PMDformer notation, a multivariate time series is written as $X \in \mathbb{R}^{L \times C}$, with length $L$ and $C$ variables; at time $t$, $x_t \in \mathbb{R}^C$. A patch length $P$ and stride $S$ are chosen, and for simplicity and as used in PMDformer, non-overlapping patches are taken with $S=P$. The number of patches is then $N=\lfloor L/P \rfloor$, and the $p$-th patch is $L$0, $L$1 (Hu et al., 25 Jun 2026).

For each patch, the per-patch mean for each variable is

$L$2

and the Patch-Mean Decoupling residuals are

$L$3

so that $L$4. The stated effect is that PMD preserves each patch’s intra-patch temporal structure and amplitudes, unlike z-score normalization, while removing only the level or trend (Hu et al., 25 Jun 2026).

Tokenization proceeds variable-wise along the patch axis. For variable $L$5 and patch $L$6, the mean-decoupled residual $L$7 is linearly projected to a $L$8-dimensional token and added to a learned patch positional embedding:

$L$9

with $C$0 and $C$1. Stacking $C$2 over patches yields $C$3 for each variable $C$4. Attention on residual shapes is then computed as

$C$5

$C$6

where $C$7 denotes the sequence of shape embeddings across variables or across patches, depending on the active module (Hu et al., 25 Jun 2026).

The scale-bias argument is made explicit by decomposing the raw patch token as $C$8. The attention logit between tokens $C$9 and $t$0 is

$t$1

with $t$2. The last three terms depend on $t$3 and can dominate when means are large or projections amplify level components. PMD zero-centers each patch, removing all $t$4-dependent terms from $t$5 and letting the residual-residual term $t$6 drive attention (Hu et al., 25 Jun 2026).

3. PMDformer architecture: trend restoration and proximal variable attention

PMD is the core idea behind PMDformer, but the full architecture combines PMD with Trend Restoration Attention (TRA) and Proximal Variable Attention (PVA). The stated end-to-end pipeline is: input and patching; PMD embedding; PVA for cross-variable focus; TRA for temporal shape attention with trend restoration; and a forecast head that adds the trend again and projects to $t$7 (Hu et al., 25 Jun 2026).

TRA is introduced because shape-only attention is excellent for matching patterns but attenuates long-range trends. For each variable $t$8, with shape embeddings $t$9 and per-patch mean sequence $x_t \in \mathbb{R}^C$0 broadcast to $x_t \in \mathbb{R}^C$1 dimensions when added, TRA computes

$x_t \in \mathbb{R}^C$2

$x_t \in \mathbb{R}^C$3

Immediately before projecting to the $x_t \in \mathbb{R}^C$4-step forecast, PMDformer restores the trend again:

$x_t \in \mathbb{R}^C$5

The stated interpretation is that $x_t \in \mathbb{R}^C$6 encode shape-to-shape similarity across patches, whereas $x_t \in \mathbb{R}^C$7 carries shape plus trend content to propagate level information. Restoring the trend in $x_t \in \mathbb{R}^C$8 prevents shape-only encodings from losing level information, while restoring again before the head ensures proper scale calibration at the output (Hu et al., 25 Jun 2026).

PVA addresses cross-variable relationships under the assumption that the most recent interactions are most predictive for the immediate future. It performs multi-head self-attention only among the $x_t \in \mathbb{R}^C$9 tokens from the last patch $P$0:

$P$1

followed by LayerNorm and FFN updates on that restricted token set. Tokens from earlier patches keep their PMD embeddings; after PVA, the refined last-patch tokens are concatenated back with earlier tokens to form $P$2. An equivalent view is a recency mask $P$3 that allows attention only among token pairs located at $P$4:

$P$5

where $P$6 has $P$7 for token pairs within $P$8 and $P$9 elsewhere. In practice, PMDformer simply restricts the token set to $S$0, reducing variable-wise attention complexity from $S$1 to $S$2 and mitigating overfitting to outdated correlations (Hu et al., 25 Jun 2026).

The architecture also uses learned patch positional embeddings $S$3, standard layer norms and residual connections, and standard MHSA with embedding size $S$4 and number of heads $S$5. An optional instance-level normalization such as RevIN before patching can be used to mitigate distribution shift (Hu et al., 25 Jun 2026).

4. Empirical performance and ablation evidence

The empirical evaluation reported for PMDformer uses the LTSF benchmarks ETTh1, ETTh2, ETTm1, ETTm2, ECL, Traffic, Weather, and Solar, with input length $S$6, prediction horizons $S$7, metrics MSE and MAE, and optimization in PyTorch with Adam and a learning-rate grid $S$8. Training is reported on NVIDIA A100. Under this setup, PMDformer achieves the best average performance on $S$9 datasets, outperforming PatchTST, TimeBase, iTransformer, TimeMixer, TQNet, SparseTSF, ModernTCN, and SOFTS (Hu et al., 25 Jun 2026).

Comparison target	Average MSE change	Average MAE change
TimeBase	$S=P$0	$S=P$1
TQNet	$S=P$2	$S=P$3
iTransformer	$S=P$4	$S=P$5

Ablation results are used to isolate the contribution of each component. For PMD versus patch z-score or removing PMD, Traffic shows MSE $S=P$6 with PMD versus $S=P$7 with standard deviation normalization and $S=P$8 with no PMD; Solar shows $S=P$9 versus $N=\lfloor L/P \rfloor$0 and $N=\lfloor L/P \rfloor$1; Weather shows $N=\lfloor L/P \rfloor$2 versus $N=\lfloor L/P \rfloor$3 for SAN and $N=\lfloor L/P \rfloor$4 with no PMD. The interpretation given is that PMD preserves intra-patch shape and amplitude, enabling better shape matching than z-score or SAN (Hu et al., 25 Jun 2026).

Further ablations concern TRA and PVA. Replacing TRA with standard self-attention degrades performance, for example on Traffic from MSE $N=\lfloor L/P \rfloor$5 to $N=\lfloor L/P \rfloor$6, which is presented as evidence for the importance of trend reintegration. Expanding PVA to all patches also hurts performance, for example on Solar from MSE $N=\lfloor L/P \rfloor$7 to $N=\lfloor L/P \rfloor$8, supporting the claim that focusing on the last patch avoids spurious historical correlations. Removing both TRA and PVA yields the largest drop, for example Traffic MSE $N=\lfloor L/P \rfloor$9 (Hu et al., 25 Jun 2026).

Sensitivity analyses report that using only the nearest patch in the cross-variable window ($p$0) gives more stable and better MSE than $p$1 on ETTh1 and Solar, and that moderate patch sizes, for example $p$2 in the range $p$3–$p$4, outperform very small or very large $p$5. These findings suggest that the decoupling mechanism is most effective when token granularity is neither too fine nor too coarse (Hu et al., 25 Jun 2026).

5. Conceptual positioning, computational profile, and limitations

PMD is explicitly positioned against three adjacent design families. Relative to Autoformer-like moving-average decomposition, PMD is tied to the attention token scale; it removes only the per-patch mean, leaving short-term fluctuations and amplitude relationships intact. Moving averages can smear high-frequency patterns and require extra hyperparameters, whereas PMD is parameter-free at the patch level and directly optimizes shape-aware attention. Relative to patch-level z-score normalization, including SAN and SIN variants, PMD subtracts only the mean, so the residuals preserve the true shape and amplitude consistency. Relative to PatchTST-style patch attention, PMDformer changes the attention mechanics so that $p$6 ignore patch means and $p$7 restores them, while also adding targeted cross-variable attention at the most proximal time (Hu et al., 25 Jun 2026).

The computational claims are similarly specific. Computing patch means and residuals costs $p$8 and is described as negligible. PVA restricts cross-variable self-attention to the last patch only, giving $p$9 instead of $L$00. TRA performs per-variable attention along $L$01 patches with cost $L$02 and is independent across variables, hence parallelizable. Because PVA avoids large cross-variable mixing over $L$03 patches, the reported overall memory and time are competitive or better than baselines, and the paper reports lower GPU memory than PatchTST, iTransformer, and ModernTCN across varying $L$04 and $L$05 (Hu et al., 25 Jun 2026).

Several limitations and edge cases are identified. Strong intra-patch nonstationary trends, such as pronounced slopes within a short patch, may not be fully captured by a constant mean; subtracting the mean can leave a residual with level drift. The described mitigations are to increase $L$06 moderately, use multi-scale patches, or append a small local-trend channel such as a per-patch slope to $L$07. Seasonal structure within a patch is not explicitly modeled by PMD; combining PMD with frequency-domain blocks or seasonal embeddings can help if needed. Heavy-tailed noise makes the mean less robust, so robust centering by median or Huberized mean is proposed as a potential extension. If cross-variable dependencies are truly long-range and stationary, restricting PVA to the last patch could drop useful signal, even though the paper’s sensitivity study shows $L$08 is best on the reported benchmarks (Hu et al., 25 Jun 2026).

These constraints also delimit the scope of the method. PMD is not a universal decomposition of all temporal structure; it is a token-scale decoupling that removes a per-patch mean so that attention concentrates on residual shape. This suggests that its strongest use case is the setting in which level bias is the dominant source of attention misalignment rather than the only source of nonstationarity.

6. PMD-style extensions to unified multimodal modeling

A distinct but related use of the PMD idea appears in the multimodal model Cheers, which presents “decoupling patch details from semantic representations” as a way to unify visual comprehension and generation. In that formulation, PMD is a conceptual mapping rather than the exact time-series operation of subtracting patch means. The low-frequency or semantic component is represented by SigLIP2-ViT features aligned to $L$09 patches after reconstructing pixels from the latent, while the high-frequency or detail component is represented by the high-frequency patch details $L$10 produced from the same patches. A gating module $L$11, conditioned on the semantic decoding trajectory, modulates how much detail residual is injected at each location and time step (Zhang et al., 13 Mar 2026).

The unified vision tokenizer begins from an input image $L$12 with $L$13 in training. A VAE encoder produces latent states $L$14 with $L$15 and $L$16, so $L$17 at $L$18. The task-dependent latent interpolation is

$L$19

with the stated convention that one samples $L$20 for image generation, fixes $L$21 for visual understanding, and sets $L$22 for language-only tasks. Pixels are reconstructed via $L$23 and then encoded with SigLIP2 as

$L$24

A Pixel-Unshuffle reduces spatial resolution by $L$25 in each dimension and projects channels to the LLM hidden size $L$26, producing $L$27 and realizing $L$28 token compression: at $L$29, the number of visual tokens drops from $L$30 to $L$31 (Zhang et al., 13 Mar 2026).

The PMD-style decomposition appears explicitly in the cascaded flow matching head. Stage 1 decodes semantics from $L$32 and upsamples by PixelShuffle to $L$33. Stage 2 injects patch details through the exact gated residual formula

$L$34

where $L$35. In the paper’s analogy to classical PMD, $L$36 plays the role of the low-frequency component and $L$37 supplies the high-frequency detail source. The model then predicts a continuous-time velocity field $L$38 for latent-space flow:

$L$39

with optional time rescaling

$L$40

The training objectives combine autoregressive text loss and flow-matching loss:

$L$41

$L$42

$L$43

The backbone is Qwen2.5-1.5B-Instruct; visual tokens use bidirectional attention, and text uses causal masking (Zhang et al., 13 Mar 2026).

The empirical argument for this PMD-style split is that stable semantics and high-fidelity details can be optimized without forcing both into a single shared representation. Cheers reports understanding scores including SEEDBench $L$44, MMBench $L$45, MMStar $L$46, ChartQA $L$47, OCRBench $L$48, RealWorldQA $L$49, POPE $L$50, AI2D $L$51, MathVista $L$52, and MMMU $L$53. On generation, it reports GenEval overall $L$54 and DPG-Bench overall $L$55, and states that it outperforms Tar-1.5B on GenEval and MMBench while requiring only $L$56 of the training cost. The analysis of High-Frequency Injection further shows that injection is low and structural early, dips mid-trajectory, and surges late to refine textures. This is presented as evidence that semantics-first, details-later decoding can stabilize reasoning while preserving fidelity (Zhang et al., 13 Mar 2026).

Within the broader usage of the term, this multimodal case should be distinguished from the exact PMD operation in PMDformer. PMD in LTSF is a concrete per-patch mean subtraction with trend restoration; PMD in Cheers is a conceptual split between semantic tokens and detail residuals. What unifies them is the decoupling logic: low-frequency patch-level structure is isolated to stabilize matching or reasoning, and residual information is reintroduced later so the final prediction or generation retains the necessary detail and scale (Hu et al., 25 Jun 2026, Zhang et al., 13 Mar 2026).