Trend Restoration Attention (TRA) in PMDformer

• Trend Restoration Attention (TRA) is a temporal attention mechanism that decouples residual shape matching from trend restoration in PMDformer.
• TRA computes queries and keys from mean-subtracted tokens while reinjecting patch means via the value pathway to preserve level information.
• Empirical ablations show that TRA mitigates scale bias and maintains local pattern fidelity, significantly improving forecasting accuracy.

Searching arXiv for the cited papers and closely related uses of “TRA”. Trend Restoration Attention (TRA) is a temporal attention mechanism introduced in PMDformer for long-term time series forecasting (LTSF). Its defining purpose is to restore long-range trend information that has been deliberately removed by Patch-Mean Decoupling (PMD), while keeping the attention weights themselves focused on patch-level shape similarity rather than scale level. Within PMDformer, TRA operates after PMD and after Proximal Variable Attention (PVA): PMD separates each patch into a mean term and a mean-subtracted residual, PVA models recent cross-variable dependencies on the last patch, and TRA then applies a per-variable Transformer encoder along the patch axis, using residual tokens for queries and keys but reinjecting the stored patch means through the value pathway. This design targets a recurrent difficulty in patch-based Transformer forecasting, namely that raw attention can be dominated by scale differences rather than by the underlying temporal shape of local patterns (Hu et al., 25 Jun 2026).

1. Architectural role in PMDformer

PMDformer is organized into four main parts: Patch-Mean Decoupling and embedding, Proximal Variable Attention, Trend Restoration Attention, and a projection layer. The input multivariate series $X \in \mathbb{R}^{L \times C}$ is first split into patches of length $S$. For each variable and each patch, PMD computes a patch mean, interpreted as a trend component, and a mean-subtracted residual, interpreted as a shape component. The residual is then embedded as a token. PVA operates on the last patch across all variables and models cross-variable dependencies in the most recent time window. TRA follows this stage and applies temporal attention independently for each variable across patches. The final projection layer performs a linear projection and adds patch means again to reconstruct forecasts at the original scale (Hu et al., 25 Jun 2026).

In this decomposition, PVA and TRA have distinct roles. PVA determines recent cross-variable interactions, whereas TRA determines which past patches are temporally similar for a given variable. The distinction is operational rather than merely descriptive: TRA is intra-variable attention with shared parameters across variables, while PVA is the mechanism that performs cross-variable mixing. The PMDformer workflow can therefore be summarized as residual tokenization after PMD, last-patch variable refinement by PVA, temporal attention per variable by TRA with trend restored in the value stream, and final level reconstruction in the projection head (Hu et al., 25 Jun 2026).

A central consequence of this placement is that TRA does not receive raw patches. It receives residual-based tokens, possibly with the last patch updated by PVA, together with the stored patch means produced earlier by PMD. That separation is what enables the mechanism to distinguish alignment from reconstruction: attention alignment is driven by residual shape, while reconstruction remains trend-aware.

2. Motivation: scale bias, shape fidelity, and trend attenuation

The immediate motivation for TRA is the claim that attention over raw patch embeddings is biased toward scale level. For two raw patch embeddings $x_i$ and $x_j$, the attention logit is written as

$$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$

where $x_i = r_i + \mu_i$, $\mu_i$ is the patch mean, $r_i$ is the residual, and $M = W_E^\top W_Q^\top W_K W_E$. In this decomposition, the first three terms depend on patch means and can dominate the residual-similarity term $r_i^\top M r_j$. PMDformer states a sufficient condition in Proposition 1 under which these mean-dependent terms dominate the logits, thereby confirming that raw attention can primarily reflect scale rather than shape (Hu et al., 25 Jun 2026).

PMD addresses this problem by subtracting the mean from each patch before tokenization. Unlike patch-level $S$0-score normalization, which subtracts the mean and divides by the standard deviation, PMD removes only the mean. The stated rationale is that standard-deviation scaling alters intrinsic local amplitudes and therefore distorts shape similarity, whereas mean subtraction preserves the original structure of the residual pattern. This establishes the first half of the design principle: queries and keys should be computed from mean-subtracted tokens if temporal matching is intended to reflect shape rather than level (Hu et al., 25 Jun 2026).

Mean subtraction, however, also weakens trend information. In PMDformer, the patch means $S$1 are treated as a long-range trend representation across patches, and the residuals $S$2 emphasize local shape and short-term dynamics. If the model were to use only mean-subtracted tokens throughout the temporal encoder, long-horizon level evolution could be attenuated. TRA is introduced to resolve that tension by decoupling the alignment signal from the reconstruction signal: residuals define attention weights, but trend is reintroduced through the values. This is closely aligned with the broader “First De-Trend then Attend” perspective in TDformer, which argues that softmax attention is poorly suited to trend extrapolation and that attention should operate on a de-trended signal while trend is handled separately and restored later (Zhang et al., 2022).

3. Formal specification

For variable $S$3 and patch index $S$4, with patch length $S$5, PMDformer defines the raw patch as

$S$6

Its mean is

$S$7

and the mean-subtracted residual is

$S$8

The embedded residual token is then

$S$9

with $x_i$0, $x_i$1, and positional embedding $x_i$2. For a fixed variable $x_i$3, the residual tokens are collected as $x_i$4, and the patch means as $x_i$5 (Hu et al., 25 Jun 2026).

TRA modifies standard scaled dot-product attention at the value pathway. For variable $x_i$6, queries and keys are computed only from the residual tokens: $x_i$7 with $x_i$8. Attention weights are then

$x_i$9

The essential modification is the definition of the value: $x_j$0 where $x_j$1 and $x_j$2 is broadcast from $x_j$3 to shape $x_j$4 by replicating each scalar patch mean along the feature dimension. The attention output is

$x_j$5

The remainder of the block follows a standard Transformer encoder pattern with residual connections, LayerNorm, and FFN: $x_j$6

$x_j$7

After TRA, the projection layer again adds back the patch means: $x_j$8 with $x_j$9 and $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$0 (Hu et al., 25 Jun 2026).

This formulation makes the distinction from standard self-attention exact. In standard attention, $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$1, $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$2, and $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$3 are all derived from the same token sequence. In TRA, $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$4 and $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$5 remain shape-only, while $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$6 is shape plus trend. The mechanism therefore preserves shape-based matching while ensuring that the aggregated representation carries level information.

4. Operational interpretation and relation to adjacent forecasting mechanisms

Operationally, TRA can be read as a sequence of four transformations. First, PMD re-centers each patch by subtracting its mean. Second, PVA refines the most recent patch across variables while still working on mean-subtracted features. Third, TRA matches historical patches using only residual-shape similarity. Fourth, the value stream and final projection restore trend so that the model does not lose the level at which similar shapes historically occurred. In intuitive terms, the mechanism is designed to answer two different questions with two different signals: which past patches look similar in shape, and at what level did those shapes occur (Hu et al., 25 Jun 2026).

This differs from standard Transformer self-attention, where trend and residual are entangled in all three of $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$7, $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$8, and $$z_{ij} = q_i^\top k_j = x_i^\top M x_j = \mu_i^\top M \mu_j + \mu_i^\top M r_j + r_i^\top M \mu_j + r_i^\top M r_j,$$9. It also differs from decomposition-based architectures such as Autoformer and FEDformer. Those models explicitly decompose trend and residual, often through moving-average or frequency-domain filters, but the trend is not reinserted through the value stream in the same patch-aligned manner. PMDformer instead performs a simple patch-level mean subtraction and restores trend directly in $x_i = r_i + \mu_i$0 and again before the output projection, rather than forecasting trend as an entirely separate branch (Hu et al., 25 Jun 2026).

TDformer provides a conceptually related but architecturally distinct reference point. TDformer decomposes the input into trend and seasonality using learned moving-average filters, predicts trend with a three-layer MLP wrapped by RevIN, predicts seasonality with Fourier attention, and then adds the two forecasts. Its theoretical analysis states that time-domain, Fourier-domain, and wavelet-domain attention are equivalent under linear conditions, while softmax makes their empirical behavior diverge; it then argues that attention should not be tasked with trend extrapolation. This is not the same mechanism as TRA, but it supports the same broader principle: de-trend before attention, then restore trend through a dedicated pathway (Zhang et al., 2022).

A further nearby comparison arises with patch normalization methods such as SAN. PMDformer contrasts PMD with mean/std patch normalization on the grounds that standard-deviation scaling changes local amplitudes and thus distorts shape. TRA inherits that commitment: it depends on mean subtraction without variance scaling so that the residual token still encodes the original local profile.

5. Empirical evidence and implementation characteristics

The empirical case for TRA in PMDformer is presented primarily through ablation experiments averaged over prediction horizons $x_i = r_i + \mu_i$1, $x_i = r_i + \mu_i$2, $x_i = r_i + \mu_i$3, and $x_i = r_i + \mu_i$4 on ETTh2, ETTm1, Traffic, and Solar. Full PMDformer, using both TRA and PVA with last-token PVA, reports ETTh2 $x_i = r_i + \mu_i$5, ETTm1 $x_i = r_i + \mu_i$6, Traffic $x_i = r_i + \mu_i$7, and Solar $x_i = r_i + \mu_i$8 in MSE/MAE. Replacing TRA with standard self-attention degrades performance to ETTh2 $x_i = r_i + \mu_i$9, ETTm1 $\mu_i$0, Traffic $\mu_i$1, and Solar $\mu_i$2. Removing TRA while keeping PVA further degrades Traffic to $\mu_i$3 and Solar to $\mu_i$4. Removing both TRA and PVA yields the largest drop, including Traffic $\mu_i$5 and Solar $\mu_i$6. Swapping the order of TRA and PVA, so that TRA precedes PVA, is also worse than the original order, with Traffic $\mu_i$7 and Solar $\mu_i$8 (Hu et al., 25 Jun 2026).

These ablations support three claims made in the PMDformer description. First, neglecting trend restoration in the value pathway harms accuracy, especially on non-stationary series. Second, simply replacing TRA with vanilla self-attention is insufficient even when PMD is retained. Third, TRA is most effective when applied after PVA, which suggests that temporal patch compression before recent cross-variable refinement is suboptimal. The paper characterizes PMDformer overall as outperforming existing state-of-the-art methods in stability and accuracy across multiple LTSF benchmarks (Hu et al., 25 Jun 2026).

In implementation terms, TRA is a parameter-shared Transformer encoder applied independently to each variable’s sequence of patches. The multi-head structure is not altered; TRA is described as a drop-in replacement for standard self-attention inside a Transformer block, with only the value definition modified. Patch means are computed once in PMD and reused twice: first in TRA when forming $\mu_i$9, and again in the projection layer. The reported patch sizes are $r_i$0, with moderate sizes $r_i$1, $r_i$2, and $r_i$3 described as the most robust. For input length $r_i$4, the number of patches is $r_i$5, and TRA has per-variable complexity $r_i$6. RevIN is used on inputs for distribution-shift handling, but this is presented as orthogonal to TRA itself (Hu et al., 25 Jun 2026).

6. Limitations, misconceptions, and terminological ambiguity

The PMDformer paper does not isolate TRA-specific limitations as a separate formal section, but several constraints are stated or implied. Because PMD represents trend as a scalar patch mean, the restored trend primarily captures level and slow drift. More complex trend structures, including nonlinear or multi-scale components, may not be fully represented by simple per-patch means. The same additive design choice appears in the value and projection stages: trend is injected through addition, which assumes that additive level adjustments are sufficient. A plausible implication is that settings with multiplicative or strongly nonlinear trend-shape interactions could require richer trend representations, such as learned filters, multi-scale trend tokens, gating mechanisms, or learned trend embeddings; these directions are explicitly mentioned as natural extensions in the PMDformer discussion (Hu et al., 25 Jun 2026).

A common misunderstanding is to treat TRA as a purely post-attention restoration step. In PMDformer, trend restoration is not confined to the output head. Patch means are reintroduced inside the attention computation itself through the value pathway and then added again before the final projection. Another misunderstanding is to read TRA as a cross-variable attention module. In PMDformer, that function belongs to PVA; TRA is explicitly per-variable and temporal, with shared parameters but no cross-variable mixing inside the TRA block (Hu et al., 25 Jun 2026).

There is also a nomenclatural ambiguity. The acronym “TRA” is used in a different 2025 paper to denote “Threshold Relative Attention,” a modified self-attention mechanism for length generalisation in decoder-only transformers. That mechanism uses ReLU thresholding on semantic logits, selective sparsity, and contextualised relative distance with a learned forget gate. It belongs to a distinct problem setting—out-of-distribution sequence length rather than LTSF trend restoration—and should not be conflated with Trend Restoration Attention in PMDformer (Opper et al., 29 Mar 2025).

Within time-series forecasting, the broader conceptual lineage of TRA is therefore best understood as part of a detrend-then-attend family of ideas. TDformer represents one version of that family by separating trend and seasonality into explicit modules and restoring trend additively after attention, whereas PMDformer implements a more localized patch-level variant in which de-biasing of attention and restoration of level occur inside a single temporal attention block. This suggests a shared design principle across otherwise different architectures: attention is most reliable when it aligns residual structure, while trend is preserved by an explicit restoration pathway rather than left implicit inside raw attention scores (Zhang et al., 2022).