Proximal Variable Attention (PVA)

• Proximal Variable Attention (PVA) is defined as a cross-variable dependency module that applies multi-head self-attention exclusively to the most recent patch tokens.
• It focuses on capturing immediate shape similarities among variable tokens to mitigate overfitting from outdated interactions in non-stationary environments.
• Empirical results demonstrate that incorporating PVA in PMDformer yields lower error metrics and improved forecasting efficiency compared to full historical attention methods.

Proximal Variable Attention (PVA) is the cross-variable dependency module introduced in PMDformer, “Patch-Mean Decoupling Information Transformer for Long-term Forecasting,” for multivariate long-term time series forecasting (LTSF) (Hu et al., 25 Jun 2026). In that architecture, PVA is designed for non-stationary settings in which cross-variable correlations vary over time and may become outdated far from the prediction horizon. Its defining operation is to apply multi-head self-attention only to the tokens of the most recent patch across variables, thereby focusing cross-variable modeling on the temporally proximal segment that the paper identifies as most relevant for forecasting. PVA is not a standalone forecasting framework; it is one of three coordinated components in PMDformer, alongside Patch-Mean Decoupling (PMD) and Trend Restoration Attention (TRA), with PMD making patch tokens shape-oriented, PVA modeling recent cross-variable shape similarity, and TRA modeling temporal dependencies while restoring trend information (Hu et al., 25 Jun 2026).

1. Problem setting and design motivation

PMDformer studies multivariate LTSF with a historical sequence

$X=\{x_t\in\mathbb{R}^C\mid t=1,2,\dots,L\},$

where $C$ is the number of variables, and the objective is to predict future values

$Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$

The paper argues that patch-based transformers are attractive for LTSF because they compress long sequences into patch tokens and can model long-range dependencies, but it identifies two obstacles: attention may confuse scale with shape similarity, and cross-variable dependencies computed over the entire historical window may overfit to outdated or weakly relevant relationships (Hu et al., 25 Jun 2026).

PVA addresses the second difficulty. The paper states that accurate forecasting “hinges on the immediate interactions between variables at the most recent time steps,” and therefore cross-variable attention should be constrained to the temporally nearest region. In this formulation, “proximal” refers specifically to the most recent patch in time, not to locality in an image-like or spatial sense. PVA is thus a recency-focused cross-variable attention module whose inductive bias is temporal proximity to the forecast horizon rather than global aggregation over all historical segments (Hu et al., 25 Jun 2026).

This design is motivated by non-stationarity. In finance, traffic, and energy, a variable relationship that was strong earlier in the sequence may no longer be predictive near the forecast horizon. A plausible implication is that restricting variable interaction modeling to the latest patch can reduce the influence of stale couplings without discarding the full historical record entirely, because temporal history is still handled elsewhere in the architecture.

2. Formal definition and tensor structure

PVA is introduced in Section 3.2, “Model Architecture,” under the subsection “Proximal Variable Attention (PVA),” and is summarized in Section 3.1 as focusing self-attention “only on the $C$ tokens of the last (proximal) patch” (Hu et al., 25 Jun 2026). The model first partitions the input into $N$ non-overlapping patches of length $S$, where

$N=\lfloor L/S\rfloor.$

For variable $i\in[C]$ and patch index $j\in[N]$, the raw patch is

$$P_j^i=(x^i_{(j-1)S+1},x^i_{(j-1)S+2},\dots,x^i_{jS})\in\mathbb{R}^S. \tag{1}$$

PMD then computes the patch mean and residual: $C$0 where $C$1 is the all-ones vector in $C$2. Each residual patch is embedded as

$C$3

with $C$4, $C$5, and positional embedding $C$6 (Hu et al., 25 Jun 2026).

PVA operates only on the most recent patch. Let $C$7 denote the index of the last patch. The model collects the most recent tokens of all $C$8 variables,

$C$9

with each token in $Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$0, equivalently $Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$1 for one sample. The PVA block is defined by

$Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$2

$Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$3

These equations describe a standard Transformer-style update consisting of MHSA with residual connection and LayerNorm, followed by FFN with residual connection and LayerNorm (Hu et al., 25 Jun 2026).

The full embedded token tensor is

$Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$4

After PVA updates the last-patch slice, the earlier historical patches $Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$5 remain unchanged, and the refined proximal tokens are concatenated back with them along the patch dimension to reconstruct the full token sequence (Hu et al., 25 Jun 2026).

3. Operational mechanism and architectural role

Mechanistically, PVA does not consume raw time points or raw patches. Its inputs are PMD-embedded residual patch tokens, so the module compares variables using mean-decoupled patch representations rather than raw levels. For each variable $Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$6, the model has a sequence of tokens

$Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$7

and PVA extracts only the final token from each variable for cross-variable interaction (Hu et al., 25 Jun 2026).

The selection rule is hard and deterministic: only the last patch is used for cross-variable attention. Earlier tokens are excluded from the cross-variable attention computation. The paper explicitly states that PVA does not search over temporal neighborhoods, does not apply a recency-weighted soft mask over multiple recent patches, and does not perform local attention over adjacent patches. The proximal inductive bias is therefore implemented by restricting the attention input set, not by introducing a learned gate, top-$Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$8 rule, sparsity prior, or proximity bias term inside the attention logits (Hu et al., 25 Jun 2026).

Within the selected set $Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.$9, every variable token can attend to every other variable token in the last patch. The paper does not expand MHSA into explicit $C$0, $C$1, and $C$2 equations for PVA, although standard Transformer semantics are implied. After the MHSA and FFN updates, the refined proximal token set is concatenated back with the untouched historical tokens to form the shape-embedding sequence used by later modules (Hu et al., 25 Jun 2026).

PVA is positioned between PMD embedding and TRA in PMDformer. The architectural sequence is: input time series, RevIN as shown in Figure 1, patching and PMD, embedding, PVA, TRA, and projection. TRA is then applied per variable along the patch axis with

$C$3

$C$4

$C$5

followed by projection

$C$6

This division of labor means that PVA handles recent cross-variable shape interactions, while TRA handles temporal modeling across all patches for each variable and restores the decoupled trend through the value pathway (Hu et al., 25 Jun 2026).

4. Relation to standard attention and computational profile

PVA differs from standard full self-attention over patch-variable tokens in two explicit ways. First, it does not attend over all $C$7 tokens jointly; it applies self-attention only to the $C$8 tokens from the most recent patch. Second, it excludes earlier historical patches from cross-variable interaction modeling, whereas a full-history alternative would compute variable-wise shape similarity across all patches (Hu et al., 25 Jun 2026).

The contrast can be stated structurally. A fuller cross-variable design would consider interactions such as

$C$9

whereas PVA models only

$N$0

The paper attributes to this restriction a reduction in complexity from

$N$1

to

$N$2

It does not provide a more detailed derivation, but the stated intuition is that full cross-variable modeling scales with the number of patches, while PVA computes pairwise variable interactions only once at the proximal patch (Hu et al., 25 Jun 2026).

In this sense, PVA is not a generic sparse-attention mechanism. It is a specific temporal restriction on cross-variable attention. A common misconception is to interpret “proximal” as local attention over several recent segments or as a soft recency-weighted mechanism; the paper’s definition is stricter. In the main model, “proximal” means the single last patch. A plausible implication is that the module should be understood less as a new attention kernel and more as a structured inductive bias on where variable dependency estimation is permitted.

5. Empirical evidence and ablation results

The main empirical evidence for PVA appears in Table 4 and the sensitivity analysis in Section 4.3 (Hu et al., 25 Jun 2026). The paper compares PMDformer with default PVA, a variant that replaces PVA with all-token cross-variable attention, variants without PVA, and a variant that swaps the order of TRA and PVA.

Setting	ETTh2	ETTm1
PMDformer (TRA, Last Token PVA)	0.337 / 0.382	0.346 / 0.371
Replace PVA with All Token	0.340 / 0.384	0.354 / 0.375
Remove PVA, keep TRA	0.340 / 0.383	0.347 / 0.371
Swap module order	0.342 / 0.385	0.350 / 0.372

Setting	Traffic	Solar
PMDformer (TRA, Last Token PVA)	0.378 / 0.234	0.181 / 0.211
Replace PVA with All Token	0.380 / 0.239	0.186 / 0.214
Remove PVA, keep TRA	0.386 / 0.240	0.194 / 0.214
Remove both TRA and PVA	0.426 / 0.288	0.222 / 0.230
Swap module order	0.379 / 0.235	0.188 / 0.216

These results support three explicit claims made in the paper. First, restricting cross-variable attention to the last patch is better than applying it to all patches. On Solar, the MSE changes from 0.181 to 0.186 when PVA is replaced by all-token variable attention, and on ETTm1 it changes from 0.346 to 0.354. Second, removing PVA degrades performance, particularly on Traffic and Solar, where the MSE changes from 0.378 to 0.386 and from 0.181 to 0.194, respectively. Third, the order of modules matters: placing PVA before TRA is better than swapping them, which the authors explain by arguing that TRA applied first “compresses patch information too early,” making later cross-variable modeling less effective (Hu et al., 25 Jun 2026).

The paper also compares PMDformer with iTransformer and reports an average MSE reduction of 11.44% and MAE reduction of 12.38%. It interprets part of this improvement through PVA, arguing that iTransformer embeds dependencies among variable tokens from the entire historical sequence and can therefore overfit to early, weakly relevant variable relationships, whereas PVA focuses on shape similarities within the most nearest patch (Hu et al., 25 Jun 2026). This is not an isolated PVA-only comparison, but it is the authors’ stated explanation.

Section 4.3 further studies the number of nearest patches used for cross-variable modeling, with $N$3, evaluated on ETTh1 and Solar using MSE across prediction horizons. The paper reports that performance generally worsens as $N$4 increases and that $N$5 gives the most stable and strongest performance. This directly reinforces the default PVA design choice that the optimal proximal window is a single most recent patch (Hu et al., 25 Jun 2026).

6. Implementation details, ambiguities, and scope

Several implementation-relevant details are recoverable from the paper. The number of patches is

$N$6

the PVA input slice is $N$7 per sample, the model typically uses input length $N$8 in experiments, and the final design uses one selected patch, that is, last token or $N$9. The PVA block uses standard MHSA, FFN, LayerNorm, and residual connections (Hu et al., 25 Jun 2026).

At the same time, the paper leaves several aspects underspecified. It does not give explicit $S$0, $S$1, and $S$2 equations for PVA; it does not specify the number of attention heads, FFN hidden dimension, dropout rates, parameter-sharing choices, exact batch tensor ordering, or whether the same PVA block is stacked multiple times. It also does not define any additional mask implementation beyond restricting attention to the last patch. As a result, exact reproduction from the paper text alone requires either the code or supplementary hyperparameter details (Hu et al., 25 Jun 2026).

The notation is also slightly overloaded around $S$3, because the same symbol is reused for the selected input slice and its refined version, although the textual meaning remains clear. Another limitation is conceptual rather than notational: the main mathematical definition hard-codes “proximal” to the last patch, while the sensitivity analysis studies $S$4 without giving a generalized formal equation for that case (Hu et al., 25 Jun 2026).

The paper’s rationale assumes that the most recent cross-variable interactions are the most predictive. It explicitly notes this as plausible in non-stationary domains, but it also implies a potential scope limitation: older inter-variable couplings may sometimes remain genuinely important. The authors do not discuss such failure modes in depth, although their evaluation over multiple $S$5 values indicates awareness of the trade-off. A plausible implication is that PVA is best interpreted as a strong recency prior for cross-variable dependency modeling rather than as a universally optimal treatment of multivariate structure.

7. Interpretive significance within PMDformer

Within PMDformer, PVA operationalizes a precise modeling claim: cross-variable dependence should be estimated from recent residual shape states rather than from the entire historical record (Hu et al., 25 Jun 2026). Because PMD removes patch means before tokenization, PVA receives tokens that emphasize local residual pattern shape rather than scale. Because PVA only uses the last patch, it suppresses reliance on old cross-variable couplings that may be stale, spurious, or weakly relevant near the forecasting horizon. Because TRA follows afterward and processes all historical patches temporally for each variable, the architecture does not discard history outright; instead, it reallocates temporal history modeling to a different branch.

This separation of concerns is central to the architecture’s logic. PMD handles scale bias by subtracting the mean of each patch, PVA models recent cross-variable shape similarity using only the proximal patch, and TRA restores the decoupled trend while modeling temporal shape matching across patches (Hu et al., 25 Jun 2026). The ablation results indicate that each choice is consequential, and that PVA is not merely an efficiency device. It is the module through which PMDformer encodes the paper’s central assumption about multivariate LTSF: the most predictive cross-variable relationships are the recent, shape-based ones closest to the forecast horizon.