---
title: Proximal Variable Attention (PVA)
url: https://www.emergentmind.com/topics/proximal-variable-attention-pva
type: topic
---

# Proximal Variable Attention (PVA)

Proximal Variable Attention (PVA) is the cross-variable dependency module introduced in PMDformer, “Patch-Mean Decoupling Information Transformer for Long-term Forecasting,” for multivariate long-term time series forecasting (LTSF) [2606.26549]. In that architecture, PVA is designed for non-stationary settings in which cross-variable correlations vary over time and may become outdated far from the prediction horizon. Its defining operation is to apply multi-head self-attention only to the tokens of the most recent patch across variables, thereby focusing cross-variable modeling on the temporally proximal segment that the paper identifies as most relevant for forecasting. PVA is not a standalone forecasting framework; it is one of three coordinated components in PMDformer, alongside Patch-Mean Decoupling (PMD) and Trend Restoration Attention (TRA), with PMD making patch tokens shape-oriented, PVA modeling recent cross-variable shape similarity, and TRA modeling temporal dependencies while restoring trend information [2606.26549].

## 1. Problem setting and design motivation

PMDformer studies multivariate LTSF with a historical sequence
\[
X=\{x_t\in\mathbb{R}^C\mid t=1,2,\dots,L\},
\]
where \(C\) is the number of variables, and the objective is to predict future values
\[
Y=\{\hat x_t\in\mathbb{R}^C\mid t=L+1,\dots,L+T\}.
\]
The paper argues that patch-based transformers are attractive for LTSF because they compress long sequences into patch tokens and can model long-range dependencies, but it identifies two obstacles: attention may confuse scale with shape similarity, and cross-variable dependencies computed over the entire historical window may overfit to outdated or weakly relevant relationships [2606.26549].

PVA addresses the second difficulty. The paper states that accurate forecasting “hinges on the immediate interactions between variables at the most recent time steps,” and therefore cross-variable attention should be constrained to the temporally nearest region. In this formulation, “proximal” refers specifically to the most recent patch in time, not to locality in an image-like or spatial sense. PVA is thus a recency-focused cross-variable attention module whose inductive bias is temporal proximity to the forecast horizon rather than global aggregation over all historical segments [2606.26549].

This design is motivated by non-stationarity. In finance, traffic, and energy, a variable relationship that was strong earlier in the sequence may no longer be predictive near the forecast horizon. A plausible implication is that restricting variable interaction modeling to the latest patch can reduce the influence of stale couplings without discarding the full historical record entirely, because temporal history is still handled elsewhere in the architecture.

## 2. Formal definition and tensor structure

PVA is introduced in Section 3.2, “Model Architecture,” under the subsection “Proximal Variable Attention (PVA),” and is summarized in Section 3.1 as focusing self-attention “only on the \(C\) tokens of the last (proximal) patch” [2606.26549]. The model first partitions the input into \(N\) non-overlapping patches of length \(S\), where
\[
N=\lfloor L/S\rfloor.
\]
For variable \(i\in[C]\) and patch index \(j\in[N]\), the raw patch is
\[
P_j^i=(x^i_{(j-1)S+1},x^i_{(j-1)S+2},\dots,x^i_{jS})\in\mathbb{R}^S. \tag{1}
\]

PMD then computes the patch mean and residual:
\[
p_j^i=\frac{1}{S}\sum_{k=1}^{S}x^i_{(j-1)S+k},\qquad r_j^i=P_j^i-p_j^i\mathbf{1}_S, \tag{2}
\]
where \(\mathbf{1}_S\) is the all-ones vector in \(\mathbb{R}^S\). Each residual patch is embedded as
\[
\tilde P_j^i:=r_j^i W_E+b_E+Z_{p_j}, \tag{3}
\]
with \(W_E\in\mathbb{R}^{S\times d}\), \(b_E\in\mathbb{R}^{d}\), and positional embedding \(Z_{p_j}\in\mathbb{R}^d\) [2606.26549].

PVA operates only on the most recent patch. Let \(N\) denote the index of the last patch. The model collects the most recent tokens of all \(C\) variables,
\[
P_N=\{P_N^1,\dots,P_N^C\},
\]
with each token in \(\mathbb{R}^d\), equivalently \(P_N\in\mathbb{R}^{C\times d}\) for one sample. The PVA block is defined by
\[
\bar P_N=\operatorname{LayerNorm}(\operatorname{MHSA}(P_N)+P_N), \tag{4}
\]
\[
\hat P_N=\operatorname{LayerNorm}(\operatorname{FFN}(\bar P_N)+\bar P_N). \tag{5}
\]
These equations describe a standard Transformer-style update consisting of MHSA with residual connection and LayerNorm, followed by FFN with residual connection and LayerNorm [2606.26549].

The full embedded token tensor is
\[
P\in\mathbb{R}^{C\times N\times d}.
\]
After PVA updates the last-patch slice, the earlier historical patches \(1,\dots,N-1\) remain unchanged, and the refined proximal tokens are concatenated back with them along the patch dimension to reconstruct the full token sequence [2606.26549].

## 3. Operational mechanism and architectural role

Mechanistically, PVA does not consume raw time points or raw patches. Its inputs are PMD-embedded residual patch tokens, so the module compares variables using mean-decoupled patch representations rather than raw levels. For each variable \(i\), the model has a sequence of tokens
\[
(\tilde P_1^i,\tilde P_2^i,\dots,\tilde P_N^i),
\]
and PVA extracts only the final token from each variable for cross-variable interaction [2606.26549].

The selection rule is hard and deterministic: only the last patch is used for cross-variable attention. Earlier tokens are excluded from the cross-variable attention computation. The paper explicitly states that PVA does not search over temporal neighborhoods, does not apply a recency-weighted soft mask over multiple recent patches, and does not perform local attention over adjacent patches. The proximal inductive bias is therefore implemented by restricting the attention input set, not by introducing a learned gate, top-\(k\) rule, sparsity prior, or proximity bias term inside the attention logits [2606.26549].

Within the selected set \(P_N\), every variable token can attend to every other variable token in the last patch. The paper does not expand MHSA into explicit \(Q\), \(K\), and \(V\) equations for PVA, although standard Transformer semantics are implied. After the MHSA and FFN updates, the refined proximal token set is concatenated back with the untouched historical tokens to form the shape-embedding sequence used by later modules [2606.26549].

PVA is positioned between PMD embedding and TRA in PMDformer. The architectural sequence is: input time series, RevIN as shown in Figure 3, patching and PMD, embedding, PVA, TRA, and projection. TRA is then applied per variable along the patch axis with
\[
Q^i=P^iW_Q,\quad K^i=P^iW_K, \tag{6}
\]
\[
A^i=\operatorname{Softmax}\!\left(\frac{Q^i(K^i)^T}{\sqrt d}\right), \tag{7}
\]
\[
V^i=P^iW_V+p^i, \tag{8}
\]
followed by projection
\[
\hat Y^i=(P^i+p^i)W_o+b_o,\qquad \hat Y^i\in\mathbb{R}^T. \tag{9}
\]
This division of labor means that PVA handles recent cross-variable shape interactions, while TRA handles temporal modeling across all patches for each variable and restores the decoupled trend through the value pathway [2606.26549].

## 4. Relation to standard attention and computational profile

PVA differs from standard full self-attention over patch-variable tokens in two explicit ways. First, it does not attend over all \(C\times N\) tokens jointly; it applies self-attention only to the \(C\) tokens from the most recent patch. Second, it excludes earlier historical patches from cross-variable interaction modeling, whereas a full-history alternative would compute variable-wise shape similarity across all patches [2606.26549].

The contrast can be stated structurally. A fuller cross-variable design would consider interactions such as
\[
(P_j^i \leftrightarrow P_{j'}^{i'}) \quad \text{for many } j,j',
\]
whereas PVA models only
\[
(P_N^i \leftrightarrow P_N^{i'}) \quad \text{for } i,i'\in[C].
\]
The paper attributes to this restriction a reduction in complexity from
\[
O(C^2N)
\]
to
\[
O(C^2).
\]
It does not provide a more detailed derivation, but the stated intuition is that full cross-variable modeling scales with the number of patches, while PVA computes pairwise variable interactions only once at the proximal patch [2606.26549].

In this sense, PVA is not a generic sparse-attention mechanism. It is a specific temporal restriction on cross-variable attention. A common misconception is to interpret “proximal” as local attention over several recent segments or as a soft recency-weighted mechanism; the paper’s definition is stricter. In the main model, “proximal” means the single last patch. A plausible implication is that the module should be understood less as a new attention kernel and more as a structured inductive bias on where variable dependency estimation is permitted.

## 5. Empirical evidence and ablation results

The main empirical evidence for PVA appears in Table 4 and the sensitivity analysis in Section 4.3 [2606.26549]. The paper compares PMDformer with default PVA, a variant that replaces PVA with all-token cross-variable attention, variants without PVA, and a variant that swaps the order of TRA and PVA.

| Setting | ETTh2 | ETTm1 |
|---|---:|---:|
| PMDformer (TRA, Last Token PVA) | 0.337 / 0.382 | 0.346 / 0.371 |
| Replace PVA with All Token | 0.340 / 0.384 | 0.354 / 0.375 |
| Remove PVA, keep TRA | 0.340 / 0.383 | 0.347 / 0.371 |
| Swap module order | 0.342 / 0.385 | 0.350 / 0.372 |

| Setting | Traffic | Solar |
|---|---:|---:|
| PMDformer (TRA, Last Token PVA) | 0.378 / 0.234 | 0.181 / 0.211 |
| Replace PVA with All Token | 0.380 / 0.239 | 0.186 / 0.214 |
| Remove PVA, keep TRA | 0.386 / 0.240 | 0.194 / 0.214 |
| Remove both TRA and PVA | 0.426 / 0.288 | 0.222 / 0.230 |
| Swap module order | 0.379 / 0.235 | 0.188 / 0.216 |

These results support three explicit claims made in the paper. First, restricting cross-variable attention to the last patch is better than applying it to all patches. On Solar, the MSE changes from 0.181 to 0.186 when PVA is replaced by all-token variable attention, and on ETTm1 it changes from 0.346 to 0.354. Second, removing PVA degrades performance, particularly on Traffic and Solar, where the MSE changes from 0.378 to 0.386 and from 0.181 to 0.194, respectively. Third, the order of modules matters: placing PVA before TRA is better than swapping them, which the authors explain by arguing that TRA applied first “compresses patch information too early,” making later cross-variable modeling less effective [2606.26549].

The paper also compares PMDformer with iTransformer and reports an average MSE reduction of 11.44% and MAE reduction of 12.38%. It interprets part of this improvement through PVA, arguing that iTransformer embeds dependencies among variable tokens from the entire historical sequence and can therefore overfit to early, weakly relevant variable relationships, whereas PVA focuses on shape similarities within the most nearest patch [2606.26549]. This is not an isolated PVA-only comparison, but it is the authors’ stated explanation.

Section 4.3 further studies the number of nearest patches used for cross-variable modeling, with \(k\in\{1,2,3,5,7,10\}\), evaluated on ETTh1 and Solar using MSE across prediction horizons. The paper reports that performance generally worsens as \(k\) increases and that \(k=1\) gives the most stable and strongest performance. This directly reinforces the default PVA design choice that the optimal proximal window is a single most recent patch [2606.26549].

## 6. Implementation details, ambiguities, and scope

Several implementation-relevant details are recoverable from the paper. The number of patches is
\[
N=\lfloor L/S\rfloor,
\]
the PVA input slice is \(P_N\in\mathbb{R}^{C\times d}\) per sample, the model typically uses input length \(L=720\) in experiments, and the final design uses one selected patch, that is, last token or \(k=1\). The PVA block uses standard MHSA, FFN, LayerNorm, and residual connections [2606.26549].

At the same time, the paper leaves several aspects underspecified. It does not give explicit \(Q\), \(K\), and \(V\) equations for PVA; it does not specify the number of attention heads, FFN hidden dimension, dropout rates, parameter-sharing choices, exact batch tensor ordering, or whether the same PVA block is stacked multiple times. It also does not define any additional mask implementation beyond restricting attention to the last patch. As a result, exact reproduction from the paper text alone requires either the code or supplementary hyperparameter details [2606.26549].

The notation is also slightly overloaded around \(P_N\), because the same symbol is reused for the selected input slice and its refined version, although the textual meaning remains clear. Another limitation is conceptual rather than notational: the main mathematical definition hard-codes “proximal” to the last patch, while the sensitivity analysis studies \(k>1\) without giving a generalized formal equation for that case [2606.26549].

The paper’s rationale assumes that the most recent cross-variable interactions are the most predictive. It explicitly notes this as plausible in non-stationary domains, but it also implies a potential scope limitation: older inter-variable couplings may sometimes remain genuinely important. The authors do not discuss such failure modes in depth, although their evaluation over multiple \(k\) values indicates awareness of the trade-off. A plausible implication is that PVA is best interpreted as a strong recency prior for cross-variable dependency modeling rather than as a universally optimal treatment of multivariate structure.

## 7. Interpretive significance within PMDformer

Within PMDformer, PVA operationalizes a precise modeling claim: cross-variable dependence should be estimated from recent residual shape states rather than from the entire historical record [2606.26549]. Because PMD removes patch means before tokenization, PVA receives tokens that emphasize local residual pattern shape rather than scale. Because PVA only uses the last patch, it suppresses reliance on old cross-variable couplings that may be stale, spurious, or weakly relevant near the forecasting horizon. Because TRA follows afterward and processes all historical patches temporally for each variable, the architecture does not discard history outright; instead, it reallocates temporal history modeling to a different branch.

This separation of concerns is central to the architecture’s logic. PMD handles scale bias by subtracting the mean of each patch, PVA models recent cross-variable shape similarity using only the proximal patch, and TRA restores the decoupled trend while modeling temporal shape matching across patches [2606.26549]. The ablation results indicate that each choice is consequential, and that PVA is not merely an efficiency device. It is the module through which PMDformer encodes the paper’s central assumption about multivariate LTSF: the most predictive cross-variable relationships are the recent, shape-based ones closest to the forecast horizon.

Source: https://www.emergentmind.com/topics/proximal-variable-attention-pva