VARMAformer: VARMA-Inspired Transformer

• VARMAformer is a Transformer-based forecasting model that integrates VARMA-inspired techniques with a cross-attention-only backbone to embed local temporal structure.
• It employs a VARMA-inspired Feature Extractor and AR/MA branching to explicitly model patch-level autoregressive and moving-average dynamics within the decoder.
• Empirical results on multiple datasets demonstrate that VARMAformer outperforms state-of-the-art methods, especially in long-horizon forecasting scenarios.

Searching arXiv for the specified paper and closely related context. VARMAformer is a Transformer architecture for multivariate time series forecasting that combines a cross-attention-only backbone with explicitly VARMA-inspired modeling of local temporal structure (Song et al., 5 Sep 2025). It is designed to address a specific limitation of streamlined Transformer forecasters: while cross-attention-only designs can improve efficiency and accuracy, they may underutilize the fine-grained local dependencies traditionally modeled by Vector AutoRegressive Moving Average processes. The model therefore introduces two coupled mechanisms—a VARMA-inspired Feature Extractor (VFE) and a VARMA-Enhanced Attention (VE-atten) module—to incorporate patch-level autoregressive and moving-average structure into a modern decoder-centric forecasting stack.

1. Architectural configuration

VARMAformer operates on an input sequence $X \in \mathbb{R}^{C \times L}$ that is first normalized by instance normalization and then partitioned into $N$ non-overlapping patches by replication padding and patching. The resulting patch sequence is

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$

The architecture omits a self-attention encoder entirely. Instead, patch embeddings are used directly as keys $K$ and values $V$ in a cross-attention decoder, while learnable dummy queries

$$Q_{\text{dummy}} \in \mathbb{R}^{C \times N \times D}$$

represent the future patches to be predicted.

The decoder consists of a stack of $L$ identical layers. Each layer contains a VE-atten block, a standard position-wise feed-forward network, LayerNorm, residual connections, and Query-Adaptive Masking, described in the paper as a form of structured stochastic depth (Song et al., 5 Sep 2025). Queries are obtained from a learnable projection of $Q_{\text{dummy}}$, whereas the patch embeddings define a persistent memory bank through $K$ and $V$.

Patch embeddings are formed by projecting each raw patch $N$0 with $N$1 into the model dimension $N$2, adding positional encoding $N$3, and injecting the VFE output through a learnable scaling parameter $N$4:

$N$5

This construction places classical time-series structure directly in the tokenization and memory representation rather than treating it as an auxiliary regularizer or post hoc correction.

2. VARMA-inspired Feature Extractor

The VFE is the module through which patch-level AR and MA structure is made explicit. For each patch index $N$6, it computes two half-dimensional feature tensors,

$N$7

and then fuses them into a full $N$8-dimensional enhancement tensor (Song et al., 5 Sep 2025).

For the autoregressive component, the model learns $N$9 scalar coefficients $$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$0 and forms

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$1

where $$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$2 and

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$3

is a learned linear layer. In the paper’s simplified VARMA notation, this corresponds to

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$4

For the moving-average component, the model defines proxy residuals for preceding patches by first-order differencing:

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$5

With learnable scalars $$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$6, it computes

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$7

where

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$8

The corresponding MA-style approximation is

$$X_p = \{x_p^{(1)}, \ldots, x_p^{(N)}\}, \qquad x_p^{(i)} \in \mathbb{R}^{C \times P}.$$9

The AR and MA branches are concatenated and fused:

$K$0

$K$1

Collecting these terms over all $K$2 yields the full enhancement tensor in $K$3. The model therefore treats VARMA structure not as an external statistical preprocessor, but as a differentiable architectural prior embedded at the patch level.

3. VARMA-Enhanced Attention

The VE-atten mechanism modifies standard cross-attention by making the query stream depend on aggregate temporal context. Standard cross-attention is written as

$K$4

VARMAformer preserves this form for score computation but first applies a learned temporal gate to the queries (Song et al., 5 Sep 2025).

The gate begins by mean-pooling the keys along the patch axis:

$K$5

This pooled context is passed through a two-layer gate network with sigmoid activation:

$K$6

so that

$K$7

Given the layer-$K$8 query tensor $K$9, the gated query tensor is defined as

$V$0

where $V$1 denotes element-wise multiplication and $V$2 is learned as a scalar or vector. Attention weights are then computed from $V$3:

$V$4

The immediate effect is that query formation depends on global patch statistics before dot-product matching is performed. The paper characterizes this as a global “VARMA-style” gate that makes cross-attention queries more context-aware. A plausible implication is that the mechanism partially compensates for the absence of self-attention by injecting a lightweight global summary directly into the decoder-side retrieval process. Each VE-atten block is followed by residual addition and LayerNorm exactly as in a standard Transformer decoder layer.

4. Training protocol and objective formulation

The primary optimization target is Mean Squared Error:

$V$5

In some experiments, Mean Absolute Error is also monitored (Song et al., 5 Sep 2025). No additional VARMA-specific regularizer is introduced; instead, the AR and MA inductive bias is imposed structurally through the VFE and its associated projection weights $V$6 and $V$7.

The reported optimization configuration uses the Adam optimizer, a fixed look-back of 96, forecast horizons $V$8, learning-rate scheduling with warm-up and decay, and early stopping on validation MSE. Query-Adaptive Masking is included to prevent query overfitting. These design choices position VARMAformer within the contemporary Transformer training regime while preserving the paper’s core claim that classical statistical structure can be integrated directly at the architectural level rather than through specialized loss engineering.

The benchmark suite comprises seven datasets: ETTm1, ETTm2, ETTh1, ETTh2, Electricity, Traffic, and Weather. The ETTm1 and ETTm2 variants are minutely, ETTh1 and ETTh2 are hourly, Electricity contains hourly power consumption, Traffic uses occupancy rate, and Weather contains climatological indicators with minutely sampling. Evaluation is reported with MSE and MAE over the four forecast horizons.

5. Empirical performance

The experimental study compares VARMAformer against CATS, PatchTST, TimesNet, Basisformer, Crossformer, FEDformer, Autoformer, and Informer (Song et al., 5 Sep 2025). Across 56 test scenarios—computed as 7 datasets, 4 horizons, and 2 metrics—the model is reported as best in 50. The paper also states that VARMAformer consistently outperforms existing state-of-the-art methods on widely used benchmark datasets.

A concrete example is provided on ETTh1 for MSE at horizon 720: VARMAformer records 0.439, compared with 0.474 for CATS and 0.544 for PatchTST. On the Weather dataset, the paper reports that VARMAformer is top in MSE at every horizon. The gains are described as increasing with longer forecast horizons, which the paper interprets as evidence that the VFE and VE-atten combination yields more robust long-range forecasts than pure cross-attention alone.

The empirical pattern is significant because the architecture simultaneously removes the self-attention encoder and introduces explicit local statistical structure. This contrasts with a common assumption that long-range forecasting gains in Transformer models must come primarily from more elaborate attention mechanisms. Here, the reported results instead associate stronger long-horizon performance with a hybrid of streamlined cross-attention and explicitly modeled AR/MA dynamics.

6. Ablation findings, limitations, and future directions

The ablation study on ETTh1 and ETTh2 isolates the contributions of the AR branch, the MA branch, and VE-atten (Song et al., 5 Sep 2025). On ETTh1, the baseline with none of these modules gives MSE 0.371 at horizon 96 and 0.474 at horizon 720. Adding AR only yields 0.364 and 0.449; MA only yields 0.364 and 0.448; VE-atten only yields 0.365 and 0.450; AR+MA yields 0.364 and 0.444; and the full AR+MA+VE-atten model yields 0.363 and 0.439. On ETTh2, the corresponding baseline is 0.287 at horizon 96 and 0.412 at horizon 720; AR only gives 0.280 and 0.409; MA only gives 0.280 and 0.406; VE-atten only gives 0.283 and 0.408; AR+MA gives 0.280 and 0.406; and AR+MA+VE-atten yields 0.279 and 0.403. The reported conclusion is that all components contribute and that the full VARMAformer is strictly best.

The limitations identified in the paper are specific. On very high-frequency, noisy data such as ETTm1 and Weather, the relative gains over strong baselines are more modest, with the stated explanation that fine-grained stochastic volatility can obscure local AR/MA signals. The current MA proxy is also explicitly described as simplistic because it relies on first-order patch differences and may therefore fail to capture more complex innovation structure. This establishes an important interpretive boundary: VARMAformer is VARMA-inspired, but it does not instantiate a full classical VARMA estimation procedure.

Future directions proposed in the paper include more robust noise-tolerant MA proxies, such as higher-order differencing or learned error estimators; extension to probabilistic forecasting through full VARMA-style covariances or quantile forecasts; integration of state-space models and other classical priors such as GARCH and seasonal ARIMA into the Transformer architecture; and cross-variate patch-level VARMA to model inter-series dynamics as a full multivariate VARMA rather than per-channel patch features. Taken together, these directions situate VARMAformer within a broader research program that seeks tighter integration between classical stochastic time-series structure and neural sequence modeling.