Self-Gating Attention for Efficient Time Series Forecasting

Abstract: Transformer architectures have shown strong potential in time series forecasting, where multi-head self-attention is widely used to capture temporal dependencies across historical timestamps. However, standard self-attention has quadratic time and memory complexity with respect to the look-back length. This cost may limit its use in resource-constrained or high-throughput forecasting systems, where fast and memory-efficient inference is important. Through qualitative and quantitative analyses, we observe that self-attention maps in time series forecasting often contain redundant patterns across different timestamps. This phenomenon can be related to the repeated temporal patterns and relatively stable temporal correlations in many real-world time series. Motivated by this observation, we propose Self-Gating Attention (SGA), a plug-and-play attention mechanism that represents the attention score with a shared learnable matrix and an input-dependent residual component. The shared matrix captures common attention patterns, while the residual component captures input-dependent variations. In this way, SGA avoids the query and key projections used in standard attention score computation, leading to linear time and score-matrix memory complexity with respect to the look-back length. We integrate SGA into several forecasting backbones and compare it with standard self-attention and lightweight attention variants on nine publicly available real-world datasets covering electricity, finance, weather, medical monitoring, human activity, and climate records. The results show that SGA improves inference efficiency on public benchmarks while maintaining competitive forecasting performance against state-of-the-art attention mechanisms. These benchmark results provide deployment-oriented evidence.

• The paper demonstrates that conventional self-attention has high redundancy in time series data, with cosine similarities between score maps frequently exceeding 0.88.
• It introduces a self-gating attention architecture that replaces query-key projections with a shared global matrix and an adaptive, lightweight residual branch, achieving linear complexity.
• Empirical results show up to 26% improvement in MSE and over 60% reduction in FLOPs, validating its efficiency and robustness across diverse forecasting tasks.

Self-Gating Attention for Efficient Time Series Forecasting

Motivation and Redundancy Analysis

Transformer architectures, with self-attention (SA) mechanisms, are prevalent in time series forecasting due to their ability to capture long-range dependencies. However, standard SA incurs quadratic time and memory complexity relative to look-back length, which becomes prohibitive in resource-constrained and high-throughput environments. The core hypothesis driving this work is that, unlike NLP sequences, time series data often exhibit stable and repeated temporal correlation structures, rendering much of SA's query-key score computation redundant.

Qualitative and quantitative analyses reveal that attention score maps across different timestamps are highly similar within the time series context. For example, average cosine similarities between SA score maps regularly exceed 0.88, even when local input trends vary. Bootstrap analysis confirms that cross-time score similarity consistently exceeds baseline measures, with strong statistical significance. This suggests a substantial proportion of attention computation is superfluous, motivating a structured approach to share and economize this computation.

Figure 1: Visualization of attention redundancy on the ETTm1. (a) Input trends and ground truth at four selected timestamps. (b) SA score maps of the TimeXer backbone. (c) Residual score maps obtained by subtracting the mean score map $\alpha$. Sim denotes the average pairwise cosine similarity.

Self-Gating Attention (SGA): Architectural Design

SGA addresses attention redundancy through a dual-pronged strategy. It reparameterizes the attention score matrix as a fusion of (1) a shared learnable score matrix capturing stable global attention patterns, and (2) an input-dependent lightweight residual matrix accommodating local sequence-specific information. This design excises both the query and key projections of standard SA, yielding a linear scaling in both computation and score-matrix storage.

The shared matrix, unique per head and initialized orthogonally, acts as a universal temporal correlation prior. The residual, constructed from normalized second-order (energy) statistics of the value-projected input, enables the model to adapt to local variations with negligible parameter and FLOP overhead. Top-$K$ logit-level sparsification is applied to both matrices post-fusion to suppress weakly correlated historical positions.

Figure 2: Illustration of SGA. SGA uses a shared attention score matrix $\bm{\mathcal{A}}$ to model common attention patterns and an input-dependent residual score matrix $\bm{\mathcal{R}}_t$ to capture input-dependent variations.

Figure 4: Structure of SGA.

This construction preserves compatibility with multi-head and cross-attention settings and can be incorporated into arbitrary Transformer-based forecasting backbones.

Empirical Evaluation

Predictive Accuracy

Benchmarks across nine datasets (covering electricity, weather, finance, medical, human activity, and climate) and seven Transformer-style backbones demonstrate that SGA consistently attains first or second rank in forecasting performance in 84/90 cases. SGA exhibits up to 26% improvement in MSE over backbone defaults and delivers robust results for both regular and irregular sampling settings, underscoring its generalizability.

In horizon-wise breakdowns and Nemenyi statistical tests, SGA achieves the lowest average rank across all backbones, statistically surpassing ProbSparse, AutoCorrelation, and vanilla SA with significance.

Figure 6: Nemenyi critical difference diagram. Horizontal bars connect methods whose differences are not statistically significant.

Efficiency

SGA significantly reduces computational burden. With TimeXer as backbone, SGA achieves over 60% reductions in both FLOPs and parameters compared to vanilla SA—0.820M FLOPs and 67.3K parameters versus 2.398M FLOPs and 197.4K parameters, respectively. SGA maintains a strictly linear complexity in look-back length $n$, in contrast to the quadratic scaling of SA and Geometry Attention. Empirical wall-time measurements yield 1.19x–1.25x speedups in training and inference.

Figure 8: Efficiency comparison with the TimeXer backbone on the ETTh1.

Scaling and Robustness

Performance is stable with increased look-back windows. Unlike competing methods, SGA sustains its accuracy as $n$ grows, with limited increase in performance variance or degradation when input sequences are extended—suggesting effective use of extended historical context without overfitting to noise.

Figure 3: Performance comparison on the ETTh1 with the TimeXer backbone under different look-back lengths $n \in \{96, 192, 336, 512, 720\}$.

SGA is robust across a broad range of hyperparameters, including Top-$K$ and dropout ratios for both the shared and residual branches. Increasing the number of heads supports minor accuracy improvements, but diminishing returns are observed when exceeding four heads.

Figure 5: (a) Performance of SGA with different look-back lengths on the ETTh1 using the TimeXer backbone. The shaded region represents the standard deviation of MSE fitted by a B-spline. (b) Performance of SGA with different numbers of attention heads on the ETTh1.

Figure 7: Sensitivity analysis of SGA hyper-parameters on the ETTm1 using the TimeXer backbone.

Ablation Analysis

Both the shared matrix and normalized-energy-based residual are essential for full performance. Removal of either branch or sparsification entails consistent degradation. Replacing the residual with MLP or full attention-based constructions yields no improvements; the lightweight residual is optimal for balancing locality and efficiency.

Forecast Output Visualization

Comparative qualitative analysis reveals that SGA more accurately tracks target sequences' high-frequency variations and abrupt changes than other efficient attention mechanisms. In settings with shared training scripts and untuned hyperparameters, SGA delivers the lowest error metrics and superior visual alignment to ground truth.

Figure 9: Prediction visualization of six attention mechanisms with the same TimeXer backbone on the ETTh1.

Theoretical and Practical Implications

SGA establishes that conventional Transformer's computational overhead in time series domains is unnecessarily high due to the intrinsic redundancy of temporal correlations. By formalizing and exploiting this structural persistence, SGA renders attention computation scalable to long sequences and complex, multivariate, or irregular data, without sacrificing predictive fidelity.

From a theoretical standpoint, SGA challenges the universal applicability of query-key attention and advocates for domain-specific reallocation of model capacity and compute. The findings motivate further delineation of task-specific attention constructs—especially those leveraging topological or statistical characteristics inherent in target modalities.

Practically, SGA unlocks Transformer-based modeling for edge inference, online forecasting, and deployment in latency- and memory-sensitive industrial, medical, or financial monitoring scenarios. By aligning architectural design with empirical attention structure, SGA provides a drop-in module for the next generation of efficient forecasting models.

Potential Extensions and Future Directions

SGA opens several lines for future research:

• Concept drift adaptation: Integrating dynamic adaptation mechanisms for the shared matrix to better accommodate abrupt regime shifts or concept drift, particularly in non-stationary settings.
• Downstream sequence modeling: Evaluating SGA's principles in autoregressive generation or other structured prediction frameworks beyond forecasting.
• Efficient multi-variate structures: Extending SGA to explicitly partition and jointly model variable-wise and temporal attention redundancy.
• Hardware-specific optimizations: Custom kernels exploiting SGA's regularity for further acceleration on edge devices or FPGAs.

Conclusion

SGA demonstrates that a combination of a shared global attention score and a simple input-dependent residual suffices for effective time series attention, yielding significant computational savings without loss of prediction quality. This work substantiates the view that attention architectures should be adapted to the statistical regularities of the data modality, and SGA constitutes a practical, extensible approach towards scalable deep forecasting (2607.02344).