iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (2310.06625v4)

Abstract: The recent boom of linear forecasting models questions the ongoing passion for architectural modifications of Transformer-based forecasters. These forecasters leverage Transformers to model the global dependencies over temporal tokens of time series, with each token formed by multiple variates of the same timestamp. However, Transformers are challenged in forecasting series with larger lookback windows due to performance degradation and computation explosion. Besides, the embedding for each temporal token fuses multiple variates that represent potential delayed events and distinct physical measurements, which may fail in learning variate-centric representations and result in meaningless attention maps. In this work, we reflect on the competent duties of Transformer components and repurpose the Transformer architecture without any modification to the basic components. We propose iTransformer that simply applies the attention and feed-forward network on the inverted dimensions. Specifically, the time points of individual series are embedded into variate tokens which are utilized by the attention mechanism to capture multivariate correlations; meanwhile, the feed-forward network is applied for each variate token to learn nonlinear representations. The iTransformer model achieves state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across different variates, and better utilization of arbitrary lookback windows, making it a nice alternative as the fundamental backbone of time series forecasting. Code is available at this repository: https://github.com/thuml/iTransformer.

Insights on iTransformer: Inverted Transformers for Time Series Forecasting

The paper "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting" proposes a novel perspective on leveraging Transformers for multivariate time series forecasting tasks. It addresses inherent inefficiencies in current approaches that apply temporal tokens, emphasizing instead the construction of variate tokens. This paper identifies and solves core challenges faced when employing standard Transformer architectures in time series problems, especially those with multivariate dimensions and long lookback windows.

Problem Statement and Challenges

Traditional Transformer-based models face significant hurdles when applied to time series forecasting. The embeddings of temporal tokens typically fuse multi-variate data, leading to potential misalignment and inefficient inter-variable attention mechanisms. This results in degraded performance, computational inefficiency, and poorly interpretable attention maps. The existing structure fails to accommodate larger lookback windows due to computational constraints and negligible modeling advantages.

Proposed Approach

The authors introduce iTransformer, a model that applies the attention and feed-forward network on inverted dimensions. Instead of embedding multiple variables for a singular time step, iTransformer treats the entire series of each variate as independent tokens. This reversal of the token approach enhances the model's capacity to capture multivariate correlations efficiently and effectively.

Key aspects of iTransformer include:

• Embedding: Each time series variate is embedded independently, allowing for better integration of series-specific information.
• Attention Mechanism: Attention is focused on these variate tokens, facilitating enhanced interpretability and revealing more accurate multivariate correlations.
• Feed-Forward Network: Applied across temporal sequences, this network ensures nonlinear representation learning, critical for capturing global time series trends.

Evaluation and Results

iTransformer demonstrates state-of-the-art performance across several real-world datasets, significantly outperforming existing Transformer models. The evaluation showcases its robust capacity to handle extensive lookback windows and effectively generalize across unseen variates. Key results highlight the model's efficiency in processing high-dimensional time series.

Practical and Theoretical Implications

The paper acknowledges the inadequacies of conventional time series tokenization within Transformers and challenges the prevailing use of temporal embedding strategies. It highlights the potential of inverted dimension models in providing more meaningful and computationally efficient solutions. This approach not only rectifies inefficiencies in multivariate representation but also paves the way for transformers to serve as fundamental backbones in complex temporal forecasting scenarios.

Future Directions

This paper opens discussions for leveraging efficient attention mechanisms tailored for multivariate processes. Future work may delve into enhancing the extraction of temporal features with advanced linear and non-linear modeling. Additionally, there's significant potential in exploring pre-training paradigms specific to time series tasks using the iTransformer architecture, to further elevate its utility across wider domains.

In conclusion, iTransformer presents a pivotal shift in Transformer application to time series forecasting, yielding promising results and establishing guidelines for further exploration in efficient architectural design for multivariate temporal data.