SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting

Abstract: The Transformer model has shown strong performance in multivariate time series forecasting by leveraging channel-wise self-attention. However, this approach lacks temporal constraints when computing temporal features and does not utilize cumulative historical series effectively.To address these limitations, we propose the Structured Channel-wise Transformer with Cumulative Historical state (SCFormer). SCFormer introduces temporal constraints to all linear transformations, including the query, key, and value matrices, as well as the fully connected layers within the Transformer. Additionally, SCFormer employs High-order Polynomial Projection Operators (HiPPO) to deal with cumulative historical time series, allowing the model to incorporate information beyond the look-back window during prediction. Extensive experiments on multiple real-world datasets demonstrate that SCFormer significantly outperforms mainstream baselines, highlighting its effectiveness in enhancing time series forecasting. The code is publicly available at https://github.com/ShiweiGuo1995/SCFormer

An Examination of SCFormer: Optimizing Multivariate Time Series Forecasting with Temporal Constraints and Historical Integration

In the domain of multivariate time series forecasting, the ongoing challenge has been to enhance the predictive accuracy of models in capturing dynamic and complex temporal patterns across multiple variables. The introduction of the structured channel-wise Transformer model, termed SCFormer, heralds an innovative approach to addressing prevalent issues in time series forecasting. Specifically, SCFormer combines two pivotal mechanisms: the imposition of temporal constraints within the Transformer architecture and the integration of cumulative historical states using High-order Polynomial Projection Operators (HiPPO).

Methodological Insights

SCFormer builds upon the established architecture of Transformers, recognized for their efficacy in sequence modeling, by introducing tailored modifications for time series forecasting. Traditional channel-wise Transformers, while adept at capturing inter-variable relationships, suffer from two notable limitations: the lack of temporal constraints in linear transformations and the inefficacy in leveraging long-term historical data beyond predefined look-back windows.

1. Temporal Constraints via Structured Linear Operations

SCFormer innovates by enforcing temporal constraints through the application of structured matrices, specifically triangular matrices, to the linear transformations within the Transformer. This structural modification ensures that in the feature computation process, future values do not exert any undue influence on the current element, which aligns with the intrinsic sequential progression of time series data. For operational efficiency, SCFormer also employs 1D convolutional layers as an alternative structuring strategy, offering identical temporal respectfulness with different parameter-sharing benefits, thereby further enhancing the model’s predictive reliability.

2. Integration of Cumulative Historical States

To address the insufficiency in historical data utilization, SCFormer employs HiPPO techniques. HiPPO provides a rigorous framework for embedding cumulative historical data into fixed-dimensional spaces, maintaining the integrity of long-term memory in the forecasting process. Unlike traditional models restricted by the finite length of look-back windows, this approach captures in-depth historical trends and dynamics, laying a robust groundwork for improved forecast accuracy.

Empirical Examination and Analytical Outcomes

The efficacy of SCFormer was tested against multiple real-world datasets including ETT, PEMS, Solar Energy, ECL, among others. SCFormer demonstrated significant improvements over baseline models. For instance, the SCFormer-triangular variation achieved substantial reductions in mean squared error (MSE) compared to existing state-of-the-art models like iTransformer and PatchTST, underscoring its enhanced ability to decode temporal structures and historical sequences. For example, across the Exchange dataset, SCFormer-triangular achieved an average MSE improvement of 16.9% over leading channel-wise models, highlighting its superior forecast precision.

Additionally, extensive ablation studies confirmed the isolated contribution of each methodological component in SCFormer. The introduction of temporal constraints led to marked improvements in feature stability and prediction accuracy. Moreover, the integration of cumulative historical states via HiPPO proved instrumental in enriching the model’s temporal breadth, as demonstrated by experiments showing further performance enhancements with elongated look-back periods.

Implications and Future Prospects

SCFormer advances the methodological frontier in time series forecasting by embedding historical comprehensiveness and temporal fidelity within Transformer architectures. Practically, this development is poised to impart significant advancements in fields necessitating precise temporal predictions, such as financial analytics, energy consumption forecasting, and logistics management.

Theoretically, SCFormer bridges critical gaps between historical data utility and model structure in time series analysis, potentially guiding future explorations in integrative approaches to sequence modeling. Future developments may explore cross-domain applications and further fine-tune SCFormer’s architecture to harness even broader patterns in larger, more complex datasets.

In conclusion, SCFormer represents a thoughtful synthesis of architectural innovation and historical insight, setting a new benchmark in multivariate time series forecasting efficacy that could inform subsequent AI research trajectories.