CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting (2202.01575v3)

Abstract: Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers. Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to first learn disentangled feature representations, followed by a simple regression fine-tuning step -- we justify such a paradigm from a causal perspective. Following this principle, we propose a new time series representation learning framework for time series forecasting named CoST, which applies contrastive learning methods to learn disentangled seasonal-trend representations. CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations, respectively. Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by a considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks. It is also robust to various choices of backbone encoders, as well as downstream regressors. Code is available at https://github.com/salesforce/CoST.

An Overview of CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

The paper introduces CoST, a novel framework for time series forecasting which uses contrastive learning to derive disentangled seasonal-trend representations. Traditional deep learning methods for time series have leveraged architectures like LSTM, RNNs, TCNs, and Transformers in an end-to-end learning setup. However, this research suggests an alternative methodology wherein representations are learned in a disentangled manner before a regression phase, asserting potential improvements in performance from a causal perspective.

Core Contributions

1. Novel Representation Framework: CoST is developed to address the issues of overfitting and spurious correlations seen in traditional end-to-end forecasting models. By disentangling the seasonal and trend components of time series data, CoST aims to create robust representations that are invariant to noise and adaptable under distribution shifts.
2. Structural Time Series Motivation: Drawing from Bayesian Structural Time Series models, the authors propose that time series data can fundamentally be decomposed into trend, seasonal, and error components. The disentangling of these elements can theoretically lead to a more stable learning process, since independent mechanisms like seasonality and trend do not influence each other directly.
3. Contrastive Learning for Disentanglement: The framework utilizes contrastive learning methods in both the time and frequency domains to derive discriminative and invariant representations for the seasonal and trend components. This involves data augmentations to simulate interventions on error components, thus distinguishing this work from classical time series decomposition tasks.
4. Empirical Evaluation: CoST's efficacy is demonstrated through extensive benchmark testing across several datasets, including Electricity Transformer Temperature, Electricity, and Weather data. The proposed method outperforms state-of-the-art models, achieving a 21.3% improvement in MSE in the multivariate setting, establishing the superiority of representation learning over traditional end-to-end approaches for time series forecasting.

Implications and Future Prospects

The implications of CoST are multifaceted. Practically, it offers a more robust means of forecasting in scenarios where distributional shifts and noise are prevalent, which is commonplace in real-world time series data. Theoretically, it advances the methodology by integrating causal insights through structural representation learning, potentially inspiring a new line of research focusing on disentangled feature learning.

The authors speculate on the utility of CoST in broader AI applications, suggesting that the disentangled approach might be extrapolated to other domains where temporal patterns and causal dependencies play crucial roles. Future research avenues could explore the integration of CoST with other self-supervised learning paradigms, further augmenting its applicability across diverse contexts. Enhanced model interpretability and the potential for real-time deployment also warrant investigation.

In conclusion, CoST represents a significant contribution to the field of time series forecasting, leveraging concepts from contrastive learning and causal inference to enhance predictability and robustness. This opens new pathways for research into disentangled representation learning, promising advancements in both theoretical understanding and practical efficacy.