2000 character limit reached
Self-Supervised Contrastive Learning for Long-term Forecasting
Published 3 Feb 2024 in cs.LG and cs.AI | (2402.02023v2)
Abstract: Long-term forecasting presents unique challenges due to the time and memory complexity of handling long sequences. Existing methods, which rely on sliding windows to process long sequences, struggle to effectively capture long-term variations that are partially caught within the short window (i.e., outer-window variations). In this paper, we introduce a novel approach that overcomes this limitation by employing contrastive learning and enhanced decomposition architecture, specifically designed to focus on long-term variations. To this end, our contrastive loss incorporates global autocorrelation held in the whole time series, which facilitates the construction of positive and negative pairs in a self-supervised manner. When combined with our decomposition networks, our contrastive learning significantly improves long-term forecasting performance. Extensive experiments demonstrate that our approach outperforms 14 baseline models in multiple experiments over nine long-term benchmarks, especially in challenging scenarios that require a significantly long output for forecasting. Source code is available at https://github.com/junwoopark92/Self-Supervised-Contrastive-Forecsating.
- A review on applications of ann and svm for building electrical energy consumption forecasting. Renewable and Sustainable Energy Reviews, 33:102–109, 2014.
- An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
- Nhits: Neural hierarchical interpolation for time series forecasting. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2023.
- The analysis of time series: an introduction with R. CRC press, 2019.
- A simple framework for contrastive learning of visual representations. In Proc. the International Conference on Machine Learning (ICML), pp. 1597–1607. PMLR, 2020.
- Stl: A seasonal-trend decomposition. J. Off. Stat, 6(1):3–73, 1990.
- Lars Dannecker. Energy time series forecasting: efficient and accurate forecasting of evolving time series from the energy domain. Springer, 2015.
- Yading: Fast clustering of large-scale time series data. Proceedings of the VLDB Endowment, 8(5):473–484, 2015.
- Unsupervised scalable representation learning for multivariate time series. 32, 2019.
- Large-scale unusual time series detection. In 2015 IEEE international conference on data mining workshop (ICDMW), pp. 1616–1619. IEEE, 2015.
- Efficient prediction of network traffic for real-time applications. Journal of Computer Networks and Communications, 2019, 2019.
- Supervised contrastive learning. 33:18661–18673, 2020.
- Reversible instance normalization for accurate time-series forecasting against distribution shift. In Proc. the International Conference on Learning Representations (ICLR), 2021.
- A comparison of artificial neural network and time series models for forecasting commodity prices. Neurocomputing, 10(2):169–181, 1996. ISSN 0925-2312. doi: https://doi.org/10.1016/0925-2312(95)00020-8. URL https://www.sciencedirect.com/science/article/pii/0925231295000208. Financial Applications, Part I.
- Shape and time distortion loss for training deep time series forecasting models. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), pp. 4191–4203, 2019.
- Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999–3007, 2017. URL http://openaccess.thecvf.com/content_ICCV_2017/papers/Lin_Focal_Loss_for_ICCV_2017_paper.pdf.
- Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In Proc. the International Conference on Learning Representations (ICLR), 2022a.
- Non-stationary transformers: Rethinking the stationarity in time series forecasting. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2022b.
- Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
- A time series is worth 64 words: Long-term forecasting with transformers. In Proc. the International Conference on Learning Representations (ICLR), 2023.
- Athanasios Papoulis and S Unnikrishna Pillai. Probability, random variables and stochastic processes. 2002.
- Deep imbalanced time-series forecasting via local discrepancy density. In Proc. of Machine Learning and Knowledge Discovery in Databases: Research Track (ECML/PKDD), pp. 139–155. Springer Nature Switzerland, 2023. doi: 10.1007/978-3-031-43424-2˙9. URL https://link.springer.com/content/pdf/10.1007/978-3-031-43424-2_9.
- Addressing big data time series: Mining trillions of time series subsequences under dynamic time warping. ACM Transactions on Knowledge Discovery from Data (TKDD), 7(3):1–31, 2013.
- Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing, 26(1):43–49, 1978.
- Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.00750, 2021.
- Deep learning for time series forecasting: a survey. Big Data, 9(1):3–21, 2021.
- Short-term traffic forecasting: Where we are and where we’re going. Transportation Research Part C: Emerging Technologies, 43:3–19, 2014.
- Micn: Multi-scale local and global context modeling for long-term series forecasting. In Proc. the International Conference on Learning Representations (ICLR), 2023.
- Learning latent seasonal-trend representations for time series forecasting. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2022.
- Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. 2022a.
- Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022b.
- Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021.
- Timesnet: Temporal 2d-variation modeling for general time series analysis. In Proc. the International Conference on Learning Representations (ICLR), 2023.
- Ts2vec: Towards universal representation of time series. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), volume 36, pp. 8980–8987, 2022.
- Are transformers effective for time series forecasting? In Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2023.
- Supervised contrastive regression. arXiv preprint arXiv:2210.01189, 2022.
- Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186, 2022.
- Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In Proc. the International Conference on Learning Representations (ICLR), 2023.
- Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2021.
- Film: Frequency improved legendre memory model for long-term time series forecasting. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2022a.
- Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. the International Conference on Machine Learning (ICML), 2022b.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.