Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting (2402.05956v5)

Published 4 Feb 2024 in cs.LG

Abstract: Transformers for time series forecasting mainly model time series from limited or fixed scales, making it challenging to capture different characteristics spanning various scales. We propose Pathformer, a multi-scale Transformer with adaptive pathways. It integrates both temporal resolution and temporal distance for multi-scale modeling. Multi-scale division divides the time series into different temporal resolutions using patches of various sizes. Based on the division of each scale, dual attention is performed over these patches to capture global correlations and local details as temporal dependencies. We further enrich the multi-scale Transformer with adaptive pathways, which adaptively adjust the multi-scale modeling process based on the varying temporal dynamics of the input, improving the accuracy and generalization of Pathformer. Extensive experiments on eleven real-world datasets demonstrate that Pathformer not only achieves state-of-the-art performance by surpassing all current models but also exhibits stronger generalization abilities under various transfer scenarios. The code is made available at https://github.com/decisionintelligence/pathformer.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  2. Unsupervised time series outlier detection with diversity-driven convolutional ensembles. Proceedings of the VLDB Endowment, 2022.
  3. NHITS: neural hierarchical interpolation for time series forecasting. In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
  4. Bayesian forecasting for financial risk management, pre and post the global financial crisis. Journal of Forecasting, 2012.
  5. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting. In International Conference on Knowledge Discovery & Data Mining (KDD), 2022.
  6. Weakly guided adaptation for robust time series forecasting. Proceedings of the VLDB Endowment, 2024.
  7. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, 2014.
  8. Graph attention recurrent neural networks for correlated time series forecasting. In International Conference on Knowledge Discovery & Data Mining (KDD), 2019.
  9. EnhanceNet: Plugin neural networks for enhancing correlated time series forecasting. In IEEE International Conference on Data Engineering (ICDE), 2021.
  10. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting. In International Joint Conference on Artificial Intelligence (IJCAI), 2022a.
  11. Towards spatio-temporal aware traffic time series forecasting. In IEEE International Conference on Data Engineering (ICDE), 2022b.
  12. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 1965.
  13. Long-term forecasting with tide: Time-series dense encoder. arXiv, 2023.
  14. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
  15. Multi-scale and hidden resolution time series models. 2006.
  16. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  17. Automatic time series forecasting: the forecast package for r. Journal of statistical software, 2008.
  18. A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection. arXiv, 2023a.
  19. Time-LLM: Time series forecasting by reprogramming large language models. arXiv, 2023b.
  20. Anomaly detection in time series with robust variational quasi-recurrent autoencoders. In IEEE International Conference on Data Engineering (ICDE), 2022a.
  21. Robust and explainable autoencoders for unsupervised time series outlier detection. In IEEE International Conference on Data Engineering (ICDE), 2022b.
  22. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations (ICLR), 2022.
  23. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR), 2015.
  24. Do simpler statistical methods perform better in multivariate long sequence time-series forecasting? In International Conference on Information & Knowledge Management (CIKM), 2022a.
  25. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
  26. Mvitv2: Improved multiscale vision transformers for classification and detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
  27. Scinet: Time series modeling and forecasting with sample convolution and interaction. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
  28. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations (ICLR), 2022b.
  29. Enabling time-dependent uncertain eco-weights for road networks. In Proceedings of the ACM on Management of Data, 2014.
  30. A unified replay-based continuous learning framework for spatio-temporal prediction on streaming data. In IEEE International Conference on Data Engineering (ICDE), 2024.
  31. Michael Mozer. Induction of multiscale temporal structure. In Advances in Neural Information Processing Systems (NeurIPS), 1991.
  32. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations (ICLR), 2023.
  33. Magicscaler: Uncertainty-aware, predictive autoscaling. Proceedings of the VLDB Endowment, 2023.
  34. Anytime stochastic routing with hybrid learning. Proceedings of the VLDB Endowment, 2020.
  35. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
  36. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
  37. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In International Conference on Learning Representations (ICLR), 2023.
  38. Attention is all you need. Advances in neural information processing systems (NeurIPS), 2017.
  39. MICN: multi-scale local and global context modeling for long-term series forecasting. In International Conference on Learning Representations (ICLR), 2023.
  40. M2TR: multi-modal multi-scale transformers for deepfake detection. In International Conference on Multimedia Retrieval (ICMR), 2022a.
  41. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In International Conference on Computer Vision (ICCV), 2021.
  42. Crossformer: A versatile vision transformer hinging on cross-scale attention. In International Conference on Learning Representations (ICLR), 2022b.
  43. Transformers in time series: A survey. In International Joint Conference on Artificial Intelligence (IJCAI), 2023.
  44. A multi-horizon quantile recurrent forecaster. arXiv, 2017.
  45. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  46. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations (ICLR), 2023a.
  47. AutoCTS: Automated correlated time series forecasting. Proceedings of the VLDB Endowment, 2022.
  48. AutoCTS+: Joint neural architecture and hyperparameter search for correlated time series forecasting. Proceedings of the ACM on Management of Data, 2023b.
  49. Connecting the dots: Multivariate time series forecasting with graph neural networks. In International Conference on Knowledge Discovery & Data Mining (KDD), 2020.
  50. Are transformers effective for time series forecasting? In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
  51. Multiple time series forecasting with dynamic graph modeling. Proceedings of the VLDB Endowment, 2024.
  52. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Association for the Advancement of Artificial Intelligence (AAAI), 2021.
  53. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning (ICML), 2022.
  54. One fits all: Power general time series analysis by pretrained lm. arXiv, 2023.
  55. Energy forecasting with robust, flexible, and explainable machine learning algorithms. AI Magazine, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Peng Chen (324 papers)
  2. Yingying Zhang (80 papers)
  3. Yunyao Cheng (5 papers)
  4. Yang Shu (17 papers)
  5. Yihang Wang (22 papers)
  6. Qingsong Wen (139 papers)
  7. Bin Yang (320 papers)
  8. Chenjuan Guo (48 papers)
Citations (19)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube