Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecasting (2401.09261v2)

Published 17 Jan 2024 in cs.LG

Abstract: Demystifying interactions between temporal patterns of different scales is fundamental to precise long-range time series forecasting. However, previous works lack the ability to model high-order interactions. To promote more comprehensive pattern interaction modeling for long-range time series forecasting, we propose a Multi-Scale Hypergraph Transformer (MSHyper) framework. Specifically, a multi-scale hypergraph is introduced to provide foundations for modeling high-order pattern interactions. Then by treating hyperedges as nodes, we also build a hyperedge graph to enhance hypergraph modeling. In addition, a tri-stage message passing mechanism is introduced to aggregate pattern information and learn the interaction strength between temporal patterns of different scales. Extensive experiments on five real-world datasets demonstrate that MSHyper achieves state-of-the-art (SOTA) performance across various settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. ETC: Encoding long and structured inputs in transformers. In Proceedings of the International Conference on Empirical Methods in Natural Language Processing, pages 268–284, 2020.
  2. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
  3. Hypergraph convolution and hypergraph attention. Pattern Recognition, 110(1):1–8, 2021.
  4. Some recent advances in forecasting and control. Journal of the Royal Statistical Society Series C, 17(2):91–109, 1968.
  5. Time-aware multi-scale RNNs for time series modeling. In Proceedings of the International Joint Conference on Artifical Intelligence, pages 2285–2291, 2021.
  6. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 146–156, 2022.
  7. Multi-scale adaptive graph neural network for multivariate time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 35(10):10748–10761, 2023.
  8. Long sequence time-series forecasting with deep learning: A survey. Information Fusion, 97(1):1–36, 2023.
  9. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting. In Proceedings of the International Joint Conference on Artifical Intelligence, pages 1–10, 2022.
  10. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations, 2021.
  11. Hypergraph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3558–3565, 2019.
  12. Star-transformer. In Proceedings of the International Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1315–1325, 2019.
  13. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations, 2020.
  14. Modeling long-and short-term temporal patterns with deep neural networks. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 95–104, 2018.
  15. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems, pages 1–11, 2019.
  16. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In Proceedings of the International Conference on Learning Representations, 2021.
  17. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181–1191, 2020.
  18. Stock selection via spatiotemporal hypergraph attention network: A learning to rank approach. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 497–504, 2021.
  19. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In Proceedings of the International Conference on Learning Representations, 2022.
  20. Multi-scale transformer language models. arXiv preprint arXiv:2005.00581, 2020.
  21. Forecasting at scale. The American Statistician, 72(1):37–45, 2018.
  22. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568–578, 2021.
  23. RobustPeriod: Robust time-frequency mining for multiple periodicity detection. In Proceedings of the International Conference on Management of Data, pages 2328–2337, 2021.
  24. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022.
  25. Graph WaveNet for deep spatial-temporal graph modeling. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1907–1913, 2019.
  26. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, pages 22419–22430, 2021.
  27. Hypergraph contrastive collaborative filtering. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 70–79, 2022.
  28. GroupNet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498–6507, 2022.
  29. HyperGCN: A new method for training graph convolutional networks on hypergraphs. Advances in Neural Information Processing Systems, pages 1–12, 2019.
  30. Learning multi-granular hypergraphs for video-based person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2899–2908, 2020.
  31. LBSN2Vec++: Heterogeneous hypergraph embedding for location-based social networks. IEEE Transactions on Knowledge and Data Engineering, 34(4):1843–1855, 2020.
  32. Multi-behavior hypergraph-enhanced transformer for sequential recommendation. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2263–2274, 2022.
  33. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In Proceedings of the International Conference on Learning Representations, 2023.
  34. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3008, 2021.
  35. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11106–11115, 2021.
  36. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proceedings of the International Conference on Machine Learning, pages 27268–27286, 2022.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now