Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ARM: Refining Multivariate Forecasting with Adaptive Temporal-Contextual Learning (2310.09488v1)

Published 14 Oct 2023 in stat.ML and cs.LG

Abstract: Long-term time series forecasting (LTSF) is important for various domains but is confronted by challenges in handling the complex temporal-contextual relationships. As multivariate input models underperforming some recent univariate counterparts, we posit that the issue lies in the inefficiency of existing multivariate LTSF Transformers to model series-wise relationships: the characteristic differences between series are often captured incorrectly. To address this, we introduce ARM: a multivariate temporal-contextual adaptive learning method, which is an enhanced architecture specifically designed for multivariate LTSF modelling. ARM employs Adaptive Univariate Effect Learning (AUEL), Random Dropping (RD) training strategy, and Multi-kernel Local Smoothing (MKLS), to better handle individual series temporal patterns and correctly learn inter-series dependencies. ARM demonstrates superior performance on multiple benchmarks without significantly increasing computational costs compared to vanilla Transformer, thereby advancing the state-of-the-art in LTSF. ARM is also generally applicable to other LTSF architecture beyond vanilla Transformer.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Conditional time series forecasting with convolutional neural networks. arXiv preprint arXiv:1703.04691, 2017.
  2. Some recent advances in forecasting and control. Journal of the Royal Statistical Society: Series C (Applied Statistics), 23(2):158–179, 1974.
  3. Li-Juan Cao and Francis Eng Hock Tay. Support vector machine with adaptive parameters in financial time series forecasting. IEEE Transactions on neural networks, 14(6):1506–1518, 2003.
  4. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019.
  5. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232–5270, 2022.
  6. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  7. Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5–10, 2004.
  8. Kyoung-jae Kim. Financial time series forecasting using support vector machines. Neurocomputing, 55(1-2):307–319, 2003.
  9. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
  10. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp.  95–104, 2018.
  11. Road traffic forecasting: Recent advances and new challenges. IEEE Intelligent Transportation Systems Magazine, 10(2):93–109, 2018.
  12. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
  13. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
  14. Using internet search data to forecast covid-19 trends: A systematic review. Analytics, 1(2):210–227, 2022. ISSN 2813-2203. doi: 10.3390/analytics1020014. URL https://www.mdpi.com/2813-2203/1/2/14.
  15. A survey on data mining techniques applied to electricity-related time series forecasting. Energies, 8(11):13162–13193, 2015.
  16. A time series is worth 64 words: Long-term forecasting with transformers, 2023.
  17. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437, 2019.
  18. An empirical overview of nonlinearity and overfitting in machine learning using covid-19 data. Chaos, Solitons & Fractals, 139:110055, 2020.
  19. A dual-stage attention-based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.02971, 2017.
  20. Deep state space models for time series forecasting. Advances in neural information processing systems, 31, 2018.
  21. Efficient content-based sparse attention with routing transformers, 2020. URL https://arxiv.org/pdf/2003.05997.pdf.
  22. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181–1191, 2020.
  23. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances in neural information processing systems, 32, 2019.
  24. Temporal pattern attention for multivariate time series forecasting. Machine Learning, 108:1421–1441, 2019.
  25. Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):75–85, 2020.
  26. Methodology for long-term prediction of time series. Neurocomputing, 70(16-18):2861–2869, 2007.
  27. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
  28. Wavenet: A generative model for raw audio. In Arxiv, 2016. URL https://arxiv.org/abs/1609.03499.
  29. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  30. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053, 2017.
  31. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, 2022.
  32. Lite transformer with long-short range attention. arXiv preprint arXiv:2004.11886, 2020.
  33. Accurate estimation of influenza epidemics using google search data via argo. Proceedings of the National Academy of Sciences, 112(47):14473–14478, 2015. doi: 10.1073/pnas.1515373112. URL https://www.pnas.org/doi/abs/10.1073/pnas.1515373112.
  34. Are transformers effective for time series forecasting?, 2022.
  35. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
  36. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.  11106–11115, 2021.
  37. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Jiecheng Lu (5 papers)
  2. Xu Han (270 papers)
  3. Shihao Yang (31 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now