Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TSPP: A Unified Benchmarking Tool for Time-series Forecasting (2312.17100v2)

Published 28 Dec 2023 in cs.LG

Abstract: While machine learning has witnessed significant advancements, the emphasis has largely been on data acquisition and model creation. However, achieving a comprehensive assessment of machine learning solutions in real-world settings necessitates standardization throughout the entire pipeline. This need is particularly acute in time series forecasting, where diverse settings impede meaningful comparisons between various methods. To bridge this gap, we propose a unified benchmarking framework that exposes the crucial modelling and machine learning decisions involved in developing time series forecasting models. This framework fosters seamless integration of models and datasets, aiding both practitioners and researchers in their development efforts. We benchmark recently proposed models within this framework, demonstrating that carefully implemented deep learning models with minimal effort can rival gradient-boosting decision trees requiring extensive feature engineering and expert knowledge.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. 41–48.
  2. George EP Box. 1970. GM Jenkins Time Series Analysis: Forecasting and Control. San Francisco, Holdan-Day (1970).
  3. Spectral temporal graph neural network for multivariate time-series forecasting. Advances in neural information processing systems 33 (2020), 17766–17778.
  4. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 (2022).
  5. Xgboost: extreme gradient boosting. R package version 0.4-2 1, 4 (2015), 1–4.
  6. Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction. International Journal of forecasting 27, 3 (2011), 635–660.
  7. Do we really need deep learning models for time series forecasting? arXiv preprint arXiv:2101.02118 (2021).
  8. Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189–1232.
  9. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence 115 (2022), 105151.
  10. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643 (2021).
  11. Google. 2017. Web Traffic Time Series Forecasting | Kaggle. https://www.kaggle.com/competitions/web-traffic-time-series-forecasting.
  12. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33 (2020), 21271–21284.
  13. Time Series Dataset Survey for Forecasting with Deep Learning. Forecasting 5, 1 (2023), 315–335. https://doi.org/10.3390/forecast5010017
  14. Time Series Dataset Survey for Forecasting with Deep Learning. Forecasting 5, 1 (2023), 315–335.
  15. Splice-2 comparative evaluation: Electricity pricing. UNSW (1999).
  16. Support vector machines. IEEE Intelligent Systems and their applications 13, 4 (1998), 18–28.
  17. Rob J Hyndman. 2020. A brief history of forecasting competitions. International Journal of Forecasting 36, 1 (2020), 7–14.
  18. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 (2018).
  19. Criteria for classifying forecasting methods. International Journal of Forecasting 36, 1 (2020), 167–177.
  20. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  21. An experimental review on deep learning architectures for time series forecasting. International Journal of Neural Systems 31, 03 (2021), 2130001.
  22. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926 (2017).
  23. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting 37, 4 (2021), 1748–1764.
  24. Do We Really Need Graph Neural Networks for Traffic Forecasting? arXiv preprint arXiv:2301.12603 (2023).
  25. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36, 1 (2020), 54–74.
  26. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38, 4 (2022), 1346–1364.
  27. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 (2019).
  28. PEMS. [n. d.]. Caltrans PeMS. https://pems.dot.ca.gov/.
  29. David Picard. 2021. Torch. manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. arXiv preprint arXiv:2109.08203 (2021).
  30. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36, 3 (2020), 1181–1191.
  31. Slawek Smyl. 2020. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting 36, 1 (2020), 75–85.
  32. Chenyu Tian and Wai Kin Chan. 2021. Spatial-temporal attention wavenet: A deep learning framework for traffic prediction considering spatial-temporal dependencies. IET Intelligent Transport Systems 15, 4 (2021), 549–561.
  33. UCI. [n. d.]. UCI Machine Learning Repository: ElectricityLoadDiagrams20112014 Data Set. https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014.
  34. UNC. 2020. M5 Forecasting - Accuracy | Kaggle. https://www.kaggle.com/competitions/m5-forecasting-accuracy/discussion/164599.
  35. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 753–763.
  36. Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arXiv:1906.00121 (2019).
  37. Temporal regularized matrix factorization for high-dimensional time series prediction. Advances in neural information processing systems 29 (2016).
  38. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 (2022).
  39. Gman: A graph multi-attention network for traffic prediction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 1234–1241.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now