TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods (2403.20150v3)
Abstract: Time series are generated in diverse domains such as economic, traffic, health, and energy, where forecasting of future values has numerous important applications. Not surprisingly, many forecasting methods are being proposed. To ensure progress, it is essential to be able to study and compare such methods empirically in a comprehensive and reliable manner. To achieve this, we propose TFB, an automated benchmark for Time Series Forecasting (TSF) methods. TFB advances the state-of-the-art by addressing shortcomings related to datasets, comparison methods, and evaluation pipelines: 1) insufficient coverage of data domains, 2) stereotype bias against traditional methods, and 3) inconsistent and inflexible pipelines. To achieve better domain coverage, we include datasets from 10 different domains: traffic, electricity, energy, the environment, nature, economic, stock markets, banking, health, and the web. We also provide a time series characterization to ensure that the selected datasets are comprehensive. To remove biases against some methods, we include a diverse range of methods, including statistical learning, machine learning, and deep learning methods, and we also support a variety of evaluation strategies and metrics to ensure a more comprehensive evaluations of different methods. To support the integration of different methods into the benchmark and enable fair comparisons, TFB features a flexible and scalable pipeline that eliminates biases. Next, we employ TFB to perform a thorough evaluation of 21 Univariate Time Series Forecasting (UTSF) methods on 8,068 univariate time series and 14 Multivariate Time Series Forecasting (MTSF) methods on 25 datasets. The benchmark code and data are available at https://github.com/decisionintelligence/TFB.
- Energy time series forecasting based on pattern sequence similarity. IEEE Transactions on Knowledge and Data Engineering 23, 8 (2010), 1230–1243.
- An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018).
- Libra: A benchmark for time series forecasting methods. In Proceedings of the ACM/SPEC International Conference on Performance Engineering. 189–200.
- George EP Box and David A Pierce. 1970. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. Journal of the American statistical Association 65, 332 (1970), 1509–1526.
- Leo Breiman. 2001. Random forests. Machine learning 45 (2001), 5–32.
- Rasmus Bro and Age K Smilde. 2014. Principal component analysis. Analytical methods 6, 9 (2014), 2812–2831.
- Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles. Proc. VLDB Endow. 15, 3 (2022), 611–623.
- Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 6989–6997.
- Pathformer: Multi-scale transformers with Adaptive Pathways for Time Series Forecasting. arXiv preprint arXiv:2402.05956 (2024).
- Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In SIGKDD. 785–794.
- Weakly guided adaptation for robust time series forecasting. Proceedings of the VLDB Endowment 17, 4 (2023), 766–779.
- Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting. In IJCAI. 1994–2001.
- Towards Spatio-Temporal Aware Traffic Time Series Forecasting. In ICDE. 2900–2913.
- EnhanceNet: Plugin Neural Networks for Enhancing Correlated Time Series Forecasting.. In ICDE. 1739–1750.
- Graph Attention Recurrent Neural Networks for Correlated Time Series Forecasting.. In MileTS19@KDD.
- STL: A seasonal-trend decomposition. J. Off. Stat 6, 1 (1990), 3–73.
- Pearson correlation coefficient. Noise reduction in speech processing (2009), 1–4.
- Long-term Forecasting with TiDE: Time-series Dense Encoder. arXiv preprint arXiv:2304.08424 (2023).
- Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
- Efficient tests for an autoregressive unit root.
- Cristian Challú Kin G. Olivares Federico Garza, Max Mergenthaler Canseco. 2022. StatsForecast: Lightning fast forecasting with statistical and econometric models. PyCon Salt Lake City, Utah, US 2022. https://github.com/Nixtla/statsforecast
- Temporal relational ranking for stock prediction. ACM Transactions on Information Systems (TOIS) 37, 2 (2019), 1–30.
- A machine learning approach to univariate time series forecasting of quarterly earnings. Review of Quantitative Finance and Accounting 55 (2020), 1163–1179.
- Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189–1232.
- Probabilistic forecasting with spline quantile function RNNs. In The 22nd international conference on artificial intelligence and statistics. PMLR, 1901–1910.
- Monash time series forecasting archive. arXiv preprint arXiv:2105.06643 (2021).
- Towards Total Traffic Awareness. SIGMOD Record 43, 3 (2014), 18–23.
- Ecomark 2.0: empowering eco-routing with vehicular environmental models and actual vehicle fuel consumption data. GeoInformatica 19 (2015), 567–599.
- Context-aware, preference-based vehicle routing. VLDB J. 29, 5 (2020), 1149–1170.
- Andrew C Harvey. 1990. Forecasting, structural time series models and the Kalman filter. (1990).
- Darts: User-friendly modern machine learning for time series. The Journal of Machine Learning Research 23, 1 (2022), 5442–5447.
- Stochastic weight completion for road networks using graph convolutional networks. In 2019 IEEE 35th international conference on data engineering (ICDE). IEEE, 1274–1285.
- Risk-aware path selection with time-varying, uncertain travel costs: a time series approach. The VLDB Journal 27 (2018), 179–200.
- Enabling time-dependent uncertain eco-weights for road networks. GeoInformatica 21, 1 (2017), 57–88.
- Dgraph: A large-scale financial dataset for graph anomaly detection. Advances in Neural Information Processing Systems 35 (2022), 22765–22777.
- Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media.
- Rob J Hyndman and Anne B Koehler. 2006. Another look at measures of forecast accuracy. International journal of forecasting 22, 4 (2006), 679–688.
- Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30 (2017).
- Benjamin Kedem and Konstantinos Fokianos. 2005. Regression models for time series analysis. John Wiley & Sons.
- Anomaly detection in time series with robust variational quasi-recurrent autoencoders. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 1342–1354.
- Robust and Explainable Autoencoders for Unsupervised Time Series Outlier Detection. In ICDE. 3038–3050.
- Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations.
- Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval. 95–104.
- Big healthcare data analytics: Challenges and applications. Handbook of large-scale distributed computing in smart healthcare (2017), 11–41.
- Doyup Lee. 2017. Anomaly detection in multivariate non-stationary time series for automatic DBMS diagnosis. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 412–419.
- Generative time series forecasting with diffusion, denoise, and disentanglement. Advances in Neural Information Processing Systems 35 (2022), 23009–23022.
- Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926 (2017).
- BasicTS: An Open Source Fair Multivariate Time Series Prediction Benchmark. In International Symposium on Benchmarking, Measuring and Optimization. Springer, 87–101.
- Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems 35 (2022), 9881–9893.
- Feature selection using principal feature analysis. In Proceedings of the 15th ACM international conference on Multimedia. 301–304.
- catch22: CAnonical Time-series CHaracteristics: Selected through highly comparative time-series analysis. Data Mining and Knowledge Discovery 33, 6 (2019), 1821–1852.
- Spyros Makridakis and Michele Hibon. 2000. The M3-Competition: results, conclusions and implications. International journal of forecasting 16, 4 (2000), 451–476.
- The M4 Competition: Results, findings, conclusion and way forward. International Journal of Forecasting 34, 4 (2018), 802–808.
- Michael W McCracken and Serena Ng. 2016. FRED-MD: A monthly database for macroeconomic research. Journal of Business & Economic Statistics 34, 4 (2016), 574–589.
- A random forest method for real-time price forecasting in New York electricity market. In 2014 IEEE PES General Meeting| Conference & Exposition. IEEE, 1–5.
- A Unified Replay-based Continuous Learning Framework for Spatio-Temporal Prediction on Streaming Data. ICDE (2024).
- Generative semi-supervised learning for multivariate time series imputation. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 8983–8991.
- Rob Muspratt and Musa Mammadov. 2023. Anomaly Detection with Sub-Extreme Values: Health Provider Billing. Data Science and Engineering (2023), 1–11.
- Guy P Nason. 2006. Stationary and non-stationary time series. (2006).
- A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 (2022).
- Kevin E O’Grady. 1982. Measures of explained variance: Cautions and limitations. Psychological Bulletin 92, 3 (1982), 766.
- N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 (2019).
- Magicscaler: Uncertainty-aware, predictive autoscaling. Proceedings of the VLDB Endowment 16, 12 (2023), 3808–3821.
- Transfer graph neural networks for pandemic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4838–4845.
- Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
- Anytime Stochastic Routing with Hybrid Learning. Proc. VLDB Endow. 13, 9 (2020), 1555–1567.
- Global transpiration data from sap flow measurements: the SAPFLUXNET database. Earth System Science Data Discussions 2020 (2020), 1–57.
- DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36, 3 (2020), 1181–1191.
- Financial time series forecasting with deep learning: A systematic literature review: 2005–2019. Applied soft computing 90 (2020), 106181.
- Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis. arXiv preprint arXiv:2310.06119 (2023).
- Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 914–921.
- A Suilin. 2017. kaggle-web-traffic. https://github.com/Arturus/kaggle-web-traffic
- A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition. Expert systems with applications 39, 8 (2012), 7067–7083.
- Monash University, UEA, UCR Time Series Regression Archive. 2020. URl: http://timeseriesregression. org ([n. d.]).
- Hiro Y Toda and Peter CB Phillips. 1994. Vector autoregression and causality: a theoretical overview and simulation study. Econometric reviews 13, 2 (1994), 259–285.
- Representation learning for early sepsis prediction. In 2019 Computing in Cardiology (CinC). IEEE, 1–4.
- Artur Trindade. 2015. ElectricityLoadDiagrams20112014. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C58C86.
- MTTPRE: a multi-scale spatial-temporal model for travel time prediction. In Proceedings of the 30th International Conference on Advances in Geographic Information Systems. 1–10.
- Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations.
- Real-time Workload Pattern Analysis for Large-scale Cloud Databases. arXiv preprint arXiv:2307.02626 (2023).
- Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment. arXiv preprint arXiv:2401.02116 (2024).
- Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186 (2022).
- Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems 34 (2021), 22419–22430.
- AutoCTS: Automated correlated time series forecasting. Proceedings of the VLDB Endowment 15, 4 (2021), 971–983.
- AutoCTS+: Joint Neural Architecture and Hyperparameter Search for Correlated Time Series Forecasting. Proceedings of the ACM on Management of Data 1, 1 (2023), 1–26.
- TME: Tree-guided Multi-task Embedding Learning towards Semantic Venue Annotation. ACM Transactions on Information Systems 41, 4 (2023), 1–24.
- Unsupervised Path Representation Learning with Curriculum Negative Sampling. In IJCAI. 3286–3292.
- Lightpath: Lightweight and scalable path representation learning. In SIGKDD. 2999–3010.
- SimpleTS: An efficient and universal model selection framework for time series forecasting. Proceedings of the VLDB Endowment 16, 12 (2023), 3741–3753.
- CGF: A Category Guidance Based PM __\__{2.52.52.52.5} Sequence Forecasting Training Framework. IEEE Transactions on Knowledge and Data Engineering (2023).
- Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 11121–11128.
- Time series forecast of sales volume based on XGBoost. In Journal of Physics: Conference Series, Vol. 1873. IOP Publishing, 012067.
- Cautionary tales on air-quality improvement in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 473, 2205 (2017), 20170457.
- Yunhao Zhang and Junchi Yan. 2022. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations.
- Multiple time series forecasting with dynamic graph modeling. Proceedings of the VLDB Endowment 17, 4 (2023), 753–765.
- Outlier detection for streaming task assignment in crowdsourcing. In WWW. 1933–1943.
- Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 11106–11115.
- Film: Frequency improved legendre memory model for long-term time series forecasting. NeurIPS ([n. d.]).
- Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning. PMLR, 27268–27286.