Model Selection for Time Series Forecasting: Empirical Analysis of Different Estimators (2104.00584v2)

Abstract: Evaluating predictive models is a crucial task in predictive analytics. This process is especially challenging with time series data where the observations show temporal dependencies. Several studies have analysed how different performance estimation methods compare with each other for approximating the true loss incurred by a given forecasting model. However, these studies do not address how the estimators behave for model selection: the ability to select the best solution among a set of alternatives. We address this issue and compare a set of estimation methods for model selection in time series forecasting tasks. We attempt to answer two main questions: (i) how often is the best possible model selected by the estimators; and (ii) what is the performance loss when it does not. We empirically found that the accuracy of the estimators for selecting the best solution is low, and the overall forecasting performance loss associated with the model selection process ranges from 1.2% to 2.3%. We also discovered that some factors, such as the sample size, are important in the relative performance of the estimators.

• The paper demonstrates that estimator accuracy in choosing the best forecasting model ranges from 7% to 10%, underscoring key challenges in model selection.
• It quantifies an average performance loss of 0.28% to 0.58%—with losses up to 1.70% for smaller samples—highlighting the need for robust estimation techniques.
• The study finds that simpler holdout methods offer faster computation compared to cross-validation, suggesting a trade-off between speed and estimation precision.

Model Selection for Time Series Forecasting: An Empirical Analysis

This paper presents a comprehensive empirical analysis of various performance estimation methods used for model selection in time series forecasting tasks. The core contribution resides in evaluating how effectively these methods can select the best forecasting model from a pool of alternatives — a critical step for ensuring accurate time series predictions.

Summary of Key Research Questions

The researchers formulated their inquiry around several pivotal questions:

1. Accuracy of Estimators: Determining the frequency with which performance estimators correctly identify the best predictive model.
2. Performance Loss: Quantifying the forecast performance deterioration when estimators fail to select the optimal model.
3. Impact of Sample Size: Investigating variations in estimator performance relative to the sample size of the time series.
4. Comparison of Averaging Approaches: Evaluating the effectiveness of combining estimator results using the average rank compared to the traditional average error approach.
5. Computational Efficiency: Assessing the computational costs associated with each estimation method.

Empirical Findings

The paper utilizes an extensive dataset comprising 3111 real-world univariate time series, incorporating 10 different estimation methods and testing 50 auto-regressive forecasting models. Below are the principal empirical findings:

• Estimator Accuracy: The analysis reveals that the accuracy of selecting the best predictive model as calibrated by these estimators ranges from 7% to 10%. Despite being substantially better than random selection, these figures indicate that model selection remains challenging for most estimators.
• Performance Loss Evaluation: The model selection process incurs an average performance loss between 0.28% and 0.58%, depending on the estimator. This loss is exacerbated by outliers, stressing the necessity of choosing robust estimation strategies under competitive forecasting scenarios.
• Effects of Time Series Sample Size: The analysis indicates a pronounced discrepancy in the performance of estimators based on the sample size. Estimators yield significantly better results when applied to time series with over 1000 observations. For time series with fewer than 1000 data points, performance declines notably, with losses ranging up to 1.70%.
• Averaging Approaches: The comparison between average rank and average error as methods for result synthesis demonstrated marginal differences, with the average rank approach offering minor performance benefits for smaller data sets.
• Execution Time: The evaluation of computational time confirmed that simple holdout techniques are faster than more complex cross-validation methods, with a notable trade-off between computational efficiency and estimator accuracy.

Implications and Future Directions

This paper underscores the challenging nature of model selection in time series forecasting and highlights the subtleties of selecting an appropriate estimation method. While the apparent performance differences among most estimators are slim, the considerable dispersion across datasets suggests a need for further refinement in handling outliers and variance in model predictions. Moreover, the interplay between model accuracy and computational efficiency necessitates a strategic balance, particularly in resource-constrained environments.

Future developments in artificial intelligence might focus on refining these estimation techniques, perhaps through adaptive or hybrid approaches that can dynamically adjust their criteria based on the characteristics of the time series data. Additionally, integrating computational considerations such as runtime or resource availability could enhance practical applicability, leading to more robust, efficient forecasting systems.

In summary, this paper provides a pivotal comparison of model selection strategies for time series forecasting, offering insights into performance trade-offs and computational considerations that could guide the development of more sophisticated forecasting systems in the future.