Why Model Selection Fails in Time Series Forecasting: An Empirical Study of Instability Across Data Regimes

Abstract: Time series forecasting models often exhibit inconsistent performance across datasets with varying statistical and structural properties. Despite the wide range of available forecasting techniques, it remains unclear whether model selection can be reliably guided by simple data characteristics. This paper investigates why rule-based model selection fails in time series forecasting by analyzing the relationship between data-regime descriptors and model performance. A descriptor-based framework is introduced to characterize time series using measurable properties, including trend strength, seasonality, noise level, and temporal dependence. Based on these descriptors, a rule-based selection mechanism is formulated to map data regimes to candidate forecasting models. The approach is evaluated on multiple real-world datasets across different domains and forecasting horizons. The results show that rule-based model selection achieves low accuracy, with correct model identification occurring in only a small fraction of cases. Significant discrepancies are observed between recommended and empirically optimal models, particularly in noisy and mixed regimes. Further analysis reveals that model performance is highly sensitive to both dataset characteristics and forecasting horizon, resulting in substantial ranking instability across scenarios. These findings explain why simple heuristic rules fail to generalize and demonstrate that forecasting performance cannot be reliably predicted using static, descriptor-based approaches. This study provides empirical evidence that model selection in time series forecasting is inherently context-dependent and highlights the need for more adaptive, data-driven strategies.

• The paper shows that rule-based model selection using static regime descriptors fails to recover optimal forecasting models.
• It employs extensive evaluations across diverse datasets and forecasting horizons to demonstrate highly unstable, context-dependent performance.
• The study highlights that low-dimensional descriptors inadequately capture complex dynamics, urging adaptive, validation-driven selection strategies.

Model Selection Instability in Time Series Forecasting: An Empirical Analysis of Data Regime Dependence

Introduction

The paper "Why Model Selection Fails in Time Series Forecasting: An Empirical Study of Instability Across Data Regimes" (2605.01608) conducts a comprehensive empirical investigation into the limitations of rule-based model selection in univariate time series forecasting. The study formalizes a data-regime descriptor framework and provides a systematic dissection of the relationships between observable time series properties and model performance, with extensive validation across domains and forecasting horizons. The central finding is that mapping low-dimensional statistical descriptors to optimal model choice is fundamentally unreliable due to high context sensitivity and inherent ranking instability. This result has substantive practical and theoretical implications for automated model selection, the feasibility of descriptor-driven pipelines, and the design of adaptive model selection systems.

Descriptor-Based Framework and Model Selection Formulation

The framework initiates with a regime-based characterization of time series. Each series is mapped to a vector of descriptors: trend strength, seasonality, noise proportion, non-stationarity, intermittency, and the count of structural breaks. These descriptors are selected to distill primary sources of variation and commonly hypothesized to relate to model classes (e.g., ARIMA for low-noise, high-dependence, ETS for strong seasonality, Random Forests for high-noise or irregularity).

Model selection is operationalized via compatibility functions—either rule-based heuristics, linear or logistic scoring, or proximity to “prototypical” regimes. Deterministic rules are formulated closely aligned with practitioner heuristics. The framework also includes an interpretability measure through local gradient analysis of compatibility functions, allowing quantification of descriptor influence.

Experimental Design and Datasets

Benchmark-scale evaluation is conducted on four real-world datasets (Energy, Tourism, Tetuan, PGCB) representing utility, financial, and industrial scenarios, each with marked heterogeneity in autocorrelation, noise, trend, and horizon targets. Preprocessing involves imputation and normalization; rolling-origin schemes ensure temporal integrity. Models span statistical (ARIMA, ETS), machine learning (Random Forest, XGBoost), and deep neural (LSTM, GRU) families, with standard cross-validation for hyperparameter tuning.

Baseline comparators include:

• Best Single Model: fixed across cases,
• Grid Search: exhaustive search across families and parameters,
• AutoML: (AutoGluon) integrating selection and tuning.

Metric evaluation encompasses RMSE and MAE, as well as computational cost and performance variance.

Empirical Findings

Distribution and Overlap of Model Performance

A fundamental empirical result is the considerable overlap in normalized RMSE distributions across all candidate models and datasets. No forecasting method demonstrates consistent superiority.

Figure 1: Distribution of relative RMSE across models. RMSEs are normalized to the best-performing model per dataset-horizon pair, evidencing substantial performance overlap and no clear winner.

Pairwise win-rate comparisons further corroborate the absence of universal dominance—most pairwise probabilities are close to 0.5, signaling that performance superiority is context-dependent and not tied to model structure alone.

Figure 2: Pairwise win rate matrix, illustrating the lack of consistent dominance and the highly contingent nature of forecasting performance across data/horizon regimes.

Rule-Based Selection Accuracy and Instability

The core experiment directly tests the accuracy of the rule-based regime-model map. Selection accuracy is consistently low (approx. 12.5%), with rule-based heuristics frequently failing to recover the empirically best model, particularly in high-noise and mixed regimes.

Figure 3: Discrepancy plot between rule-based and optimal empirical model selections per dataset-horizon scenario; frequent and significant mismatches are observed.

Selection accuracy does not improve when conditioned on noise regime, trend magnitude, or autocorrelation.

Figure 4: Regime-wise selection accuracy—a small fraction of correct predictions, with rule-based heuristics failing entirely in both high- and low-noise scenarios.

Ranking Variability and Horizon Sensitivity

Rank variance analysis confirms that model performance is extremely unstable across dataset-horizon pairs; even models with close average RMSEs exhibit whiplash changes in rank.

Figure 5: Model rank instability (variance) as a function of dataset-horizon pairings, highlighting the volatility of model ordering.

Further, model rankings are highly horizon-dependent: models that excel in short-term prediction may falter on longer horizons and vice versa. Linear/statistical models improve in longer-horizon settings in some cases, contrary to standard narrative assumptions.

Figure 6: Horizon-dependent mean relative RMSE, evidencing that model rankings are not invariant but change with the prediction horizon.

Statistical Significance and Baseline Comparison

Statistical testing (Friedman test, $p = 0.3476$) affirms that performance differences among canonical models are not statistically significant across evaluated cases, reinforcing the finding of weak performance separation.

Practical and Theoretical Implications

Implications for Automated Forecasting Pipelines

The results expose the central limitation of rule-based or static model selection protocols: low-dimensional descriptors such as autocorrelation, noise level, or seasonality, whether expert-designed or machine-computed, do not reliably encode the complexity needed for robust model selection. This undermines the use of “fast” prescreen filters in AutoML pipelines for time series—empirical validation, not heuristic mapping, remains indispensable.

These findings also clarify that neither domain knowledge nor standard statistical diagnostics are sufficient—hierarchical or meta-learning approaches that adaptively learn deeper, non-linear relationships may be required.

Instability as a Challenge for Meta-Learning

Significant instability in both model performance and ranking, especially across horizons, limits the transferability of regime–model associations. Descriptor-extraction quality becomes unreliable for short, irregular, or extreme series, further decohering static mappings.

For meta-learning and transfer learning protocols, this means the search for robust, generalizable meta-features for model selection is substantially more challenging than in supervised classification or regression.

On Interpretability and Empirical Validation

Gradient-based interpretability analytics, while useful for local explanation, are not a panacea; most model–descriptor relationships remain weak in the presence of high-variance or structurally complex data. The need to validate even before using “interpretable” rules is accentuated.

Future Research Directions

Suggested avenues include: integration of learned higher-order representations, direct handling of forecasting-horizon effects in selection policies, expansion to multivariate and probabilistic settings, and evaluating methods in online adaptation or streaming data settings to ascertain the regime–model mapping evolution under data drift and covariate shift.

Limitations

• Short time series and noisy/incomplete data can undermine descriptor computation and, hence, model selection accuracy.
• The framework is currently univariate; multivariate dependence patterns may necessitate entirely different descriptors and selection policies.
• Real-time deployment is hampered by computational demands of complex descriptors (e.g., spectral analysis, change-point detection).

Conclusion

The paper delivers a compelling empirical rejection of rule-based model selection grounded in static data-regime descriptors for time series forecasting. Performance instability is intrinsic to the problem: model superiority is not robustly recoverable from low-dimensional quantifications of temporal data structure. This work both clarifies the empirical limits of heuristics and motivates the development of adaptive, validation-driven, or learning-based selection strategies capable of addressing context sensitivity, horizon dependence, and rank instability in real-world forecasting pipelines. Static mapping approaches, even if interpretable, are not sufficient for context-robust forecasting model selection.