Comparative Empirical Forecasting

• Comparative empirical forecasting is the systematic comparison of forecasting methods under shared targets and protocols, emphasizing reproducible out-of-sample evaluation.
• It leverages various validation techniques, such as rolling-origin and blocked cross-validation, to accurately measure performance across different domains.
• Empirical studies reveal regime-dependent trade-offs where model selection, calibration, and computational cost critically influence forecasting accuracy.

Comparative empirical forecasting is the systematic comparison of forecasting methods under a common experimental design: shared targets, horizons, training data, preprocessing, validation protocol, and scoring rules. In contemporary practice, it spans classification and regression, point and probabilistic forecasting, single-series and multi-task settings, and domains as different as weather, epidemic incidence, electricity demand, earthquakes, financial volatility, sparse retail demand, and annual hurricane counts. Its central concern is not only which model attains the lowest error, but also how model choice, training-window design, feature space, calibration, discrimination, and computational cost interact under reproducible out-of-sample evaluation (Rahman et al., 2024, Zhao et al., 2021, Brehmer et al., 2024).

1. Conceptual basis

Comparative empirical forecasting treats forecast evaluation as a model-selection problem over a candidate set $\mathcal{F}$. In one formulation, the selected model is

$$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$

where $\hat{R}_{\text{est},E}(f)$ is the estimated risk under an estimation method $E$. The comparison then asks two distinct questions: how often the oracle model $f^*$ is selected, and what performance loss is incurred when it is not. In a large-scale study of 3111 time series and 50 candidate regressors, selection accuracy was low for all estimators, while the overall forecasting performance loss associated with model selection ranged from 1.2% to 2.3%; sample size materially affected estimator rankings, and single holdout was consistently weak relative to blocked or prequential schemes (Cerqueira et al., 2021).

The comparative object is often broader than “which algorithm is best.” In state-level COVID-19 forecasting, the relative impacts of model selection, hyperparameter tuning, and training-window length were evaluated separately, and model selection emerged as the dominant factor in both performance improvement and performance variation across 7-day and 28-day horizons. The same study found a recurring risk–benefit trade-off: dimensions that deliver larger gains also incur larger penalties when poorly chosen. In multi-task forecast combination, local and global weighting schemes can be unified through a coupling penalty $\gamma$ that interpolates between task-specific and shared weights, and empirical results on Eurozone indicators showed that global combinations can surpass local combinations when task relatedness is exploited rather than ignored (Zhao et al., 2021, Thompson et al., 2022).

This suggests that comparative empirical forecasting is best understood as an experimental discipline over forecasting systems rather than as a simple leaderboard of model classes.

2. Targets, data, and task formulations

The targets used in comparative studies vary markedly in scale, horizon, and stochastic structure. A weather study based on NASA POWER used a single-station point for Dhaka, Bangladesh, with data from January 1, 2003 to January 1, 2023 and 16 meteorological variables, including T2M, T2MDEW, T2MWET, QV2M, RH2M, PRECTOT, and wind features. The targets were precipitation and average temperature. Although both are inherently continuous, the study evaluated them with classification metrics and confusion matrices, implying discretization into classes; however, the thresholds or bin edges were not reported. Features were scaled with MinMaxScaler, a Pearson-style correlation heatmap was used for feature analysis, and the train/test split was a random 85%/15%, not a time-aware split (Rahman et al., 2024).

Other comparative settings are explicitly time-series-native. Daily newly confirmed COVID-19 cases and daily deaths were forecast for California, New York, Texas, Minnesota, and Hawaii at 7-day and 28-day horizons using JHU-CSSE data; hourly next-step regional load was forecast from a three-year hourly dataset with weather covariates and lagged load features; annual hurricane counts in the North Atlantic were forecast one year ahead from 1981–2022 data; and weekly expected counts of M4+ earthquakes were forecast over 8,993 grid cells in Italy for 5,514 overlapping seven-day test periods. These examples collectively cover univariate and multivariate inputs, continuous and count outputs, and horizons from one step ahead to annual prediction (Zhao et al., 2021, Maity et al., 23 Jan 2026, Colombo et al., 2024, Brehmer et al., 2024).

A further distinction is between dense and sparse demand processes. In large-scale retail forecasting on the Favorita data, series were explicitly characterized by ADI and $CV^2$, with reported proportions of smooth, erratic, lumpy, and intermittent regimes and an average zero-sales percentage of 57.2%. In that setting, the comparative question shifted from “which forecaster is best overall” to “which forecaster should be routed to each series and horizon” (Zhang, 4 Jun 2025).

3. Comparative design and validation protocols

Comparative studies usually juxtapose heterogeneous model families. The Dhaka weather study evaluated Gradient Boosting, AdaBoost, ANN, Stacking Random Forest, Stacking Neural Network, and Stacking KNN; the COVID study compared SARIMA, SEIR-HCD, and Transformer-based ACTS; building-energy transfer-learning experiments compared vanilla Transformer, Informer, and PatchTST under six data-centric transfer strategies; sparse-demand routing compared ETS, Croston, Naïve, Moving Average, LightGBM regressor, DeepAR, and PatchTST under rule-based, LightGBM, and InceptionTime routers; and financial-network volatility work compared DST-ARCH, Spatial GARCH-X, circular ST-GARCH, STEGARCH, DCC-GARCH, and BEKK under alternative network matrices $W$ (Spencer et al., 2024, Zhang, 4 Jun 2025, Chrisko et al., 2 Mar 2026).

The decisive methodological issue is how the out-of-sample comparison is constructed. Time-aware protocols include rolling-origin or walk-forward evaluation, expanding-window one-year-ahead prediction for annual hurricane counts, fixed temporal splits such as 70%/15%/15% in short-term energy demand, and six rolling train/test windows of 2,000/500 hours in EU electricity price forecasting. By contrast, some studies rely on repeated holdouts or even random splits. The weather classification study used a random 85%/15% split with no walk-forward validation; a household-electricity comparison varied the number of holdout test instances rather than using blocked time-series cross-validation. These choices are consequential because they determine whether the experiment mimics deployment or permits leakage across time (Zhao et al., 2021, Colombo et al., 2024, Andrei et al., 2024, Bilal et al., 2022).

Comparative design also includes specialized setups for nested and multi-step models. For direct $h$-step-ahead nested regressions, one proposal tests the encompassing moment

$$E\!\left[e_{1,t+h|t}\big(e_{1,t+h|t}-e_{2,t+h|t}\big)\right]=0$$

and constructs a standard-normal statistic by replacing a full-sample mean with a split-sample mean, thereby bypassing the variance degeneracy that affects classical nested-model comparisons under recursive expanding windows (Pitarakis, 2023).

4. Metrics, scoring rules, and inference

Metric choice defines what is meant by “better.” In classification-oriented comparisons, the standard confusion-matrix quantities are $$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$0, $$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$1, $$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$2, and $$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$3, with

$$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$4

and

$$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$5

In regression and deterministic forecasting, the most common measures are

$$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$6

alongside MAPE, WAPE, MSE, and RMSLE. Comparative studies often report several of these simultaneously to avoid single-metric conclusions (Rahman et al., 2024, Zhao et al., 2021).

Probabilistic and count forecasting require strictly proper scoring rules and reliability diagnostics. For mean forecasts of count data, the quadratic score $$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$7 and the Poisson score $$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$8 are consistent scoring functions for expectations; Murphy diagrams decompose such scores across threshold-specific elementary scores; and isotonic-regression recalibration supports score decompositions into miscalibration (MCB), discrimination (DSC), and uncertainty (UNC). In probabilistic univariate forecasting, VAEneu replaces the reconstruction term in a conditional VAE with the Continuous Ranked Probability Score,

$$\hat{f} = \arg\min_{f \in \mathcal{F}} \hat{R}_{\text{est},E}(f),$$9

thereby optimizing sharpness and calibration jointly. In annual hurricane forecasting, pinball loss and PIT diagnostics complement MAE and directional accuracy (Brehmer et al., 2024, Koochali et al., 2024, Colombo et al., 2024).

Formal inference varies across studies. Some use paired $\hat{R}_{\text{est},E}(f)$0 tests and Wilcoxon signed-rank tests to show statistically significant differences between model families, as in the Polynomial Classifier versus RBFNN comparison; others use Friedman ranking tests, as in EU electricity price forecasting; and many use Diebold–Mariano tests for equal predictive accuracy, including spatiotemporal volatility forecasting and earthquake forecast comparison. Forecast-evaluation theory also shows that conditionally unbiased proxies for the target moment can preserve expected loss differences while reducing the variance of loss differentials, thereby increasing the power of comparative tests. For nested direct multi-step forecasts, the encompassing statistic described above yields an asymptotically standard normal test under recursive expanding windows (Nguyen et al., 2 May 2025, Holzmann et al., 2021, Pitarakis, 2023, Chrisko et al., 2 Mar 2026).

5. Empirical regularities across domains

Several comparative regularities recur. In the Dhaka weather study, AdaBoost was the best performer for both precipitation and temperature classification, with 92.51% accuracy for precipitation and 91.70% accuracy with F1 92% for temperature; the reported margins over other methods were small, and the feature analysis showed precipitation most strongly correlated with dew point, specific humidity, and relative humidity, while temperature correlated most strongly with wet-bulb temperature and specific humidity. In COVID forecasting, model selection generated the largest average improvement and the largest variation in performance across all four tasks, with accuracy improvement from model selection reported as 27.38% for 7-day confirmed cases and 27.96% for 28-day confirmed cases. In next-hour regional load forecasting, the proposed LSTM achieved test MAE 0.0164, RMSE 0.0193, MAPE 2.42%, with $\hat{R}_{\text{est},E}(f)$1 and $\hat{R}_{\text{est},E}(f)$2. In yearly hurricane forecasting, simpler models outperformed more complex ones: QR+PP achieved MAE 1.41 and APLF 1.19, while ARIMA+PP achieved the lowest APLF at 1.09. In EU electricity prices, PatchTST ranked first in 20 of 27 countries, TimesFM was consistently second-best, and ARIMA was weakest at the 96-hour horizon (Rahman et al., 2024, Zhao et al., 2021, Maity et al., 23 Jan 2026, Colombo et al., 2024, Andrei et al., 2024).

Other results show strong regime dependence. In a lightweight seasonal versus non-seasonal comparison, the Polynomial Classifier was more accurate and faster for non-seasonal gold and crude-oil series, whereas RBFNN was superior on seasonal weather and beer-production series, with paired $\hat{R}_{\text{est},E}(f)$3 tests and Wilcoxon tests confirming significance. In financial-network volatility forecasting, the Dynamic Spatiotemporal ARCH model achieved the lowest RMSFE and MAFE across network specifications, especially under Granger-filtered correlation networks, and did so at minimal computational cost. In sparse-demand forecasting, an InceptionTime router improved NWRMSLE by up to 11.8% over the best single-model benchmark and was 4.67x faster at inference than PatchTST. In earthquake forecasting, LRWA attained the best average Poisson score at 2.69, LM attained the best average quadratic score at 0.8407, and the analysis revealed substantial lack of calibration for several models (Nguyen et al., 2 May 2025, Chrisko et al., 2 Mar 2026, Zhang, 4 Jun 2025, Brehmer et al., 2024).

A further frontier is architectural substitution rather than family selection. In one-step-ahead solar-power forecasting, a QLSTM reduced test MAE from 0.0116 to 0.0058 and MSE from $\hat{R}_{\text{est},E}(f)$4 to $\hat{R}_{\text{est},E}(f)$5 relative to a classical LSTM under matched preprocessing and training, with a highly significant paired test on test loss. This suggests that comparative empirical forecasting now also encompasses controlled comparisons between classical and quantum-enhanced sequence models under identical data pipelines (Khan et al., 2023).

6. Limitations, threats to validity, and best practices

The literature repeatedly documents methodological weaknesses that can inflate apparent gains. The Dhaka weather comparison did not report target binning thresholds, cross-validation, walk-forward validation, missing-value treatment, or hyperparameters for ANN and stacking, and it used a random split that may risk temporal leakage. Household-electricity comparisons reported relative model rankings but did not use blocked rolling-origin evaluation or formal significance tests. Such omissions reduce reproducibility and blur the distinction between empirical performance and experimental artifact (Rahman et al., 2024, Bilal et al., 2022).

A second recurring issue is instability in the model-selection layer itself. In large-scale time-series model selection, exact selection accuracy was only 0.07–0.10 across estimators, even though overall loss remained relatively small; in COVID forecasting, model selection, training length, and hyperparameter tuning each improved performance when chosen well, but the same dimensions also produced the harshest penalties when chosen badly. These findings support several robust design rules: prefer blocked or rolling-origin evaluation; compare multiple model families rather than tuning a single favorite; choose training windows that reflect the operative regime; and report improvement together with sensitivity or variation, not just mean score (Cerqueira et al., 2021, Zhao et al., 2021).

Best practice increasingly extends beyond point accuracy. Reliability curves via isotonic regression, consistency bands, Murphy diagrams, and MCB–DSC decompositions make calibration failures visible even when average scores are strong. Transfer-learning studies show that negative transfer can arise from feature-space mismatch, especially weather-feature incompatibility in building-energy forecasting. Proxy-based evaluation shows that, when unbiased proxies for the target moment are available, comparative tests can gain power without changing the ranking in expectation. Taken together, these results suggest that rigorous comparative empirical forecasting should combine time-aware validation, strong and simple baselines, explicit target definitions, multiple complementary metrics, calibration diagnostics, and statistically principled model comparison (Brehmer et al., 2024, Spencer et al., 2024, Holzmann et al., 2021).