Two-Step Meta-Learning for Forecast Ensembles

• The paper introduces a two-step meta-learning framework that extracts descriptive features and trains meta-models to weight heterogeneous forecasters, enhancing overall forecast accuracy.
• It combines traditional handcrafted feature extraction with deep representation learning, employing methods like XGBoost, random forests, and neural stacking.
• Benchmarking on datasets such as M4 demonstrates significant performance gains over individual forecasting models, highlighting ensemble adaptability across various series types.

Two-step meta-learning for time-series forecasting ensembles refers to a family of methodologies that jointly leverage descriptive time-series features and learned meta-models to automate both the selection and weighting of individual forecasters, producing adaptive ensemble predictions. By extracting high-dimensional meta-features summarizing the structural and statistical characteristics of each series, and training a meta-learner to map these features to optimal weights or rankings, these frameworks aim to maximize forecast accuracy across heterogeneous datasets and series types. This approach subsumes both feature-based ensemble weighting (as typified by FFORMA) and more recent representation-learning meta-fusion schemes (such as DeFORMA), and is empirically validated as state-of-the-art on competitive benchmarks such as M4.

1. Meta-Feature Extraction and Representation Architectures

Two-step meta-learning ensembles begin by extracting a high-dimensional feature vector for each univariate time series. Early frameworks, such as those in (Vaiciukynas et al., 2020), compute up to 390 handcrafted features per series using a range of R packages (e.g., catch22, tsfeatures, TSEntropies, fractaldim), applied to original, differenced, and log-transformed series. Features span autocorrelation, trend/seasonality strength, Hurst exponent, entropy, stationarity tests, and anomaly scores.

Later instantiations for nonseasonal series (such as (Cawood et al., 2021)) focus on lower-dimensional yet highly predictive descriptors, e.g., coefficient of variation, KPSS statistic, SVD entropy, and lag-1 autocorrelation, empirically selecting those with the highest rank correlation to base forecaster accuracy.

Instead of hand-engineered features, deep representation learning as in DeFORMA (Zyl, 2023), replaces stepwise extraction with a 1D ResNet-18 backbone, augmented with temporal differentiating and moving average "heads." This network consumes (truncated/padded) sequences and outputs a compact embedding $\mathbf{z}_t$, learned jointly end-to-end for the forecasting objective.

2. Base Model Pools and Forecast Generation

A defining property of these meta-learning ensembles is the heterogeneous pool of base forecasting models. (Vaiciukynas et al., 2020) evaluates 22 univariate forecasters, including classical statistical, decomposition-based, neural, and tree-based models (e.g., ARIMA, SARIMA, Theta, Prophet, ETS, BATS, TBATS, NNAR, XGBoost variants, thief-ARIMA). Practical ensemble deployments typically elicit predictions from as many as 20–30 candidate models. For the COVID-19 application (Cawood et al., 2021), the base models are Prophet (additive trend, no seasonal/holiday) and LSTM.

Each base method is independently fit or applied to a hold-out or sliding window from the training data, producing one- or multi-step-ahead forecasts. The out-of-sample accuracy (as measured by sMAPE, MAAPE, MASE, or OWA) is tracked for subsequent meta-learning.

3. Meta-Learner Design, Training, and Weight Mapping

The second stage constructs a meta-learner $\mathcal{F}$, mapping series-specific features (or embeddings) to weights or rankings over the base forecasting pool. Frameworks diverge in meta-model choice and weight computation:

• Feature-based forecast model averaging (FFORMA, (Cawood et al., 2022)):

A gradient-boosted tree (XGBoost) is trained on each training series' features $f_n$ and the vector of model losses $L_{nm}$, optimizing a custom objective:

$\bar{L}_n = \sum_{m=1}^M w_m(f_n)\,L_{nm}$

where $w_m(f_n)$ is the softmax of unnormalized scores $p_m(f_n)$ output by the tree (i.e., $w_m(f) = \exp(p_m(f)) / \sum_{j=1}^M \exp(p_j(f))$). Gradients and Hessians are supplied explicitly for the vector-valued output.

• Reciprocal-rank weighting (Two-step meta-learning ensemble, (Vaiciukynas et al., 2020)):

Two random forest regressors are employed: a "Ranker" predicts relative score $\hat s_i$ for each method, and a "Capper" recommends ensemble size $\hat{k}$. The ensemble comprises the top-$\mathcal{F}$0 ranked models, with weights $\mathcal{F}$1, $\mathcal{F}$2 being the predicted rank.

• Neural stacking/meta-learners (Zyl, 2023, Cawood et al., 2021):

Multilayer perceptrons (MLPs) take as input both base forecasts and either handpicked meta-features (Cawood et al., 2021) or deep embeddings (Zyl, 2023). The output is typically linearly combined via a softmax gating, yielding $\mathcal{F}$3 with $\mathcal{F}$4, $\mathcal{F}$5.

Meta-learners are hyperparameter-tuned via Bayesian optimization or early stopping, and are regularized using $\mathcal{F}$6 penalties or dropout to prevent overfitting.

4. End-to-End Algorithmic Structure

The general two-step meta-learning workflow can be abstracted in the following algorithmic template:

1. Offline / meta-learner training:
   • For each reference series:
     1. Partition into training/validation or sliding windows.
     2. Extract meta-features $\mathcal{F}$7 or compute embedding $\mathcal{F}$8.
     3. Fit each base model, record hold-out forecasts and accuracy $\mathcal{F}$9.
   • Train meta-learner (tree, RF, or MLP) to minimize feature-weighted predictive loss.
2. Online / ensemble prediction for new series:
   • Extract meta-features / generate embedding.
   • Obtain new base forecasts.
   • Use trained meta-learner to assign weights or ranks.
   • Produce the ensemble forecast as the weighted sum of base model predictions.

This structure applies to all identified variants, with architectural substitutions for representation module (statistical features vs. deep embeddings) and meta-learner head (tree, RF, or neural).

5. Empirical Performance and Comparative Evaluation

Comprehensive benchmark results are available for M4 and other large-scale forecasting competitions:

• FFORMA (Cawood et al., 2022):

Achieves state-of-the-art mean OWA (overall weighted average of sMAPE and MASE) of 0.788 "global" (monthly/quarterly/yearly), outperforming ES-RNN (0.825) and N-BEATS (0.799) on the same splits. Median-OWA improvements consistently favor FFORMA across all main subsets. In the daily subset, neural stacking (NN-STACK) marginally outperforms FFORMA when base accuracies are highly similar.

• Two-step weighted meta-ensemble (Vaiciukynas et al., 2020):

On 12,561 M4 micro-economic series and over 38,000 extended horizon splits, achieves weighted ensemble sMAPE of 9.21%, improving upon Theta (11.05%) and naive "Comb" (11.41%) by large margins, with consistent gain across frequency and horizon types.

• DeFORMA (Zyl, 2023):

On M4, achieves best mean OWA in Schulze-rank (aggregate across all frequencies), with mean-OWA 0.802 on Monthly, and wins overall rank 1. Ablation confirms that representation learning yields marginal gains over FFORMA for many frequencies, with more pronounced improvements on hard (e.g., daily, yearly) subsets.

6. Robustness, Applicability, and Limitations

The two-step meta-learning ensemble framework demonstrates pronounced robustness for diverse series and domains. FFORMA exhibits stable superiority (1–6% OWA gain) over leading single hybrid models (ES-RNN, N-BEATS) for both seasonal and nonseasonal data (Cawood et al., 2022). When base methods have indistinguishable accuracy, neural stacking can yield incremental benefits, as on the daily subset.

Framework requirements include a sufficiently large and diverse training set, a broad model pool (statistical + ML), careful cross-validation to avoid overfitting, and—if applicable—Bayesian optimization for hyperparameters. For representation-learning schemes, greater data volume is critical, and limitations remain for multivariate or hierarchical time-series, as well as exogenous or covariate-enriched settings (Zyl, 2023).

Identified limitations include computational cost (e.g., 390 features per series in (Vaiciukynas et al., 2020)), limitations for out-of-domain deployment, and potential misestimation of optimal ensemble size. The methodology does not currently support automatic learning of ensemble size and content in a single differentiable model, nor probabilistic interval aggregation. Some methods lack formal statistical significance testing, reporting only consistent cross-validated gains.

7. Extensions and Directions for Research

Recent and suggested extensions address transfer learning (pretrain/fine-tune), unified representation- and meta-learning architectures, expansion to multivariate and hierarchical data, and the adoption of advanced fusion heads (attention, gating networks). Replacement of hand-crafted features by deep sequence autoencoders and exploration of probabilistic dynamic weighting have been highlighted as promising areas. Continued empirical benchmarking and taxonomy development at the intersection of feature-based and representation-based meta-fusion remain open directions (Zyl, 2023).

---

Key citations: (Vaiciukynas et al., 2020, Cawood et al., 2022, Gastinger et al., 2021, Cawood et al., 2021, Zyl, 2023).