Transformer-XGB: Hybrid Forecasting Model

• The paper demonstrates that Transformer-XGB sequentially couples a Transformer network for feature extraction with an XGBoost regressor, enhancing predictive performance in regression and nowcasting tasks.
• The hybrid architecture leverages dense Transformer embeddings and gradient boosting to efficiently process both tabular and time-series data.
• The model achieves strong performance metrics in tasks such as concrete strength prediction and weather nowcasting, though with some trade-offs in predictive uncertainty.

Transformer-XGBoost (Transformer-XGB) refers to a class of hybrid models which sequentially combine a Transformer neural network as a feature extractor or forecaster with an XGBoost gradient-boosted tree regressor for tabular or time series prediction tasks. This architecture is distinguished by its capacity to exploit the attention-driven representation learning capabilities of Transformers and the structured, interpretable, and efficient decision-making mechanism of XGBoost. Notable recent implementations span high-performance concrete strength estimation in material science (Chakma et al., 25 Dec 2025) and adaptive nowcasting in meteorology (Sun, 2024).

1. Architecture and Data Flow

A Transformer-XGB model is characterized by a two-stage sequential pipeline. The first stage is a Transformer, which encodes the structured input (tabular features or sequential time-series) into a dense, high-dimensional representation that synthesizes information via self-attention. The second stage comprises an XGBoost regressor, which consumes this Transformer output and generates the final predictions.

Model Schematic Table

Stage	Inputs	Outputs
Transformer	Numeric features (tabular) or history window	Latent summary
XGBoost Regressor	Transformer-generated summary (plus context)	Prediction

In tabular regression tasks, each scalar feature $x_k$ is linearly embedded into $\mathbb{R}^d$ before attention-based encoding. In time-series nowcasting, the input is a window of $k$ multivariate observations, each embedded and combined with positional encodings before the stacked Transformer encoder layers. The output of the Transformer is aggregated (via mean-pooling, summation, or concatenation; specific method sometimes unspecified) into a summary vector, which is then supplied as input to XGBoost.

2. Mathematical and Computational Formulation

The Transformer-XGB model leverages the following canonical operations:

• Embedding (affine): $$x_k \rightarrow E(x_k) = x_k W^E + b^E\in\mathbb{R}^d$$
• Scaled Dot-Product Attention:

$$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

• Multi-Head:

$$\mathrm{MultiHead}(H) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O$$

• Feed-Forward Network (per position): $\mathrm{FFN}(h)=\mathrm{ReLU}(hW_1+b_1)W_2+b_2$
• XGBoost Regression: For each sample $i$ at boosting round $t$,

$$\hat{y}_i = \sum_{k=1}^t f_k(h^*),\quad \mathcal{L}^{(t)} = \sum_i \ell(y_i,\hat{y}_i^{(t-1)}+f_t(h^*)) + \Omega(f_t)$$

where $\Omega(f) = \gamma T + \frac{1}{2}\lambda\|w\|^2$.

No joint end-to-end loss over the entire pipeline is defined; training is strictly sequential: the Transformer is first trained via MSE to generate suitable embeddings or forecasts, after which XGBoost is fitted to the Transformer outputs (Chakma et al., 25 Dec 2025, Sun, 2024).

3. Hyperparameter Optimization and Training Protocol

Typical hyperparameters for the Transformer layer include:

• Encoder layers $L=2$
• Hidden size $d=128$
• Self-attention heads $h=4$
• Dropout $=0.1$
• Optimizer: Adam with learning rate $0.001$ (or $1\mathrm{e}{-4}$), early stopping if validation MSE stagnates (patience $\sim10$ epochs) (Chakma et al., 25 Dec 2025, Sun, 2024).

For XGBoost:

• $n\_estimators=100\text{-}500$
• Max depth $=4\text{-}10$
• Learning rate $=0.01\text{-}0.1$
• Subsample $=0.8$, colsample_bytree $=0.8$
• Regularization: $\mathrm{reg\_alpha}=1.0$, $\mathrm{reg\_lambda}=1.0$
• Random search and $10$-fold cross-validation are standard (Chakma et al., 25 Dec 2025).

The protocol involves data normalization, 80\%/20\% training/testing splits, and early-stopping on both Transformer and XGBoost modules.

4. Performance and Comparative Assessment

The Transformer-XGB hybrid models have demonstrated strong predictive metrics but distinct trade-offs in uncertainty:

• Material Science Regression (Chakma et al., 25 Dec 2025):
  • Test R²/Uncertainty:
  • Compressive Strength (CS): $R^2=0.981$, $U_\mathrm{norm}\approx24.3\%$
  • Flexural Strength (FS): $R^2=0.967$, $U_\mathrm{norm}\approx43.6\%$
  • Tensile Strength (TS): $R^2=0.978$, $U_\mathrm{norm}\approx48.7\%$
  • The Transformer-XGB model achieved competitive $R^2$ compared to ET-XGB and RF-LGBM baselines, but consistently had the highest uncertainty, indicating the lowest generalization reliability among all tested models.
• Time Series Nowcasting (Sun, 2024):
  • Weather Forecasting (100 epochs):
  • BTTF (Transformer–XGB): $\mathrm{RMSE}=2.3290$, $R^2=0.9407$
  • Pure Transformer: $\mathrm{RMSE}=2.5820$, $R^2=0.9264$
  • Pure XGBoost: $\mathrm{RMSE}=3.9678$, $R^2=0.8288$
  • The hybrid yielded up to $\sim10\%$ RMSE improvement over pure Transformer and $\sim40\%$ over pure XGBoost at 100 epochs.

Component ablations established that both modules were synergistic: removing either (e.g., using only Transformer or only XGBoost) degraded performance (Sun, 2024).

5. Interpretability and Feature Attribution

While proper SHAP-based analysis was not applied to the Transformer-XGB in (Chakma et al., 25 Dec 2025) (due to highest uncertainty and lower overall performance than ET-XGB or RF-LGBM), XGBoost’s inherent feature importance statistics were leveraged in (Sun, 2024). High F-scores among forecasted variables (e.g., Apparent Temperature, Humidity, WindSpeed) indicated critical drivers guiding the XGBoost "decision maker" for actionable interventions, such as resource allocation during adverse weather.

A plausible implication is that the pipeline’s interpretability predominantly derives from the XGBoost stage, given the Transformer’s intermediate dense representations are not inherently interpretable via SHAP or analogous mechanisms in published studies.

6. Domain-Specific Implementations

• Material Science (Concrete Strength Prediction, (Chakma et al., 25 Dec 2025)):
  • Input: 18 numeric mixture and specimen features (e.g., Cement, Silica Fume, aspect ratios)
  • Output: CS, FS, and TS predictions
  • Notable: No positional encoding (features unordered), Transformer acts as a contextual token-wise encoder.
• Nowcasting (Weather, (Sun, 2024)):
  • Input: Historical time-series windows with multivariate meteorological variables and positional encoding
  • Output: Multi-horizon forecasts and state correction terms
  • Notable: Explicit use of sinusoidal positional encoding, sequence-to-sequence prediction, and direct intervention in present state conditioned on predicted futures.

7. Applications, Limitations, and Prospects

Transformer-XGB models are applicable to any domain where structured (tabular) or sequential (time-series) features benefit from nonlinear feature extraction prior to tabular decision modeling—examples include concrete mix optimization (Chakma et al., 25 Dec 2025) and operational nowcasting (Sun, 2024).

Key limitations are: (a) the absence of end-to-end training, precluding co-adaptation of the two components, and (b) reduced generalization reliability compared to ensemble-tree baselines, as reflected in higher predictive uncertainty. Interpretability remains largely limited to the tree-based backend; Transformer-learned features have yet to be directly elucidated in published studies.

The hybrid offers a modular template for combining attention-based representation learning with interpretable tree-based reasoning in high-stakes engineering and operational settings. For domains prioritizing uncertainty quantification and transparency, further innovation in joint training and attention interpretability is anticipated.