XStacking: Explanation-Augmented Stacking

• XStacking is an ensemble framework that augments traditional stacking by incorporating model-agnostic Shapley value attributions for transparent predictions.
• It transforms base model outputs into enriched meta-features, which drives superior discriminative performance and inherent interpretability.
• Empirical evaluations show that XStacking improves accuracy in various classification and regression tasks with only moderate computational overhead.

Explanation-Augmented Stacking (XStacking) is a framework for ensemble machine learning that unifies the predictive benefits of stacked generalization with inherent interpretability. Unlike traditional stacking—which trains a meta-learner on the predictions of multiple base models but often suffers from limited interpretability and, when base predictions are highly correlated, muted performance gains—XStacking introduces model-agnostic feature attributions (specifically Shapley values) as core meta-features. This transforms meta-learner inputs from opaque predictions to explicit explanations, enabling both higher discriminative capacity and built-in interpretability in the ensemble’s final output (Garouani et al., 23 Jul 2025).

1. Motivation and Objectives

Traditional stacking employs a two-level ensemble where base learners output raw predictions $\{\hat y_i^{(k)}\}_{k=1}^K$ that are consumed by a meta-learner $\mathcal{E}$. This approach tends to yield higher predictive accuracy, but also presents two fundamental drawbacks:

• If base model predictions are highly correlated or not sufficiently complementary, the resulting meta-feature space is weak, limiting achievable meta-learner improvements.
• The ensemble behaves as a compositional black box: it is typically unclear which original features influence the final prediction, making post-hoc explanation complex and approximate.

XStacking addresses these challenges by:

• Generating Shapley value vectors for each base model and each data point, capturing model-agnostic feature attributions.
• Providing both base predictions and these Shapley attributions as the meta-learner’s input.
• Constructing a meta-feature space that is both richer (often leading to higher accuracy) and explanation-aligned, so interpretability is intrinsic rather than bolted on post hoc (Garouani et al., 23 Jul 2025).

2. Algorithmic Structure and Mathematical Foundations

Let $\mathcal{D} = \{ (x_i, y_i) \}_{i=1}^m$ be a dataset, $x_i \in \mathbb{R}^d$, and $f_1, \dots, f_K$ denote $K$ first-stage base learners.

1. Training Base Learners: Each $f_k$ is fit to $\mathcal{D}$, typically using $K$-fold cross-validation for robustness.
2. Computation of Shapley Additive Explanations: For each base model $f_k$ and instance $x_i$, a Shapley vector $\phi(f_k, x_i) \in \mathbb{R}^d$ is computed using

$$\phi_j(f_k, x) = \sum_{S \subseteq N \setminus \{j\}} \frac{|S|! \cdot (|N| - |S| - 1)!}{|N|!} \big[ f_k(x_{S \cup \{j\}}) - f_k(x_S) \big], \;\; N = \{1, \dots, d\}.$$

1. Dynamic Feature Transformation: The meta-feature vector is constructed as

$$x'_i = [\; \phi(f_1,x_i) \| \phi(f_2,x_i) \| \dots \| \phi(f_K,x_i) \;] \in \mathbb{R}^{K d}$$

Optionally, the original features $x_i$ and/or base predictions $\hat y_i^{(k)}$ may be concatenated.

2. Training the Meta-Learner: The augmented dataset $\mathcal{D}' = \{ (x'_i, y_i) \}_{i=1}^m$ is used to train a second-stage model $g$ (e.g., SVM, XGBoost). For a new $x$:

$$\hat y = g\bigl( T(x, \{ \phi(f_k, x) \}_{k=1}^K ) \bigr )$$

where $T$ is the transformation defined above.

Pseudocode:

For k in 1..K: Train f_k on D For i in 1..m: For k in 1..K: yhat_i^k = f_k(x_i) φ_i^k = SHAP(f_k, x_i) x'_i = Concatenate( φ_i^1, φ_i^2, ..., φ_i^K ) D' = { (x'_i, y_i) }_{i=1}^m Train g on D' Return E(x): For k in 1..K: φ^k(x) = SHAP(f_k, x) x' = [ φ^1(x) ∥ ... ∥ φ^K(x) ] Return g(x')

(Garouani et al., 23 Jul 2025)

3. Computational Efficiency and Scalability

• Base Model Training: $O(K \cdot C_{\mathrm{train}})$, where $C_{\mathrm{train}}$ is the per-model training cost.
• Shapley Value Generation: Exact computation per instance for each model is $O(2^d)$, but, in practice, approximations are employed:
  • Kernel SHAP: $O(S \cdot d)$ per instance ($S \ll 2^d$).
  • Model-specific SHAP (e.g., TreeSHAP): $O(d)$ per instance for trees.
• Overall Overhead: $O(m \cdot \sum_{k=1}^K C_{\mathrm{SHAP}}(f_k))$ for $m$ instances.
• Meta-learner Training: Training on $\mathbb{R}^{K d}$ is of similar order to standard stacking in $\mathbb{R}^K$, but with increased feature dimensionality.

Several scalability strategies are recommended:

• Use a sampled subset of instances for explanation synthesis.
• Employ feature selection or dimensionality reduction.
• Parallelize SHAP computation.

This suggests that, with modern approximation and selection techniques, XStacking can be deployed on large datasets with moderate computational overhead (Garouani et al., 23 Jul 2025).

4. Empirical Evaluation

XStacking was empirically validated on 29 diverse real-world classification and regression datasets, with both SVM and XGBoost meta-learners:

• Classification (17 datasets):
  • With SVM meta-learner, XStacking matched or surpassed standard stacking in 16/17 cases.
  • With XGBoost, improvements were noted in 14/17 cases.
  • Notable accuracy gains: "adult" (+2.6 percentage points), "vehicle" (+5.9 points).
• Regression (12 datasets):
  • Outperformed classical stacking in 11/12 tasks (SVM meta-learner).
  • Example: "cpu_small" dataset — mean squared error reduced from 22.4 to 11.3 (SVM), and further to 7.6 (XGBoost).
• Statistical Significance: Improvements were statistically significant under the Wilcoxon test ($p < 0.01$).
• Efficiency: SHAP value computation introduced only moderate overhead, with accuracy improvements compensating for the increased cost.
• Interpretability: The meta-learner’s input space, being constructed from Shapley attributions, allows direct tracing of final ensemble output to underlying feature contributions, enabling fully transparent ensemble explanations (Garouani et al., 23 Jul 2025).

5. Comparative Interpretability and Accuracy

Classical stacking coupled with post-hoc explanation treats the two parts of the pipeline as independent: predictions are produced, then explanations are generated in a separate step. This separation leads to explanations that are approximate and can be fragile.

In contrast, XStacking:

• Integrates explanation into the core stacking mechanism, so interpretability is inherent and not an auxiliary afterthought.
• Achieves predictive performance on par with or superior to traditional stacking—owing to the expanded and more informative meta-feature space.
• Offers a meta-model where parameters and structure are fully transparent at inference due to the use of Shapley inputs, supporting robust feature attribution for final predictions.

A plausible implication is that XStacking avoids the interpretive limitations and brittleness associated with post-hoc methods, while simultaneously expanding the capacity for model improvement via explanation-driven meta-features (Garouani et al., 23 Jul 2025).

6. Implementation Variants and Design Parameters

• The transformation $T$ admits flexibility: meta-features may be limited to Shapley vectors, or may concatenate raw features and predictions as needed.
• The choice of base learners ($f_k$) and the meta-learner ($g$) is model-agnostic; standard choices include SVM and XGBoost, but the approach is extensible to arbitrary base model families or stacking depths.
• SHAP-based explanations are model-agnostic but, in practice, are most efficient for models supporting fast or approximate SHAP computation.

This suggests that XStacking’s architecture is compatible with a wide spectrum of ensemble configurations and can incorporate recent advances in model-agnostic explainability, further enhancing both scalability and transparency (Garouani et al., 23 Jul 2025).

7. Practical Implications and Use Cases

By embedding explanations in meta-learner training, XStacking provides a recipe for creating ensemble systems that balance predictive effectiveness and transparency. This is particularly pertinent in domains requiring responsible machine learning, such as regulated industries, scientific discovery, and applications where explainable AI is non-negotiable. Its design renders model decisions auditable down to feature-level contributions from individual base learners, while typically improving both classification and regression performance metrics compared to traditional stacking approaches (Garouani et al., 23 Jul 2025).