Stacked Models: Design and Applications

• Stacked models are multi-level constructions that integrate outputs, latent codes, or hidden states from base learners via a meta-learner to refine predictions.
• They are applied across ensemble learning, deep architectures, and optimization, providing compositional aggregation and improved model reliability.
• Stacked models enable interpretable weight assignments and variance reduction, facilitating robust performance and effective error control in complex systems.

A stacked model is a modeling construction in which outputs, latent codes, hidden states, or replicated subsystems are arranged in multiple levels or coupled layers so that a later stage combines, refines, or constrains earlier-stage computations. In statistical learning, the term most commonly denotes stacked generalization, where predictions from multiple base learners become inputs to a meta-learner. In other literatures, the same term refers to stacked recurrent layers, stacked autoregressive queues, stacked tensorial operators, low-rank parameter-sharing across repeated blocks, or directly coupled replicas in Ising-type systems. The common principle is compositional aggregation: model diversity, architectural depth, or replicated structure is exploited through an explicit second-stage combination rather than treated as an unstructured collection of outputs (Li et al., 14 Feb 2026, Ploshchik et al., 2022, Abe et al., 14 Jan 2026).

1. Scope of the term

A stacked model is not a single algorithmic object. It is a family of constructions in which one layer of computation is built on top of another. In ensemble learning, the canonical form is a predictor

$h(x) = g(f_1(x), \ldots, f_M(x)),$

where $f_1,\ldots,f_M$ are base learners and $g$ is a meta-learner (Chatzimparmpas et al., 2020). In deep architectures, stacking may mean repeated blocks whose hidden states feed subsequent blocks, or stage-wise latent-code refinement. In physics and optimization, it can mean that multiple replicas or subsystems are directly coupled through an additional interaction term rather than trained independently (Abe et al., 14 Jan 2026, Rossini et al., 26 Jan 2026).

Context	Stack construction	Representative sources
Ensemble learning	Base predictions passed to a meta-learner	(Ploshchik et al., 2022, Chatzimparmpas et al., 2020)
Inference-time coordination	Black-box model outputs aggregated by learned reliabilities	(Li et al., 14 Feb 2026)
Deep architectures	Repeated or multi-stage latent/state composition	(Karevan et al., 2018, Wang et al., 2020, Seo et al., 2023)
Physics and optimization	Direct coupling of replicas or subsystems	(Abe et al., 14 Jan 2026, Rossini et al., 26 Jan 2026)

This terminological breadth matters because the mathematical role of the “stack” changes across fields. In some settings it is a supervised meta-model, in others an unsupervised or semi-supervised reliability combiner, and in others a structural constraint on dynamics or operator approximation. A plausible implication is that “stacked model” is best understood as a design pattern defined by composition and aggregation, not by a fixed training recipe.

2. Meta-ensembles and collective inference

The standard stacked-generalization pipeline constructs meta-level features from out-of-fold predictions. If $\{(x_i,y_i)\}_{i=1}^N$ are labeled instances and $f_1,\ldots,f_M$ are base learners, a standard formulation uses out-of-fold predictions $f_m^{(-i)}(x_i)$ to form

$z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$

then trains a meta-model $g$ on $\{(z_i,y_i)\}$ (Ploshchik et al., 2022). This out-of-fold construction is the standard safeguard against leakage. Relative to bagging and boosting, stacking does not merely average or sequentially reweight learners; it learns a second-stage mapping over their predictions (Chatzimparmpas et al., 2020).

A recent formulation extends this logic to black-box foundation models exposed only through APIs. “StackingNet” coordinates diverse models purely at inference time and learns only from their outputs, without access to internal parameters or original training data (Li et al., 14 Feb 2026). For regression, its aggregate predictor is

$$\widehat{H}(\mathbf{x}) = \sum_{j=1}^{M} w_j h_j(\mathbf{x}) + b,$$

with initialization $f_1,\ldots,f_M$0, $f_1,\ldots,f_M$1, and non-negativity constraints $f_1,\ldots,f_M$2, $f_1,\ldots,f_M$3 used to stabilize learning on normalized label spaces (Li et al., 14 Feb 2026). For classification, one-hot outputs are stacked into $f_1,\ldots,f_M$4, and aggregation is

$f_1,\ldots,f_M$5

The same framework supports supervised, unsupervised, and semi-supervised objectives, including consensus pseudo-labeling and a weight-normalization regularizer (Li et al., 14 Feb 2026).

The distinctive contribution of this reliability-weighted variant is not merely improved accuracy. The learned weights are directly interpretable as reliabilities, enabling ranking and pruning. In reported evaluations, the reliability order correlated strongly with ground-truth performance, with Kendall’s $f_1,\ldots,f_M$6 on seven of eight HELM datasets, and corrupted models subjected to random injection or label flipping received near-zero weights and were detected reliably when inspecting half the ensemble (Li et al., 14 Feb 2026). The same paper reports that on academic paper rating, StackingNet trained with 1% labeled data achieved the lowest mean absolute error across ICLR2025/2024 and NeurIPS2024/2023 datasets, while on eight HELM classification datasets U-StackingNet ranked first on average among unsupervised aggregators and S-StackingNet ranked first among supervised methods (Li et al., 14 Feb 2026).

This line of work frames stacking as collective inference rather than only post hoc blending. The stack becomes an explicit mechanism for variance reduction, calibration, bias mitigation, and adversary filtering, all at inference time.

3. Structured weighting, censoring, and input-dependent stacking

Several statistical literatures use stacking to define a weighted combination of candidate estimators under domain-specific constraints. In survival analysis, stacked survival models estimate a conditional survival function by convexly combining candidate parametric, semi-parametric, and non-parametric models,

$f_1,\ldots,f_M$7

with weights learned by minimizing an IPCW Brier-score criterion over a time grid (Wey et al., 2013). The method uses out-of-fold predictions to avoid rewarding overfitting, and the paper reports that stacking is among the top two methods in all six modest-dimension simulation scenarios and top two in every high-dimensional scenario, while consistently outperforming cross-validated single-model selection (Wey et al., 2013). Because the mean-squared-error decomposition contains cross-terms involving biases, variances, and correlations, the candidate library is deliberately heterogeneous; this suggests that stacking is especially useful when learners occupy different bias–variance regimes.

On graphs, dynamic stacked generalization replaces fixed meta-weights with topology-dependent functional coefficients. For binary node classification, the meta-model is

$f_1,\ldots,f_M$8

where $f_1,\ldots,f_M$9 are base-model outputs and $g$0 is a topological covariate such as degree or closeness centrality (Han et al., 2016). Smoothness is enforced through spline penalties, yielding a convex penalized logistic objective. The reported simulations show that this dynamic version matches static methods when there is no dependence on topology, but outperforms them in nonlinear dependence settings; on Cora and PubMed it achieves the highest mean accuracy among compared methods (Han et al., 2016). Here the stack is explicitly local: the optimal combination rule is allowed to vary across the graph.

Bayesian hierarchical stacking generalizes the same idea to probabilistic model averaging with input-varying weights. The stacked predictive density is

$g$1

with softmax weights

$g$2

and hierarchical priors that partially pool the logit functions $g$3 across groups or covariates (Yao et al., 2021). The paper gives theory bounds showing that complete-pooling stacking weights behave like probabilities of local model optimality under separation conditions, and that allowing weights to vary across inputs improves performance when model adequacy is heterogeneous in $g$4 (Yao et al., 2021). This is stacking as Bayesian gating, but with the experts pre-fit and the combination optimized through cross-validated predictive densities rather than joint expert training.

A different use appears in missing-data methodology. In stacked multiple imputation, all $g$5 completed datasets are vertically concatenated and the substantive model is incorporated through per-row weights proportional to the analysis likelihood,

$g$6

with standard errors derived from Louis’s observed-information identity rather than Rubin’s rules (Beesley et al., 2019). The paper reports that the method produced unbiased parameter estimates when the analysis model was correctly specified and that the proposed Louis-based estimator achieved near-nominal 95% coverage in simulations (Beesley et al., 2019). In this setting, stacking is neither a prediction ensemble nor a repeated network architecture; it is an inferential device for representing imputational uncertainty within one weighted analysis.

4. Stacking in deep neural architectures

In deep learning, stacked models often denote repeated or multi-stage architectures in which intermediate representations are re-used rather than merely ensembled. A two-layer spatio-temporal stacked LSTM for weather forecasting uses $g$7 independent first-layer LSTMs, one per location, and concatenates their hidden states into a second LSTM that learns cross-location interactions (Karevan et al., 2018). With the total first-layer neuron count matched to the conventional baseline, the first-layer input parameterization is reduced by a factor of $g$8. On daily temperature forecasting across five cities, the paper reports substantial improvements in several horizons, such as minimum-temperature 3-day-ahead Nov/Dec MAE decreasing from 3.04 to 1.72 and MSE from 14.16 to 4.22 (Karevan et al., 2018). Here stacking means layered recurrent composition plus structured spatial factorization.

For very deep sequential recommendation, StackRec uses iterative layer stacking rather than training a deep network from scratch. The key empirical observation is that in a trained NextItNet, cosine similarity between adjacent block outputs exceeds 0.90 from block 2 onward on ML20 and Kuaibao (Wang et al., 2020). This motivates duplicating pre-trained blocks to form deeper models, using either adjacent-block stacking or cross-block stacking. In continual-learning and train-from-scratch scenarios, the paper reports $g$9 speedups in wall-clock training while matching or slightly improving recommendation quality; for example, ML20 training time for a 32-block model decreases from 490 minutes to 280 minutes in the TS scenario (Wang et al., 2020). This suggests a distinct sense of stacking: depth expansion by parameter reuse across similar blocks.

Video prediction offers another formulation. IAM4VP uses a stacked autoregressive mechanism with a fixed observed queue $\{(x_i,y_i)\}_{i=1}^N$0 and a future queue $\{(x_i,y_i)\}_{i=1}^N$1 that progressively stores features of predicted frames (Seo et al., 2023). Each future frame is predicted from the same observed context plus the stacked queue of previous predictions: $\{(x_i,y_i)\}_{i=1}^N$2 Unlike pure autoregression, every horizon remains anchored to the observed inputs; unlike pure non-autoregression, successive outputs are explicitly coupled. On Moving MNIST, IAM4VP achieves MSE 15.3, MAE 49.2, and SSIM 0.966, improving on MIMO-VP and SimVP, and it also reports state-of-the-art results on TrafficBJ, Human3.6, SEVIR, and ICAR-ENSO (Seo et al., 2023).

The stacked tensorial neural network extends stacking to reduced-order operator learning for a parametric PDE. Multiple tensorial neural networks $\{(x_i,y_i)\}_{i=1}^N$3 are blended through parameter-conditioned softmax weights: $\{(x_i,y_i)\}_{i=1}^N$4 With $\{(x_i,y_i)\}_{i=1}^N$5, $\{(x_i,y_i)\}_{i=1}^N$6, and 195,890 parameters, the representative model reports average relative $\{(x_i,y_i)\}_{i=1}^N$7 errors of 0.011 on in-range test group 1 and approximately 0.017 on the zero-flux test group, together with batch speedups of about $\{(x_i,y_i)\}_{i=1}^N$8 on GPU and $\{(x_i,y_i)\}_{i=1}^N$9 on CPU relative to GMRES (Wagner, 2023). The stack here is a parameter-conditioned mixture of low-rank operator surrogates.

A related parameter-efficiency perspective appears in LORS, where repeated decoder modules share a base parameter and retain only low-rank residuals per layer: $f_1,\ldots,f_M$0 Applied to AdaMixer on MS COCO, LORS reduces decoder parameters by about 70% while preserving or improving accuracy; for example, with a ResNet-50 backbone and 300 queries under the $f_1,\ldots,f_M$1 schedule, AP increases from 47.0 to 47.6 while total parameters drop from 139M to 60M (Li et al., 2024). Here the stack remains architecturally deep, but most parameters are no longer layer-specific.

A two-stage latent formulation appears in conditional molecular generation. The stacked adversarial model first learns shared and modality-specific latent codes for molecules and gene expression, then reuses the Stage-1 expression latents in Stage 2 to refine molecule generation (Liu et al., 2021). On LINCS L1000, validity improves from 64.7% to 69.1%, and overall similarity to template drugs increases from 0.247 to 0.265, with a small drop in diversity from 0.85 to 0.83 (Liu et al., 2021). In this sense, stacking is a stage-wise latent refinement strategy.

5. Replica coupling, quantum systems, control stacks, and reserving

Outside mainstream predictive modeling, stacked models can denote direct coupling of replicated or layered physical subsystems. In replica-coupled Ising machines, the stacked model uses a ring topology in replica space,

$f_1,\ldots,f_M$2

so that adjacent replicas interact directly (Abe et al., 14 Jan 2026). Ferromagnetic coupling $f_1,\ldots,f_M$3 promotes agreement, while antiferromagnetic coupling $f_1,\ldots,f_M$4 disfavors alignment. On quadratic assignment problems solved with simulated annealing, the paper reports that the ferromagnetically coupled stacked model maintains $f_1,\ldots,f_M$5 across $f_1,\ldots,f_M$6 at $f_1,\ldots,f_M$7, and yields the best approximation ratios across tested sizes $f_1,\ldots,f_M$8, whereas the penalty-spin model suffers “cooperation collapse” at large $f_1,\ldots,f_M$9 because centralized averaging washes out sparse solution information (Abe et al., 14 Jan 2026).

A different quantum usage considers two stacked quantum Ising subsystems coupled by a local Hamiltonian term preserving the $f_m^{(-i)}(x_i)$0 symmetry of each subsystem (Rossini et al., 26 Jan 2026). For identical subsystems, the model is equivalent to the quantum Ashkin–Teller chain, and along the critical line $f_m^{(-i)}(x_i)$1, $f_m^{(-i)}(x_i)$2, it exhibits central charge $f_m^{(-i)}(x_i)$3, fixed $f_m^{(-i)}(x_i)$4, fixed $f_m^{(-i)}(x_i)$5, and continuously varying correlation-length exponent

$f_m^{(-i)}(x_i)$6

In two dimensions, the same symmetry-preserving coupling yields multicritical behavior governed by effective O(2) symmetry with 3D XY exponents $f_m^{(-i)}(x_i)$7, $f_m^{(-i)}(x_i)$8, and $f_m^{(-i)}(x_i)$9 (Rossini et al., 26 Jan 2026). In this literature, stacking denotes coupled subsystems whose critical behavior is altered by the interlayer structure itself.

Control theory uses the same word in yet another way. In stacked adaptive dynamic programming with unknown dynamics, one stack appears in the Kalman filter, which aggregates $z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$0 past state/control pairs to estimate a local linear model, and another stack appears in the actor update, which optimizes a sequence of future controls over a horizon of length $z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$1 by minimizing the sum of predicted Q-function values (Osinenko et al., 2020). The stacked actor update is

$z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$2

In the reported nonlinear case study, stacked ADP-Q converges faster to the origin than single-step ADP-Q, MPC, and gradient descent on the running cost (Osinenko et al., 2020). This is stacking as multi-step horizon coupling.

Insurance reserving provides a statistical–actuarial example. A stacked model based on a hybridized ANN combines Random Forest, Gradient Boosting, ANN, Chain Ladder factors, and CSR outputs to predict scaled cumulative claims runoff (Ramos-Pérez et al., 2020). The second-level ANN ingests $z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$3, $z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$4, $z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$5, $z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$6, and $z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$7, then uncertainty is added through a log-normal simulation model. The paper reports markedly lower \%RMSE for reserves than classical benchmarks in several lines of business; for example, in Other Liability, \%RMSE($z_i = [f_1^{(-i)}(x_i), \ldots, f_M^{(-i)}(x_i)],$8) decreases from 5.274% for ODP and 5.086% for Mack to 0.722% for the stacked ANN (Ramos-Pérez et al., 2020). Here the stack combines machine-learning and actuarial reserve models within one meta-network.

6. Evaluation, interpretability, and recurrent limitations

Because stacked models introduce an additional combining layer, evaluation often focuses not only on predictive accuracy but also on reliability, calibration, and interpretability. In visual analytics for stacking, StackGenVis treats data instances, algorithms, and models as aligned analytic spaces. It supports eight validation metrics, interactive feature selection, model pruning, and provenance tracking of stack variants (Chatzimparmpas et al., 2020). MetaStackVis extends this logic from a single logistic-regression metamodel to multiple metamodel families and pairwise metamodel comparisons, using HDBSCAN clustering, UMAP projections, and a zone-based matrix to inspect complementarity and oracle upper bounds for metamodel pairs (Ploshchik et al., 2022). The tool was evaluated through expert interviews and is explicitly designed to expose the trial-and-error nature of stack construction.

Explainability can also be propagated through the stack. In static route-free ETA prediction, a two-level ensemble combines first-level Random Forest, XGBoost, and FCNN models with second-level meta-models, then explains both levels with LIME and SHAP (Schleibaum et al., 2022). The paper defines three joining methods for explanations: JM1 preserves per-base contributions as a higher-dimensional object, JM2 aggregates them by a weighted sum, and JM3 reweights meta-level importances around a threshold to accentuate above-average contributors (Schleibaum et al., 2022). On the NYC test set, the best stacked model, L2-FCNN, attains MAE 169.429 s, MRE 0.2023, and MAPE 22.912%, outperforming the best single level-1 model at MAE 178.232 s (Schleibaum et al., 2022). The explanations indicate that time-of-day and distance dominate, while temperature has a small local contribution.

A clinical prediction example emphasizes calibration. QISICGM combines Random Forests, Extra Trees, a transformer-based neighbor-sequence model, CNN-Seq, and an FFNN through a 17-dimensional out-of-fold meta-feature vector and a logistic-regression meta-learner (II, 12 Sep 2025). On the augmented PIMA Indians Diabetes dataset, the stacked ensemble reports out-of-fold F1 score 0.8933, AUC 0.8699, and Brier score 0.12, with isotonic regression applied to each base learner before stacking (II, 12 Sep 2025). This suggests that stacking is increasingly used not only to maximize discrimination but also to produce clinically interpretable and calibrated probabilities.

Across these literatures, several recurrent limitations are explicit. First, leakage control is indispensable: out-of-fold meta-features, cross-validation, or held-out diagonals are standard safeguards (Ploshchik et al., 2022, Wey et al., 2013, Chatzimparmpas et al., 2020). Second, gains depend on diversity: highly correlated models, heavily shared training data, or uniformly weak base predictors reduce the benefit of aggregation (Li et al., 14 Feb 2026, Wey et al., 2013). Third, no-pooling or overly flexible local weighting can overfit, which is why hierarchical priors, simplex constraints, non-negativity constraints, or visualization-guided pruning recur across methods (Yao et al., 2021, Han et al., 2016, Li et al., 14 Feb 2026). Fourth, stacking can increase inference or training complexity unless accompanied by compression or sharing mechanisms, as in StackRec and LORS (Wang et al., 2020, Li et al., 2024).

Taken together, the research record indicates that a stacked model is best treated as a structured second-order design: a model over models, layers, stages, or replicas. Whether the goal is collective inference across black-box foundation models, censoring-aware risk minimization, spatio-temporal fusion, parameter-efficient deep stacks, or replica stabilization in optimization, the defining operation is the same—an additional level of organization is introduced to exploit complementarity, regularize redundancy, or impose coordination (Li et al., 14 Feb 2026, Yao et al., 2021, Abe et al., 14 Jan 2026).