Multi-Level Attention Stacked Ensemble

• Multi-Level Attention Stacked Ensemble is a broad framework that integrates attention mechanisms at various representational or modal levels with stacked fusion strategies.
• These architectures employ iterative cross-modal reasoning, depth-selective attention, and meta-learning to achieve state-of-the-art performance in tasks like image captioning, scene labeling, and medical imaging.
• The approach emphasizes dynamic fusion and adaptive attention, though ambiguities persist due to varied interpretations of 'stacked' and 'ensemble' across the literature.

“Multi-Level Attention Stacked Ensemble” is best treated as an umbrella designation for architectures that combine attention mechanisms across more than one representational, modal, or decision level, and then combine the resulting signals through stacked refinement, hierarchical fusion, or explicit meta-learning. In the literature represented here, the expression does not denote a single standardized architecture; instead, it covers several related patterns, including stacked cross-modal reasoning for image captioning, multi-level attention refinement in text matching, attention-weighted fusion of multi-scale scene-labeling branches, self-attention-based stacked generalization for tabular prediction, and hierarchical meta-learning over heterogeneous medical-image models (Pourkeshavarz et al., 2023, Tay et al., 2018, Fan et al., 2016, Tan et al., 2023, Rahman et al., 5 Jul 2026, Saha et al., 27 Jul 2025). By contrast, the arXiv entry “Stacked ensemble-based mutagenicity prediction model using multiple modalities with graph attention network” (Liyaqat et al., 2024) cannot substantiate any such method, because the supplied document is described as a manuscript template with no substantive methodological, experimental, or mathematical content.

1. Terminology and scope

The phrase combines three ideas whose meanings vary substantially across papers. “Multi-level attention” may refer to attention applied at several feature depths, several modalities, several decoder stages, or several attention heads across transformer layers. “Stacked” may refer to serially composed blocks inside one end-to-end network, to classical stacked generalization with base learners and a higher-level combiner, or to a coarse-to-fine decoder whose later stages refine earlier outputs. “Ensemble” may denote heterogeneous independently trained predictors, or may be used more loosely for internal multi-branch architectures whose outputs are fused by a learned combiner.

This variation is explicit in the surveyed work. “Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning” defines stacking as repeated cross-modal consolidation layers inside one encoder-decoder model, with only a later experiment introducing an actual ensemble of separately trained models (Pourkeshavarz et al., 2023). “Tab-Attention: Self-Attention-based Stacked Generalization for Imbalanced Credit Default Prediction” uses stacked generalization language, but its combiner is a self-attention fusion module over multi-view neural predictors rather than a conventional out-of-fold meta-learner (Tan et al., 2023). “PulmoSight-XAI: An Explainable Multi-View Attention Ensemble with Gradient Boosting Meta-Learning for Multi-Label Chest X-Ray Classification” is closer to the classical ensemble sense, because it explicitly organizes Level-0 CNNs, Level-1 gradient-boosting meta-learners, and Level-2 stacking or alpha blending (Rahman et al., 5 Jul 2026). “Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers” uses “stacked” in a mechanistic sense: attention heads form a serial retrieval circuit across layers, not a collection of independent experts (Musat, 2024).

A plausible implication is that the label is only informative when the locus of hierarchy is stated precisely: representation hierarchy, modality hierarchy, view hierarchy, ensemble hierarchy, or optimization hierarchy.

2. Forms of multi-level attention

One major family applies attention repeatedly across sequential stages. In SCFC for image captioning, a Context-Aware Attributes representation is used as the initial textual component, and stacked Cross-modal Feature Consolidation layers then iteratively compute region relevance, attended visual summaries, and refined cross-modal representations. Each layer performs one reasoning step, and the output of one level conditions the next. The model therefore treats attention as an iterative cross-modal reasoning process rather than a one-shot fusion operation (Pourkeshavarz et al., 2023).

A second family distributes attention across feature depths. In the chest X-ray framework of PulmoSight-XAI, the backbone exposes stage features $L_1, L_2, L_3$, with the shallow stage passed to fusion without CBAM refinement and the deeper stages passed through channel attention followed by spatial attention. The paper characterizes this as a “multi-scale feature fusion strategy augmented with Convolutional Block Attention Modules (CBAM),” so attention operates selectively at deeper semantic stages while shallower detail-preserving stages bypass it (Rahman et al., 5 Jul 2026). A related but earlier design appears in “Multi-level Contextual RNNs with Attention Model for Scene Labeling,” where pool3, pool4, and pool5 branches each produce a score map, and an attention module assigns per-location softmax weights over levels before fusion (Fan et al., 2016).

A third family inserts attention refinement between stacked encoders. CSRAN uses stacked BiLSTMs, but after each recurrent layer a CAFE-based attention block injects cross-sequence comparison features into the next layer. The paper explicitly calls this “multi-level attention refinement,” because alignment evidence discovered at one layer is appended before the next recurrent transformation (Tay et al., 2018). In this setting, attention does not merely pool outputs; it changes what deeper layers receive.

A fourth family concerns layered attention circuits in transformers. The retrieval analysis in (Musat, 2024) shows that successful models solve chained retrieval by letting early heads create pairwise linkages and later heads exploit information written into the residual stream by earlier heads. Here “multi-level attention” is neither multimodal nor multi-scale. It is depth-compositional: later attention heads depend on earlier ones to make the next hop possible.

These variants all satisfy a broad multi-level criterion, but they do so through different mechanisms: stagewise re-attention, depth-selective attention, inter-layer refinement, or compositional head circuits.

3. Meanings of stacking and ensemble

The strongest source of ambiguity lies in “stacked ensemble.” In some systems, stacking is internal to one differentiable architecture. Stack-VS is exemplary: it uses a sequence of decoder stages, each re-optimizing visual and semantic attention and passing its hidden state and attended contexts to the next stage. This is a stacked decoder, not a classical ensemble, because stages are not independently trained predictors and there is no separate meta-learner over frozen base outputs (Wei et al., 2019).

In other systems, stacking is closer to stacked generalization. Tab-Attention first creates one global view and six local views, trains view-specific MLP branches, concatenates the resulting global representation and local outputs, and then applies self-attention followed by a final sigmoid layer. The method is therefore a stacked architecture over multi-view base learners, but the paper does not describe out-of-fold prediction generation or a separate stagewise training protocol, so it only partially matches strict classical stacking (Tan et al., 2023).

PulmoSight-XAI is more literal. It trains five Level-0 CNN backbones per view, constructs meta-features from probabilities, test-time augmentation statistics, predictive entropy, cross-model mean, standard deviation, range, and pairwise differences, then fits per-class XGBoost, LightGBM, and CatBoost meta-learners at Level-1, followed by logistic regression stacking or class-wise alpha blending at Level-2. This is a hierarchical meta-learning stack in the conventional ensemble sense (Rahman et al., 5 Jul 2026).

MASE for lung cancer detection is similarly explicit at the logit level. Three pretrained backbones produce class logits, a model-wise attention MLP learns one scalar weight per model, a class-wise attention MLP learns one scalar weight per class, and a compact meta-learner maps $[\mathbf{m};\mathbf{c}]$ to final logits. Its stacking is therefore base learners plus attention-conditioned meta-classification (Saha et al., 27 Jul 2025).

By contrast, “Multi-attention Networks for Temporal Localization of Video-level Labels” combines multiple attention branches by max pooling within the model and combines heterogeneous models by weighted summation across the final system. The paper explicitly describes the final ensemble as a weighted summation of 12 models, so it is an ensemble, but not a stacked ensemble in the meta-learning sense (Zhang et al., 2019). “Attention-based Ensemble for Deep Metric Learning” is likewise a true ensemble of learners, but its diversity comes from multiple attention masks over the same feature tensor, so it is attention-based and ensemble-based without being multi-level or stacked in the same sense (Kim et al., 2018).

4. Fusion mechanisms and mathematical organization

Although the implementations differ, the surveyed models repeatedly organize attention around a small set of fusion patterns.

One pattern is additive cross-modal attention. In Stack-VS, visual attention and semantic attention are computed separately at each stage, with scores depending on current branch states and previous-stage attended contexts. The attended vectors are then fused through projected summation inside the language LSTM input,

$$x_t^{i,L} = \mathrm{FC}^V(v_t) + \mathrm{FC}^S(e_t) + h_t^{i,V} + h_t^{i,S},$$

so fusion is dynamic, stage-specific, and conditioned on both modalities (Wei et al., 2019).

A second pattern is self-attention over stacked learner outputs. In Tab-Attention, the concatenated object

$F^* = concat(F_G^2, F_1^3, F_2^3, \ldots, F_6^3)$

is projected into $Q$, $K$, and $V$, with attention written as

$A = softmax(Q * K), \qquad F^{attention} = A * V.$

Attention therefore acts as an adaptive combiner over global and local predictor outputs rather than over raw input columns (Tan et al., 2023).

A third pattern is level-wise soft gating. In ML-CRNNs, the fused score at spatial position $i$ and class $c$ is

$[\mathbf{m};\mathbf{c}]$0

The attention weights are thus spatially varying and level-specific, generalizing average and max fusion into learned per-location mixtures (Fan et al., 2016).

A fourth pattern is selective attention plus tabular meta-learning. In PulmoSight-XAI, CBAM refines deeper feature maps through

$[\mathbf{m};\mathbf{c}]$1

after which pooled multi-scale vectors are concatenated and later transformed again into meta-features for gradient-boosting learners. Here attention is not the final combiner; it shapes the representation before ensemble stacking (Rahman et al., 5 Jul 2026).

A fifth pattern is attention over models and classes. In MASE, the concatenated logits $[\mathbf{m};\mathbf{c}]$2 drive two parallel MLPs: $[\mathbf{m};\mathbf{c}]$3 followed by a meta-learner on $[\mathbf{m};\mathbf{c}]$4. This produces an input-dependent and class-sensitive stacking rule rather than fixed averaging (Saha et al., 27 Jul 2025).

These formulations show that “attention” in such systems is not restricted to token weighting. It can gate levels, models, classes, modalities, stages, or ensemble members, and it can appear either before or after the stacked combination.

5. Representative application domains and empirical claims

The architectural motif has appeared across captioning, scene labeling, text matching, tabular risk prediction, medical imaging, video localization, and metric learning. The following examples illustrate how the same broad design language maps to different technical objectives.

System	Domain	Reported outcome
SCFC (Pourkeshavarz et al., 2023)	Image captioning	Outperforms various state-of-the-art benchmarks on MSCOCO and Flickr30K
ML-CRNNs (Fan et al., 2016)	Scene labeling	Attention fusion outperforms average- and max-pooling fusion strategies
ABE-8 (Kim et al., 2018)	Deep metric learning	Recall@1 of 85.2 on CARS-196
Tab-Attention (Tan et al., 2023)	Credit default prediction	About 32.92% Recall$[\mathbf{m};\mathbf{c}]$5 and 16.05% f1$[\mathbf{m};\mathbf{c}]$6 average improvement
PulmoSight-XAI (Rahman et al., 5 Jul 2026)	Multi-label chest X-ray classification	Macro-average AUROC 0.9319 for frontal and 0.9154 for lateral
MASE (Saha et al., 27 Jul 2025)	Lung nodule classification in CT	98.09 accuracy and 0.9961 AUC

The evidence for level-aware fusion is especially direct in scene labeling. ML-CRNNs$[\mathbf{m};\mathbf{c}]$7 reports 91.9 pixel / 77.2 class on CamVid, 86.9 / 57.7 on SiftFlow, and 87.2 / 78.4 on Stanford-background, exceeding both average and max fusion on all three datasets (Fan et al., 2016). In deep metric learning, ABE-8$[\mathbf{m};\mathbf{c}]$8 reports Recall@1 of 85.2 on CARS-196, 60.6 on CUB-200-2011, 76.3 on SOP, and 87.3 on In-shop, while the divergence loss prevents learner collapse; without it, the ensemble Recall@1 on CARS-196 drops to 69.7 (Kim et al., 2018). In chest X-ray classification, PulmoSight-XAI attributes gains to “two design decisions: the multi-scale fusion paradigm … and the structured meta-learning layer,” with TTA alone yielding only modest average gains of about $[\mathbf{m};\mathbf{c}]$9–$$x_t^{i,L} = \mathrm{FC}^V(v_t) + \mathrm{FC}^S(e_t) + h_t^{i,V} + h_t^{i,S},$$0 (Rahman et al., 5 Jul 2026). In CT nodule classification, MASE reports 31 total misclassifications against 46 for the best individual model, corresponding to a 32.6% relative reduction in test errors, and also reports Wilcoxon signed-rank p-values of $$x_t^{i,L} = \mathrm{FC}^V(v_t) + \mathrm{FC}^S(e_t) + h_t^{i,V} + h_t^{i,S},$$1, $$x_t^{i,L} = \mathrm{FC}^V(v_t) + \mathrm{FC}^S(e_t) + h_t^{i,V} + h_t^{i,S},$$2, and $$x_t^{i,L} = \mathrm{FC}^V(v_t) + \mathrm{FC}^S(e_t) + h_t^{i,V} + h_t^{i,S},$$3 against the three base models (Saha et al., 27 Jul 2025).

At the same time, the evidentiary quality is uneven. Several papers provide strong end-task gains but limited internal ablation. PulmoSight-XAI does not provide a clean isolated table for “with vs. without CBAM” or “with vs. without multi-scale fusion” (Rahman et al., 5 Jul 2026). MASE does not provide explicit ablation tables for model-attention-only, class-attention-only, or simple averaging (Saha et al., 27 Jul 2025). Stack-VS likewise does not provide a clean table isolating stacked decoding, visual attention, and semantic attention (Wei et al., 2019). This suggests that empirical success does not always imply precise attribution of gains to each architectural level.

6. Ambiguities, misconceptions, and evidentiary limits

A recurrent misconception is to treat all uses of “stacked,” “multi-attention,” and “ensemble” as interchangeable. The literature does not support that equivalence. Parallel multi-attention over one sequence, as in the YouTube-8M localization model, is not the same as multi-level attention refinement between stacked recurrent layers (Zhang et al., 2019). A serial stack of transformer heads implementing chained retrieval is not an ensemble of independent predictors (Musat, 2024). An attention-based ensemble with multiple masks over one feature tensor is not automatically a multi-level model (Kim et al., 2018). Conversely, a multi-level attention architecture may have no classical ensemble stage at all, as in CSRAN or Stack-VS (Tay et al., 2018, Wei et al., 2019).

A second misconception is that the presence of a graph attention network or any attention block is sufficient to establish a “Multi-Level Attention Stacked Ensemble.” The supplied record for (Liyaqat et al., 2024) is important precisely because it shows the opposite. The document is described as containing no model architecture, no modalities, no graph construction procedure, no GAT details, no stacked ensemble, no datasets, no experiments, no SHAP analysis, and no mathematical formulation beyond template structure. It therefore cannot serve as evidence for a multi-level attention stacked ensemble, a single-model GAT, or a multimodal stacked architecture (Liyaqat et al., 2024).

A third issue is nomenclature drift. Some papers use “stacked” to mean multi-stage reasoning inside one network; others mean late-stage model combination; others mean layer depth. Some use “ensemble” only for a final weighted sum, while others reserve it for independently trained learners combined by a meta-learner. A plausible implication is that any rigorous use of the phrase should specify at least three axes: where attention is applied, what is stacked, and how outputs are fused.

Taken together, the surveyed literature supports a narrow conclusion and a broad one. The narrow conclusion is that there is no single canonical architecture called “Multi-Level Attention Stacked Ensemble.” The broad conclusion is that a stable design pattern does exist: architectures gain expressive power when they preserve multiple representational levels, let attention act selectively over those levels or over model outputs, and then combine the resulting signals through serial refinement or learned meta-combination rather than fixed averaging.