Transformer Stacking Overview

• Transformer stacking is the composition of transformer layers or models using deep stacking, cross-layer fusion, and recurrent sharing to enhance computational expressivity.
• It includes techniques like vertical depth composition, explicit stack memory augmentation, and gradual depth growth to improve reasoning, generalization, and parameter efficiency.
• Ensemble stacking leverages heterogeneous transformer predictors via a meta-learner to correct errors and boost performance in tasks such as translation, parsing, and image analysis.

Transformer stacking denotes a family of constructions in which transformers are composed across depth, across training stages, across auxiliary memory structures, or across model outputs. In the narrow architectural sense, it refers to the canonical composition of encoder or decoder blocks into a deep network. In adjacent literatures, the same term also covers cross-layer fusion, recurrent parameter sharing, gradual depth growth during pretraining, explicit differentiable stack mechanisms inserted into or between transformer layers, and stacked generalization in which multiple transformer predictors are combined by a meta-learner. Formal-language studies add a mechanistic sense: transformers trained on counter languages can develop internal representations consistent with stack structure, and those representations can be causally necessary for correct prediction (Yang et al., 2022, Li et al., 2024, Singh, 2 Jun 2026).

1. Scope and terminological variants

The supplied literature uses “stacking” for several distinct but related operations.

Sense of stacking	Core mechanism	Representative work
Vertical depth composition	Multiple transformer layers are composed sequentially; capabilities depend on the number of attention layers	(Chen et al., 2024)
Cross-layer grouping and fusion	Adjacent encoder or decoder layers are grouped and fused instead of using only the top layer	(Yang et al., 2022)
Recurrent or shared-parameter stacking	A single layer is reused across depth, or most parameters are shared with small layer-private residuals	(Dabre et al., 2021, Li et al., 2024)
Stagewise growth during training	A smaller model is expanded into a deeper model during pretraining	(Saunshi et al., 2024)
Explicit stack-structured augmentation	Differentiable push, pop, and no-op mechanisms are inserted into attention or between layers	(Li et al., 2024, Zhang et al., 21 Jul 2025)
Parser-state or nested transformer stacking	Specialized heads or cascaded transformers model stack/buffer state or spatial-temporal structure	(Astudillo et al., 2020, Mao et al., 2022)
Ensemble stacking	Multiple transformer models, or a transformer plus a non-transformer model, are combined by a meta-learner	(Nguyen et al., 2023, Roy et al., 26 May 2026)
Emergent latent stacks	Hidden states linearly encode stack depth, and this encoding can be causally necessary	(Tiwari et al., 3 Feb 2025, Singh, 2 Jun 2026)

A recurrent source of ambiguity is that some papers use “stacking” to mean deeper composition of layers, whereas others use it to mean a literal stack memory or a stacked ensemble. The overlap is not merely terminological. Across these settings, the central issue is how information is preserved, transformed, or selectively exposed across multiple computational stages.

2. Depth as compositional computation

Case studies on synthetic sequence tasks isolate a direct role for depth in the computational expressivity of transformers. A transformer with only one attention layer can excel in memorization but falls short in reasoning, generalization, and contextual generalization; two attention layers suffice for reasoning and generalization; contextual generalization may necessitate three attention layers (Chen et al., 2024). The same work identifies a class of simple operations that a single attention layer can execute—copying, parsing, matching, and mapping—and argues that more complex behaviors arise by stacking layers so that these primitive operations compose across depth.

This perspective makes layer stacking a mechanism for multi-stage computation rather than a purely statistical increase in capacity. In the synthetic constructions, a two-layer model can solve in-context question answering by a copying operation followed by a matching operation, while a three-layer model can combine parsing, copying, and matching for contextual generalization (Chen et al., 2024). A plausible implication is that depth is serving as an execution trace over intermediate representations, not simply as a larger feature extractor.

The optimization side is more delicate. Signal-propagation analysis shows that stacking self-attention layers can result in rank collapse of token representations at initialization, with token correlations converging toward perfect alignment as depth increases (Noci et al., 2022). In the extreme rank-1 case, gradients with respect to query and key parameters vanish. The same analysis shows that appropriate depth-dependent scaling of the residual branches can prevent this pathology; with residual weights scaled as $O(1/\sqrt{L})$, token correlations remain nontrivially dependent on the input rather than collapsing to $1$ in the infinite-depth limit (Noci et al., 2022).

Taken together, these results indicate that stacking is computationally consequential in two distinct senses: it enables composition of primitive operations, but it also changes the forward and backward signal geometry in ways that can either support or obstruct training.

3. Cross-layer fusion and parameter-efficient deep stacks

Standard transformer usage in neural machine translation typically stacks encoder and decoder layers and then exploits only the top-layer representation, effectively treating lower layers as trivial or redundant (Yang et al., 2022). GTrans modifies this assumption by dividing encoder and decoder layers into adjacent groups and fusing the resulting group representations through learned weights. For the encoder, the fused representation is

$$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$

with $M=\lceil L_e/T_e\rceil$ and $\alpha_i=\min(iT_e,L_e)$; analogous grouping and fusion are used in the decoder, including probability fusion across decoder groups for prediction (Yang et al., 2022). The formulation includes two limiting cases: $T_e=1$ yields dense fusion across all layers, while $T_e=L_e$ recovers the vanilla transformer. Reported experiments indicate that the method can be scaled up to 60 encoder layers and 36 decoder layers (Yang et al., 2022).

A more aggressive reduction of parameter redundancy appears in recurrent stacking. Instead of using distinct parameters for each layer, the same layer is recursively applied:

$Y = L_1(L_1(L_1(L_1(L_1(L_1(X)))))).$

In machine translation, a 6-recurrence recurrently stacked model has the parameter count of a 1-layer vanilla model, and the reported parameter counts for 6-layer vanilla versus 6-recurrence recurrent stacking are 69.8M versus 30.4M on GCP Ja→En, 57.7M versus 20.2M on KFTT Ja→En, and 94.4M versus 57.7M on ASPEC Ja→En (Dabre et al., 2021). The same study reports that a 6-layer recurrently stacked model significantly outperforms a 1-layer vanilla model with identical parameter count; on ASPEC Ja→En, the scores are BLEU=27.20 versus BLEU=23.28, while a 6-layer vanilla model reaches BLEU=28.77 (Dabre et al., 2021). Transfer learning and sequence-level knowledge distillation are reported to compensate for the performance drop induced by direct training of recurrently stacked models (Dabre et al., 2021).

Depth-wise LSTMs offer a different answer to the same problem of inter-layer aggregation. Instead of residual addition, $$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$8 the sequence of stacked layers is treated as a “vertical” sequence and connected with LSTM gating. Layer normalization and feed-forward computation are absorbed into the depth-wise LSTM, so that attention sub-layers remain unchanged while inter-layer communication becomes gated and recurrent (Xu et al., 2020). On WMT 14, the reported BLEU scores move from 27.55 to 28.53 for Transformer Base on En-De, from 39.54 to 40.10 on En-Fr, from 28.83 to 29.58 for Transformer Big on En-De, and from 41.92 to 43.11 on En-Fr (Xu et al., 2020). The paper attributes the gains to selective aggregation and improved signal propagation through depth.

LORS moves further toward parameter-efficient stacking by decomposing each layer parameter matrix as

$$W_i = W^{\text{shared}} + W^{\text{private}_i}, \qquad W^{\text{private}_i} = \sum_{k=1}^{K} B_{ik}A_{ik}.$$

Most parameters are shared across layers, while a small low-rank residual remains layer-specific (Li et al., 2024). In the stacked decoder of AdaMixer, the reported decoder parameter count drops from 110M to 35M for 6 layers, a 70% reduction, while AP changes from 42.7 to 42.6 under a 1x schedule and from 47.0 to 47.6 under a 3x schedule in the reported MS COCO experiments (Li et al., 2024). This suggests that a large fraction of what conventional layer stacking buys can be retained through shared structure plus low-rank layer-specific adaptation.

4. Gradual depth growth as a training strategy

A distinct meaning of stacking appears in training curricula that grow model depth stage by stage. In gradual stacking, training is split into stages, and a smaller model from one stage initializes a deeper model in the next (Saunshi et al., 2024). The stagewise training budget is allocated according to

$T_i \propto i^\alpha,$

with schedules using $1$0 considered in the reported experiments (Saunshi et al., 2024).

MIDAS modifies gradual stacking by replicating the middle block rather than the final block when the model is expanded. Its growth operator is given as

$1$1

where the middle block is duplicated during expansion (Saunshi et al., 2024). The method is reported to speed up LLM training by up to 40%, and with the Prop2 schedule it matches baseline validation log perplexity with about 24% less compute (Saunshi et al., 2024).

The notable claim is not only efficiency but inductive bias. On 1B-parameter models, the reported all-tasks average increases from 24.0% for baseline training to 26.8% for MIDAS, open-book QA rises from 33.3% to 36.3%, and math word problems rise from 23.5% to 29.0% (Saunshi et al., 2024). Similar trends are reported at 2B and 8B scale. On fine-tuned GSM8K, the reported 2B results are 5.3% for baseline versus 10.4% for MIDAS, and the 8B results are 12.3% versus 15.2% (Saunshi et al., 2024).

The paper further introduces reasoning primitives—Induction Copying, Variable Assignment, and Pre-School Math—to probe the effect directly. In the selected 5-shot results provided, MIDAS reaches 24.3% versus 14.9% on Copying, 28.3% versus 21.6% on Variable Assignment depth-1, 45.0% versus 23.9% after fine-tuning on Variable Assignment depth-2, and 69.5% versus 62.0% on Pre-School Math with calculator (Saunshi et al., 2024). The authors conjecture that the effect is linked to looped or weight-shared models: duplicated middle layers remain highly similar after training, and MIDAS produces a cosine-similarity structure closer to looped models than end-stacking growth does (Saunshi et al., 2024).

5. Explicit stack-structured augmentations

Another research line uses “stacking” literally by adding stack-like state to the architecture. In “stack attention,” a differentiable stack over positions is maintained with soft push, skip, and pop operations. At position $1$2, the stack distribution is updated as

$1$3

and the stack attention output is

$1$4

The mechanism is inserted as an additional sub-layer after self-attention and feed-forward computation in each transformer block (Li et al., 2024). Reported results show that the resulting stack transformer solves Reverse String at 100% and Stack Manipulation at about 93%, while both vanilla transformers and the stack-augmented model remain substantially weaker on Modular Arithmetic and Solve Equation (Li et al., 2024). The stated conclusion is that the mechanism enables the transformer to model some, but not all, deterministic context-free languages.

StackTrans introduces differentiable hidden state stacks between transformer layers rather than modifying attention itself (Zhang et al., 21 Jul 2025). For each hidden state, the model predicts soft push, pop, and no-op actions, updates a fixed-size stack, and then performs a global read over the stack before passing the representation to the next layer. The architecture is explicitly designed to remain compatible with existing frameworks like flash-attention and is reported to scale from 360M to 7B parameters (Zhang et al., 21 Jul 2025). On formal-language benchmarks, the supplied summary states that StackTrans outperforms standard transformers by 30%+ on regular expression and deterministic context-free tasks, achieves near-100% test accuracy in most cases, generalizes across sequence lengths, and incurs only about 10% extra training and inference time with the low-rank multi-head stack design (Zhang et al., 21 Jul 2025).

Transition-based parsing yields another explicit use of stack structure. Stack-Transformers modify decoder cross-attention so that one head attends only to the current stack and one head attends only to the current buffer, with masks changing as parsing actions update parser state (Astudillo et al., 2020). The reported development-set results on Penn Treebank and AMR2.0 move from LAS 92.0 to 94.1 and from Smatch 77.7 to 79.5 when using one stack head and one buffer head (Astudillo et al., 2020). The paper emphasizes that explicit state modeling matters most for smaller models and limited training data.

A more general nested composition appears in reinforcement learning backbones. Transformer in Transformer cascades an inner transformer, which processes a single observation, with an outer causal transformer, which processes the history of observation embeddings, thereby extracting spatial-temporal representations in a pure-transformer backbone (Mao et al., 2022). Here stacking does not denote a pushdown stack, but it does denote a hierarchical composition of transformer modules across two levels of abstraction.

6. Emergent stack representations in hidden states

Formal-language analysis has provided a mechanistic notion of transformer stacking that is neither architectural nor ensemble-based: the latent state of the model can behave as if it contains a stack. In counter-language experiments, transformers are trained on Dyck-1 and Shuffle-$1$5 languages, which can be formulated in terms of stacks whose depths correspond to counter values (Tiwari et al., 3 Feb 2025). The reported model is an encoder-only transformer with 1 layer, 4 attention heads, hidden size 64, embedding dimension 32, and causal masking; it is trained on next-token prediction with mean squared error between predicted and true valid-token vectors (Tiwari et al., 3 Feb 2025).

Probing analyses show that even linear probes can recover stack depths from the hidden states. The latent variable is the stack depth at each token position, and selectivity is defined as the difference between probe accuracy on true labels and on randomized control labels (Tiwari et al., 3 Feb 2025). The supplied summary reports high probing accuracy for Dyck-1 and even higher values for Shuffle-$1$6 as $1$7 increases, in both learned models and an algorithmically compiled transformer produced by Tracr (Tiwari et al., 3 Feb 2025). The key limitation stated in that work is that probing shows presence of information, not whether the model uses that information causally.

The subsequent causal study closes that gap. In counter-language modeling with an encoder-only, single-layer transformer of $1$8 trained to perfect accuracy on Shuffle-$1$9 for $$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$0, linear probes are trained to predict stack depth from hidden states

$$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$1

with cross-entropy loss, and the weight matrix is decomposed by SVD,

$$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$2

to extract the top right singular vector $$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$3 as a principal stack direction (Singh, 2 Jun 2026). The hidden state is then ablated along that direction before decoding,

$$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$4

According to the supplied findings, targeted ablation causes positional accuracy to drop substantially as $$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$5 increases and makes sequential accuracy collapse almost to zero even for moderate $$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$6, whereas ablation along random directions has no significant effect (Singh, 2 Jun 2026). The reported interpretation is that the transformer does not merely encode stack depth correlates; it directly relies on this representation for prediction, and there are no alternate or redundant pathways compensating for the removed information (Singh, 2 Jun 2026).

A common misconception in this area is that probe recoverability is sufficient evidence of mechanism. The two papers jointly distinguish these claims: the earlier work establishes recoverable stack-like encoding, while the later work establishes causal necessity under targeted intervention (Tiwari et al., 3 Feb 2025, Singh, 2 Jun 2026).

7. Stacked generalization with transformer models

In ensemble learning, transformer stacking refers to stacked generalization rather than depth. A set of base learners produces probabilities or logits, and a meta-learner is trained on those outputs to produce the final prediction. In the ALTA 2023 shared task on AI-generated text detection, four encoder-only transformers—ALBERT, ELECTRA, RoBERTa, and XLNet—serve as weak learners, each using the final-layer [CLS] token with a single fully connected classifier; their predictions are concatenated and passed to a logistic regression meta-learner (Nguyen et al., 2023). The reported test accuracies are 0.9311 for ELECTRA, 0.9361 for XLNet, 0.9567 for ALBERT, 0.9572 for RoBERTa, and 0.9694 for the ensemble, while the official shared-task test-set score is 0.9555 (Nguyen et al., 2023).

A similar framework is used for multi-class customer review classification, but with a transformer as the meta-learner. BERT, ELECTRA, and DistilBERT are fine-tuned as base learners, their probability vectors are concatenated, and RoBERTa acts as the meta-level classifier (Krishnan, 2023). On the UAE Ministry of Economy Customer Satisfaction dataset, the supplied results report accuracy 0.90, precision 0.91, recall 0.88, and loss 0.21 for stacking, compared with 0.87, 0.88, 0.85, and 0.29 for RoBERTa alone (Krishnan, 2023). On OpSpam, the reported stacked results are accuracy 0.94, precision 0.94, recall 0.93, and loss 0.09, above the best single-transformer accuracy of 0.92 from RoBERTa (Krishnan, 2023).

The ensemble sense of stacking also extends beyond pure-transformer collections. In structured-data churn prediction, an FT-Transformer is combined with XGBoost by out-of-fold stacking and a logistic regression meta-learner,

$$h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),$$7

with calibration-aware training (Roy et al., 26 May 2026). The reported hybrid results are 62.10% F1, 0.861 AUC-ROC, and 0.647 PR-AUC, outperforming the MLP baseline by 3.37 F1 points and 0.027 AUC, and achieving the lowest reported ECE at 0.038 (Roy et al., 26 May 2026). The accompanying ablation reports MLP F1 58.73, XGBoost F1 58.21, FT-Transformer F1 61.00, simple average 61.45, and full stacking 62.10 (Roy et al., 26 May 2026).

In medical imaging, TT-Stack uses seven lightweight vision transformers—RepViT, DaViT, EfficientViT, MobileViT, FasterViT, MViT, and PVT v2—as base models, concatenates their 2-class logits into a 14-dimensional vector, and applies logistic regression as the meta-learner (Osman et al., 1 Dec 2025). The supplied evaluation reports 99.33% accuracy, 100% precision, 96% recall, 97.96% F1-score, and 99.97% ROC-AUC for the ensemble, while EfficientViT and PVT v2 are the best individual backbones with 99.33% validation accuracy, 97.96% F1-score, and 1.000 ROC-AUC (Osman et al., 1 Dec 2025).

Across these settings, ensemble stacking exploits diversity in pretraining regimes, backbone architectures, or inductive biases. This suggests that “transformer stacking” in the ensemble sense is best understood as learned error correction across heterogeneous predictors, rather than as a property of transformer depth itself.