---
title: Transformer Stacking Overview
url: https://www.emergentmind.com/topics/transformer-stacking
type: topic
---

# Transformer Stacking Overview

Transformer stacking denotes a family of constructions in which transformers are composed across depth, across training stages, across auxiliary memory structures, or across model outputs. In the narrow architectural sense, it refers to the canonical composition of encoder or decoder blocks into a deep network. In adjacent literatures, the same term also covers cross-layer fusion, recurrent parameter sharing, gradual depth growth during pretraining, explicit differentiable stack mechanisms inserted into or between transformer layers, and stacked generalization in which multiple transformer predictors are combined by a meta-learner. Formal-language studies add a mechanistic sense: transformers trained on counter languages can develop internal representations consistent with stack structure, and those representations can be causally necessary for correct prediction [2207.14467] [2405.04515] [2606.03398].

## 1. Scope and terminological variants

The supplied literature uses “stacking” for several distinct but related operations.

| Sense of stacking | Core mechanism | Representative work |
|---|---|---|
| Vertical depth composition | Multiple transformer layers are composed sequentially; capabilities depend on the number of attention layers | [2404.01601] |
| Cross-layer grouping and fusion | Adjacent encoder or decoder layers are grouped and fused instead of using only the top layer | [2207.14467] |
| Recurrent or shared-parameter stacking | A single layer is reused across depth, or most parameters are shared with small layer-private residuals | [2106.10002], [2403.04303] |
| Stagewise growth during training | A smaller model is expanded into a deeper model during pretraining | [2409.19044] |
| Explicit stack-structured augmentation | Differentiable push, pop, and no-op mechanisms are inserted into attention or between layers | [2405.04515], [2507.15343] |
| Parser-state or nested transformer stacking | Specialized heads or cascaded transformers model stack/buffer state or spatial-temporal structure | [2010.10669], [2212.14538] |
| Ensemble stacking | Multiple transformer models, or a transformer plus a non-transformer model, are combined by a meta-learner | [2310.18906], [2606.07582] |
| Emergent latent stacks | Hidden states linearly encode stack depth, and this encoding can be causally necessary | [2502.01432], [2606.03398] |

A recurrent source of ambiguity is that some papers use “stacking” to mean deeper composition of layers, whereas others use it to mean a literal stack memory or a stacked ensemble. The overlap is not merely terminological. Across these settings, the central issue is how information is preserved, transformed, or selectively exposed across multiple computational stages.

## 2. Depth as compositional computation

Case studies on synthetic sequence tasks isolate a direct role for depth in the computational expressivity of transformers. A transformer with only one attention layer can excel in memorization but falls short in reasoning, generalization, and contextual generalization; two attention layers suffice for reasoning and generalization; contextual generalization may necessitate three attention layers [2404.01601]. The same work identifies a class of simple operations that a single attention layer can execute—copying, parsing, matching, and mapping—and argues that more complex behaviors arise by stacking layers so that these primitive operations compose across depth.

This perspective makes layer stacking a mechanism for multi-stage computation rather than a purely statistical increase in capacity. In the synthetic constructions, a two-layer model can solve in-context question answering by a copying operation followed by a matching operation, while a three-layer model can combine parsing, copying, and matching for contextual generalization [2404.01601]. A plausible implication is that depth is serving as an execution trace over intermediate representations, not simply as a larger feature extractor.

The optimization side is more delicate. Signal-propagation analysis shows that stacking self-attention layers can result in rank collapse of token representations at initialization, with token correlations converging toward perfect alignment as depth increases [2206.03126]. In the extreme rank-1 case, gradients with respect to query and key parameters vanish. The same analysis shows that appropriate depth-dependent scaling of the residual branches can prevent this pathology; with residual weights scaled as $O(1/\sqrt{L})$, token correlations remain nontrivially dependent on the input rather than collapsing to $1$ in the infinite-depth limit [2206.03126].

Taken together, these results indicate that stacking is computationally consequential in two distinct senses: it enables composition of primitive operations, but it also changes the forward and backward signal geometry in ways that can either support or obstruct training.

## 3. Cross-layer fusion and parameter-efficient deep stacks

Standard transformer usage in neural machine translation typically stacks encoder and decoder layers and then exploits only the top-layer representation, effectively treating lower layers as trivial or redundant [2207.14467]. GTrans modifies this assumption by dividing encoder and decoder layers into adjacent groups and fusing the resulting group representations through learned weights. For the encoder, the fused representation is
$$
h^{f}_{e} = \Phi_e(H_e)=\text{LN}\left(\frac{1}{M}\sum_{i=1}^{M}\sigma(w_i^e) h_{\alpha_i}^e\right),
$$
with $M=\lceil L_e/T_e\rceil$ and $\alpha_i=\min(iT_e,L_e)$; analogous grouping and fusion are used in the decoder, including probability fusion across decoder groups for prediction [2207.14467]. The formulation includes two limiting cases: $T_e=1$ yields dense fusion across all layers, while $T_e=L_e$ recovers the vanilla transformer. Reported experiments indicate that the method can be scaled up to 60 encoder layers and 36 decoder layers [2207.14467].

A more aggressive reduction of parameter redundancy appears in recurrent stacking. Instead of using distinct parameters for each layer, the same layer is recursively applied:
$$
Y = L_1(L_1(L_1(L_1(L_1(L_1(X)))))).
$$
In machine translation, a 6-recurrence recurrently stacked model has the parameter count of a 1-layer vanilla model, and the reported parameter counts for 6-layer vanilla versus 6-recurrence recurrent stacking are 69.8M versus 30.4M on GCP Ja→En, 57.7M versus 20.2M on KFTT Ja→En, and 94.4M versus 57.7M on ASPEC Ja→En [2106.10002]. The same study reports that a 6-layer recurrently stacked model significantly outperforms a 1-layer vanilla model with identical parameter count; on ASPEC Ja→En, the scores are BLEU=27.20 versus BLEU=23.28, while a 6-layer vanilla model reaches BLEU=28.77 [2106.10002]. Transfer learning and sequence-level knowledge distillation are reported to compensate for the performance drop induced by direct training of recurrently stacked models [2106.10002].

Depth-wise LSTMs offer a different answer to the same problem of inter-layer aggregation. Instead of residual addition,
```text
x_{l+1} = x_l + SubLayer(x_l),
```
the sequence of stacked layers is treated as a “vertical” sequence and connected with LSTM gating. Layer normalization and feed-forward computation are absorbed into the depth-wise LSTM, so that attention sub-layers remain unchanged while inter-layer communication becomes gated and recurrent [2007.06257]. On WMT 14, the reported BLEU scores move from 27.55 to 28.53 for Transformer Base on En-De, from 39.54 to 40.10 on En-Fr, from 28.83 to 29.58 for Transformer Big on En-De, and from 41.92 to 43.11 on En-Fr [2007.06257]. The paper attributes the gains to selective aggregation and improved signal propagation through depth.

LORS moves further toward parameter-efficient stacking by decomposing each layer parameter matrix as
$$
W_i = W^{\text{shared}} + W^{\text{private}_i},
\qquad
W^{\text{private}_i} = \sum_{k=1}^{K} B_{ik}A_{ik}.
$$
Most parameters are shared across layers, while a small low-rank residual remains layer-specific [2403.04303]. In the stacked decoder of AdaMixer, the reported decoder parameter count drops from 110M to 35M for 6 layers, a 70% reduction, while AP changes from 42.7 to 42.6 under a 1x schedule and from 47.0 to 47.6 under a 3x schedule in the reported MS COCO experiments [2403.04303]. This suggests that a large fraction of what conventional layer stacking buys can be retained through shared structure plus low-rank layer-specific adaptation.

## 4. Gradual depth growth as a training strategy

A distinct meaning of stacking appears in training curricula that grow model depth stage by stage. In gradual stacking, training is split into stages, and a smaller model from one stage initializes a deeper model in the next [2409.19044]. The stagewise training budget is allocated according to
$$
T_i \propto i^\alpha,
$$
with schedules using $\alpha=1,2,3$ considered in the reported experiments [2409.19044].

MIDAS modifies gradual stacking by replicating the middle block rather than the final block when the model is expanded. Its growth operator is given as
$$
G_{\text{midas}}(f,b)=f_{1,b}\circ\cdots\circ
f_{\lceil n/2\rceil,b}\circ f_{\lceil n/2\rceil,b}\circ\cdots\circ f_{n,b},
$$
where the middle block is duplicated during expansion [2409.19044]. The method is reported to speed up language model training by up to 40%, and with the Prop2 schedule it matches baseline validation log perplexity with about 24% less compute [2409.19044].

The notable claim is not only efficiency but inductive bias. On 1B-parameter models, the reported all-tasks average increases from 24.0% for baseline training to 26.8% for MIDAS, open-book QA rises from 33.3% to 36.3%, and math word problems rise from 23.5% to 29.0% [2409.19044]. Similar trends are reported at 2B and 8B scale. On fine-tuned GSM8K, the reported 2B results are 5.3% for baseline versus 10.4% for MIDAS, and the 8B results are 12.3% versus 15.2% [2409.19044].

The paper further introduces reasoning primitives—Induction Copying, Variable Assignment, and Pre-School Math—to probe the effect directly. In the selected 5-shot results provided, MIDAS reaches 24.3% versus 14.9% on Copying, 28.3% versus 21.6% on Variable Assignment depth-1, 45.0% versus 23.9% after fine-tuning on Variable Assignment depth-2, and 69.5% versus 62.0% on Pre-School Math with calculator [2409.19044]. The authors conjecture that the effect is linked to looped or weight-shared models: duplicated middle layers remain highly similar after training, and MIDAS produces a cosine-similarity structure closer to looped models than end-stacking growth does [2409.19044].

## 5. Explicit stack-structured augmentations

Another research line uses “stacking” literally by adding stack-like state to the architecture. In “stack attention,” a differentiable stack over positions is maintained with soft push, skip, and pop operations. At position $i$, the stack distribution is updated as
$$
a_i = \alpha_{i,\text{push}} a_i^{(\text{push})}
      + \alpha_{i,\text{skip}} a_i^{(\text{skip})}
      + \alpha_{i,\text{pop}} a_i^{(\text{pop})},
$$
and the stack attention output is
$$
(\text{StackAttention}(H))_{:,i}=\sum_{n=0}^{N} a_i(n) h_n.
$$
The mechanism is inserted as an additional sub-layer after self-attention and feed-forward computation in each transformer block [2405.04515]. Reported results show that the resulting stack transformer solves Reverse String at 100% and Stack Manipulation at about 93%, while both vanilla transformers and the stack-augmented model remain substantially weaker on Modular Arithmetic and Solve Equation [2405.04515]. The stated conclusion is that the mechanism enables the transformer to model some, but not all, deterministic context-free languages.

StackTrans introduces differentiable hidden state stacks between transformer layers rather than modifying attention itself [2507.15343]. For each hidden state, the model predicts soft push, pop, and no-op actions, updates a fixed-size stack, and then performs a global read over the stack before passing the representation to the next layer. The architecture is explicitly designed to remain compatible with existing frameworks like flash-attention and is reported to scale from 360M to 7B parameters [2507.15343]. On formal-language benchmarks, the supplied summary states that StackTrans outperforms standard transformers by 30%+ on regular expression and deterministic context-free tasks, achieves near-100% test accuracy in most cases, generalizes across sequence lengths, and incurs only about 10% extra training and inference time with the low-rank multi-head stack design [2507.15343].

Transition-based parsing yields another explicit use of stack structure. Stack-Transformers modify decoder cross-attention so that one head attends only to the current stack and one head attends only to the current buffer, with masks changing as parsing actions update parser state [2010.10669]. The reported development-set results on Penn Treebank and AMR2.0 move from LAS 92.0 to 94.1 and from Smatch 77.7 to 79.5 when using one stack head and one buffer head [2010.10669]. The paper emphasizes that explicit state modeling matters most for smaller models and limited training data.

A more general nested composition appears in reinforcement learning backbones. Transformer in Transformer cascades an inner transformer, which processes a single observation, with an outer causal transformer, which processes the history of observation embeddings, thereby extracting spatial-temporal representations in a pure-transformer backbone [2212.14538]. Here stacking does not denote a pushdown stack, but it does denote a hierarchical composition of transformer modules across two levels of abstraction.

## 6. Emergent stack representations in hidden states

Formal-language analysis has provided a mechanistic notion of transformer stacking that is neither architectural nor ensemble-based: the latent state of the model can behave as if it contains a stack. In counter-language experiments, transformers are trained on Dyck-1 and Shuffle-$k$ languages, which can be formulated in terms of stacks whose depths correspond to counter values [2502.01432]. The reported model is an encoder-only transformer with 1 layer, 4 attention heads, hidden size 64, embedding dimension 32, and causal masking; it is trained on next-token prediction with mean squared error between predicted and true valid-token vectors [2502.01432].

Probing analyses show that even linear probes can recover stack depths from the hidden states. The latent variable is the stack depth at each token position, and selectivity is defined as the difference between probe accuracy on true labels and on randomized control labels [2502.01432]. The supplied summary reports high probing accuracy for Dyck-1 and even higher values for Shuffle-$k$ as $k$ increases, in both learned models and an algorithmically compiled transformer produced by Tracr [2502.01432]. The key limitation stated in that work is that probing shows presence of information, not whether the model uses that information causally.

The subsequent causal study closes that gap. In counter-language modeling with an encoder-only, single-layer transformer of $d_{\text{model}}=64$ trained to perfect accuracy on Shuffle-$k$ for $k\in\{2,4,6,8\}$, linear probes are trained to predict stack depth from hidden states
$$
f(h)=Wh+b,
$$
with cross-entropy loss, and the weight matrix is decomposed by SVD,
$$
W = U\Sigma V^T,
$$
to extract the top right singular vector $w=V[0]$ as a principal stack direction [2606.03398]. The hidden state is then ablated along that direction before decoding,
$$
h' = h - \alpha (w \cdot h) w,
\qquad \alpha\in[0,1].
$$
According to the supplied findings, targeted ablation causes positional accuracy to drop substantially as $\alpha$ increases and makes sequential accuracy collapse almost to zero even for moderate $\alpha$, whereas ablation along random directions has no significant effect [2606.03398]. The reported interpretation is that the transformer does not merely encode stack depth correlates; it directly relies on this representation for prediction, and there are no alternate or redundant pathways compensating for the removed information [2606.03398].

A common misconception in this area is that probe recoverability is sufficient evidence of mechanism. The two papers jointly distinguish these claims: the earlier work establishes recoverable stack-like encoding, while the later work establishes causal necessity under targeted intervention [2502.01432] [2606.03398].

## 7. Stacked generalization with transformer models

In ensemble learning, transformer stacking refers to stacked generalization rather than depth. A set of base learners produces probabilities or logits, and a meta-learner is trained on those outputs to produce the final prediction. In the ALTA 2023 shared task on AI-generated text detection, four encoder-only transformers—ALBERT, ELECTRA, RoBERTa, and XLNet—serve as weak learners, each using the final-layer [CLS] token with a single fully connected classifier; their predictions are concatenated and passed to a logistic regression meta-learner [2310.18906]. The reported test accuracies are 0.9311 for ELECTRA, 0.9361 for XLNet, 0.9567 for ALBERT, 0.9572 for RoBERTa, and 0.9694 for the ensemble, while the official shared-task test-set score is 0.9555 [2310.18906].

A similar framework is used for multi-class customer review classification, but with a transformer as the meta-learner. BERT, ELECTRA, and DistilBERT are fine-tuned as base learners, their probability vectors are concatenated, and RoBERTa acts as the meta-level classifier [2308.11519]. On the UAE Ministry of Economy Customer Satisfaction dataset, the supplied results report accuracy 0.90, precision 0.91, recall 0.88, and loss 0.21 for stacking, compared with 0.87, 0.88, 0.85, and 0.29 for RoBERTa alone [2308.11519]. On OpSpam, the reported stacked results are accuracy 0.94, precision 0.94, recall 0.93, and loss 0.09, above the best single-transformer accuracy of 0.92 from RoBERTa [2308.11519].

The ensemble sense of stacking also extends beyond pure-transformer collections. In structured-data churn prediction, an FT-Transformer is combined with XGBoost by out-of-fold stacking and a logistic regression meta-learner,
$$
\hat{p}_{stack}=\sigma(w_0 + w_1 p_{FT} + w_2 p_{XGB}),
$$
with calibration-aware training [2606.07582]. The reported hybrid results are 62.10% F1, 0.861 AUC-ROC, and 0.647 PR-AUC, outperforming the MLP baseline by 3.37 F1 points and 0.027 AUC, and achieving the lowest reported ECE at 0.038 [2606.07582]. The accompanying ablation reports MLP F1 58.73, XGBoost F1 58.21, FT-Transformer F1 61.00, simple average 61.45, and full stacking 62.10 [2606.07582].

In medical imaging, TT-Stack uses seven lightweight vision transformers—RepViT, DaViT, EfficientViT, MobileViT, FasterViT, MViT, and PVT v2—as base models, concatenates their 2-class logits into a 14-dimensional vector, and applies logistic regression as the meta-learner [2512.02091]. The supplied evaluation reports 99.33% accuracy, 100% precision, 96% recall, 97.96% F1-score, and 99.97% ROC-AUC for the ensemble, while EfficientViT and PVT v2 are the best individual backbones with 99.33% validation accuracy, 97.96% F1-score, and 1.000 ROC-AUC [2512.02091].

Across these settings, ensemble stacking exploits diversity in pretraining regimes, backbone architectures, or inductive biases. This suggests that “transformer stacking” in the ensemble sense is best understood as learned error correction across heterogeneous predictors, rather than as a property of transformer depth itself.

Source: https://www.emergentmind.com/topics/transformer-stacking