Recursive-Transformer Architectures

• Recursive-Transformer is a class of architectures that iteratively reuses a single Transformer module for refinement and deeper representation.
• They employ diverse recursive patterns, such as shared-parameter depth recursion and output-conditioned iterative refinement, to enhance model performance.
• These models improve parameter efficiency and allow dynamic composition, though they can trade off compute cost and require stabilization mechanisms.

“Recursive-Transformer” denotes a family of Transformer architectures in which attention-based computation is reused through iteration, refinement, or explicit structural composition rather than being executed only once through a fixed stack of distinct layers. In recent literature, the term is not standardized: it has been used for weight-shared depth recursion in language and vision models, feedback-based refinement loops that reuse previous outputs, tree-structured composition over latent parses, and recursive estimation pipelines in causal modeling (Shen et al., 2021, Gao et al., 2022, Zhang et al., 2023, Hu et al., 2021). A plausible implication is that recursion is best understood as an architectural primitive—reusing a Transformer or Transformer-like operator across steps—rather than as a single canonical model class.

1. Terminological scope

Across the cited literature, “recursive Transformer” is used for several distinct but related constructions. In LBNet, recursion is explicitly described as “not a recurrent connection across time steps of a sequence,” but as depth-wise reuse of the same Transformer module with shared parameters (Gao et al., 2022). By contrast, EvoPose feeds previous 3D pose predictions back into a transformer-style refinement block, RNGTr recursively refines discrete graphs, and R2D2 performs recursive composition over a differentiable binary tree (Zhang et al., 2023, Mohammadshahi et al., 2020, Hu et al., 2021).

Pattern	Representative formulation	Representative papers
Shared-parameter depth recursion	Reuse the same block across depth steps	LBNet (Gao et al., 2022), SReT (Shen et al., 2021), ReSSFormer (You et al., 2 Oct 2025), Relaxed Recursive Transformers (Bae et al., 2024)
Output-conditioned iterative refinement	Re-feed previous predictions into a Transformer refinement step	EvoPose (Zhang et al., 2023), RNGTr (Mohammadshahi et al., 2020), RecursiveVLM (Xu et al., 9 Feb 2026)
Structured recursion	Recurse over latent trees, event iterations, or recursive causal residuals	R2D2 (Hu et al., 2021), UTHP (Zhang et al., 2021), TERRA (Shi et al., 25 Oct 2025), STAIRS-Former (Jeon et al., 12 Mar 2026)
Memory- or resolution-augmented looping	Shared core plus explicit memory or multi-resolution schedule	MeSH (Yu et al., 9 Oct 2025), SpiralFormer (Yu et al., 12 Feb 2026), SoftMoR (Lee et al., 1 Jul 2026), Ouroboros (Jaber et al., 2 Apr 2026)

This terminological breadth corrects a common misconception: “recursive” does not necessarily mean temporal recurrence in the RNN sense, nor does it always mean universal parameter tying across all layers. In the literature summarized here, recursion may refer to depth, outputs, structure, memory state, or latent estimation procedures, depending on the task formulation.

2. Recurrence over depth and shared-parameter loops

The most canonical usage defines recursion as recurrent-in-depth application of a shared Transformer block. LBNet formalizes this directly as

$$f_{TM}^{\circlearrowright}(F) = \underbrace{f_{TM}(f_{TM}(\dots f_{TM}(F)\dots))}_{S\ \text{times}},$$

with all applications sharing the same weights, and places two such recursively applied Transformer modules after a Symmetric CNN in single-image super-resolution (Gao et al., 2022). SReT applies the same idea to vision transformers, reusing MHSA+FFN within each recursive block and thereby increasing logical depth without increasing the number of distinct parameter sets (Shen et al., 2021).

Later looped LLMs generalize this depth-sharing pattern. ReSSFormer defines the recurrent core as

$H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$

where the same Transformer-like block is reused over fixed-depth iterations and coupled to hierarchical memory (You et al., 2 Oct 2025). Relaxed Recursive Transformers make the same principle explicit at pretrained LLM scale by cycling through a reduced set of unique layers,

$$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$

so that a model with $L$ logical layers can be represented by only $L' = L/B$ distinct layers reused $B$ times (Bae et al., 2024).

Within this lineage, recursion primarily serves parameter efficiency and effective depth scaling. SReT reports that its sliced recursive design can build transformers with “more than 100 or even 1000 shared layers” while keeping a compact 13~15M parameter range (Shen et al., 2021). Relaxed Recursive Transformers similarly treat recursion as a conversion mechanism for pretrained LLMs: a standard stack is compressed into a smaller looped core, then partially re-specialized through low-rank residuals (Bae et al., 2024). This suggests that depth-wise recursion is now a distinct design axis alongside width, expert routing, and context length.

3. Output-conditioned iterative refinement

A second meaning of recursive Transformer emphasizes feedback from a model’s own predictions. EvoPose is explicit on this point: the “Recursive Transformer” part is its Recursive Refinement module, which initializes $X_0=\mathbf{0}$, refines features by

$$S_t^r = \mathrm{RSA}(S_t^e, P^e) - \mathrm{MLP}(X_{t-1}) + S_t^e,$$

and regresses a new 3D pose

$X_t = \mathrm{RegHead}(S_t^r)$

for a fixed number of rounds, with $L=2$ in experiments (Zhang et al., 2023). Here recursion is not merely an outer loop around a static predictor; the same structure-aware attention machinery is reapplied while conditioned on the previous 3D estimate.

RNGTr instantiates the same idea for graph refinement. Given a sentence and an initial graph $H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$0, it encodes the sentence together with that graph, predicts all edges in parallel, and decodes a new graph $H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$1; recursion occurs across graph states, while each individual step remains non-autoregressive (Mohammadshahi et al., 2020). This allows a parser to repair global inconsistencies over successive passes without abandoning parallel edge scoring.

RecursiveVLM transfers the pattern to large multimodal models. It loops a shared Transformer decoder across recursion steps $H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$2, but rejects the naive recurrence

$H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$3

because of scale and distribution misalignment between shallow multimodal embeddings and deep hidden states. Instead it constructs the next-step input by aggregating selected intermediate layers through modality-specific Recursive Connectors and trains all steps jointly with a Monotonic Recursion Loss (Xu et al., 9 Feb 2026).

These systems share a common logic: recursion is used to reinterpret a previous estimate, not merely to deepen a representation. A plausible implication is that this form of recursion is closest in spirit to iterative optimization or latent-space self-correction, with Transformer blocks serving as learned update operators.

4. Structured, temporal, and causal recursion

Not all recursive Transformers are looped-depth models. R2D2 realizes recursion over a latent binary tree induced by a differentiable CKY-style chart. Each span cell $H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$4 stores a span representation and probabilities; for each split $H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$5, a Transformer-based composition function

$H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$6

produces a candidate parent vector and merge score, and straight-through Gumbel-Softmax selects a split while preserving differentiability (Hu et al., 2021). The recursive computation is therefore over phrase structure rather than over a flat depth axis.

UTHP uses yet another form: a shared encoder layer is recursively reapplied to asynchronous event sequences, optionally with Adaptive Computation Time so that different positions can halt after different numbers of iterations (Zhang et al., 2021). The model remains a Transformer Hawkes process in its output intensity parameterization, but replaces a fixed stack of distinct layers with universal, recursively reused depth.

TERRA embeds a Transformer inside a recursive causal-estimation procedure. Its recursion is carried by blipped outcomes,

$H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$7

while the Transformer parameterizes propensities, conditional means, and blip functions for longitudinal heterogeneous treatment effect estimation (Shi et al., 25 Oct 2025). The model is “recursive” because the loss and target residuals are constructed backward through time from the model’s own outputs.

STAIRS-Former combines structural and temporal recursion. Within layer $H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$8, it updates recursive states by

$H^{(t+1)} = \mathrm{Block}(H^{(t)}, M^{(t)}),$9

and then recycles low- and high-level history tokens across environment steps in offline multi-task multi-agent reinforcement learning (Jeon et al., 12 Mar 2026). In this setting, recursion is interleaved with temporal memory updates rather than confined to a single feed-forward pass.

Taken together, these examples show that the term extends beyond looped depth-sharing to include recursive composition over trees, adaptive iteration over event histories, and recursive residual constructions in structured estimation problems.

5. Mechanisms for specialization and stability

A recurrent problem in recursive Transformer design is that repeated application of a shared block can collapse toward undifferentiated computation or near-identity mappings. SReT addresses this by inserting non-shared Non-linear Projection Layers,

$$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$0

between recursive applications, explicitly to avoid trivial recursive solutions (Shen et al., 2021). MeSH goes further and diagnoses two pathologies of naive recursion—“undifferentiated computation” and “information overload”—then externalizes persistent state into a memory buffer

$$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$1

with per-iteration write/read routers, so that the shared core is not forced to store long-lived memory, working state, and output features in a single tensor (Yu et al., 9 Oct 2025).

Another strategy is to relax strict weight tying. Relaxed Recursive Transformers preserve a shared core but add depth-specific LoRA residuals, initialized by truncated SVD of the difference between original and shared layers, so that the looped model approximates the original pretrained stack while remaining compact (Bae et al., 2024). Ouroboros pushes this idea into input-conditioned weight generation: a Controller hypernetwork reads the current hidden state, produces per-step diagonal modulation vectors for frozen SVD-initialized LoRA bases, and combines them with gated recurrence and per-step LayerNorm (Jaber et al., 2 Apr 2026). The gate is bias-initialized to roughly 88% state retention, and the paper reports that without gated recurrence recursive layer application makes the model strictly worse (Jaber et al., 2 Apr 2026).

A third strategy is to exploit intermediate recursive states rather than discarding them. SoftMixture-of-Recursions learns token-wise soft weights over all recursion steps and forms the final token representation as a weighted sum of all step outputs, while SpiralFormer compresses the sequence to an iteration-specific resolution $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$2, runs the shared core on the shorter sequence, then upscales and causally right-shifts the resulting token-level update before reintegration (Lee et al., 1 Jul 2026, Yu et al., 12 Feb 2026). This suggests that effective recursive design increasingly depends on explicit mechanisms for step differentiation, state management, or multiscale aggregation rather than on bare parameter reuse.

6. Empirical behavior across domains

Empirical results vary by task, but the strongest outcomes usually appear when recursion is paired with domain-specific inductive structure rather than used as a bare compression device.

System	Domain	Reported outcome
EvoPose	Monocular 3D human pose estimation	Human3.6M ablation: 45.8 MPJPE with SPR+RR vs 57.3 without both (Zhang et al., 2023)
LBNet	Single-image super-resolution	Set5 ×4: 32.23 / 0.8949 with RT vs 32.07 / 0.8929 without RT (Gao et al., 2022)
DRT	Single-image deraining	Rain100L: 37.61 dB with 1.18M parameters; 1.3% of HiNet’s 88.7M (Liang et al., 2022)
ReSSFormer	Long-context language modeling and reasoning	At 4k tokens: 77.8% accuracy, 172 G FLOPs, 95 ms (You et al., 2 Oct 2025)
RecursiveVLM	Large multimodal models	Under Data1: average score 58.86 at step 2 vs 55.31 for the standard Transformer baseline (Xu et al., 9 Feb 2026)
SR‑ViT	ImageNet-1K classification	Increasing recursion depth from 1 to 4 improves top-1 from 79.83% to 82.48% with only 1.7M additional parameters (Lee et al., 1 Jul 2026)

The same literature also reports saturation or failure modes. LBNet finds that recursion improves until $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$3 and then essentially saturates at $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$4 (Gao et al., 2022). DRT reports that the variant without recursion performs worst among its own ablations (Liang et al., 2022). RecursiveVLM shows that naive multimodal recursion can be worse than a non-recursive baseline until connectors and monotonic supervision are added (Xu et al., 9 Feb 2026). SoftMoR reports that naive recursive ViTs improve at $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$5 but can degrade at $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$6 unless intermediate states are softly combined (Lee et al., 1 Jul 2026). These observations collectively indicate that recursive depth is rarely beneficial in isolation.

7. Limitations, ambiguities, and open problems

A first limitation is that recursion often saves parameters without proportionally saving computation or latency. LBNet’s Recursive Transformer changes Mult-Adds only marginally, from $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$7G to $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$8G, but runtime still increases from $$h_t^\ell = f\big(h_t^{\ell-1}; \Phi'_{((\ell-1)\bmod L')+1}\big),$$9s to $L$0s (Gao et al., 2022). SR‑ViT improves ImageNet accuracy as recursion depth increases, yet FLOPs rise from 8.5G at $L$1 to 16.3G at $L$2 for SR‑ViT-S (Lee et al., 1 Jul 2026). Recursive design therefore often trades parameters for compute rather than eliminating compute.

A second limitation is fixed or weakly adaptive depth. EvoPose fixes $L$3 refinement rounds, ReSSFormer fixes $L$4, RecursiveVLM trains with $L$5, and UTHP caps its adaptive recursion with a small maximum iteration count (Zhang et al., 2023, You et al., 2 Oct 2025, Xu et al., 9 Feb 2026, Zhang et al., 2021). Even when ACT-style mechanisms exist, as in UTHP, the broader literature reviewed here still relies predominantly on hand-set recursion budgets rather than learned termination criteria.

A third limitation is domain-specific sensitivity. EvoPose depends on skeleton definition and joint indexing, and its gains are much larger with ground-truth 2D than with noisier detections (Zhang et al., 2023). RecursiveVLM attributes naive multimodal failure to scale and distribution misalignment between deep and shallow states (Xu et al., 9 Feb 2026). Ouroboros improves training-distribution loss substantially, but the Controller does not yet improve held-out loss over the compressed baseline, which the paper attributes to frozen downstream layers (Jaber et al., 2 Apr 2026).

Finally, there is a conceptual ambiguity that remains unresolved: the field does not yet use “recursive Transformer” in a single, stable sense. Some papers mean strict weight-sharing across depth; others mean iterative prediction correction, recursive structure induction, hierarchical memory, or recursive residual estimation. A plausible implication is that future work may converge less on one canonical recursive architecture than on a set of reusable design motifs—shared-depth looping, step-specific modulation, explicit memory, soft aggregation over iterations, and multiresolution scheduling—that can be composed differently across tasks (Yu et al., 9 Oct 2025, Yu et al., 12 Feb 2026).