Impact of recurrent depth on transformer performance

Determine how recurrent depth affects the performance of transformer-based language models by rigorously characterizing the relationship between the number of recurrent iterations and model performance across tasks and conditions.

Background

The work situates depth-recurrence within broader research on depth in transformers and latent reasoning. Although many studies examine depth effects in transformers, a clear understanding of how recurrent depth specifically influences performance is not established.

Clarifying the performance implications of recurrence depth would inform architectures and training strategies for models that exploit test-time compute by looping internal layers.

References

Many works study the impact of depth for transformers both theoretically and practically \citep{levine2020depth,merrill2022saturated,mcleish2025gemstones,zuo2025falcon,merrill2025little,csordas2025language}, it is still an open question how recurrent depth impacts the performance of transformers.

— Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence  (2511.07384 - McLeish et al., 10 Nov 2025) in Appendix A, Extended Related Works

Applying such interventions during generation would require evaluating their effects on subsequent states, answer quality, and computation cost. The present measurements identify recoverable losses within individual updates; how these losses accumulate over a recurrent trajectory remains to be established.

— Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers  (2609.16665 - Guo et al., 15 Sep 2026) in Section Limitations, Appendix

We have not established whether training with sampled depth from initialization avoids the peak performance penalty.

— Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models  (2609.19934 - Dau et al., 17 Sep 2026) in Section 5, “Future Work,” subsection “Pretraining with Sampled Depth from Scratch”

Both checkpoints evaluated here are substantially undertrained relative to modern standards; hence, the conclusion that distinct iterations contribute near zero may reflect the training phase rather than architectural limits.

— Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models  (2609.19934 - Dau et al., 17 Sep 2026) in Section 5, “Future Work,” subsection “Does Depth Utilization Emerge Over Training?”

We have yet to study an intermediate approach in which each stack is recirculated exactly $r$ times.

— Recirculation  (2608.17981 - Mozer et al., 18 Aug 2026) in Section “Limitations and future directions,” item “Multiple recirculation iterations”

Each data point in Figure~\ref{fig:compute} is a model that is trained separately, so these results do not establish whether an already trained model benefits from additional inference passes.

— LAST: Looped Audio Spectrogram Transformer  (2610.01926 - Al-Tahan et al., 1 Oct 2026) in Section 4.1, Accuracy and computation

Future work can further study how such latent trajectories develop across reasoning depth and whether different stages of computation benefit from different prediction horizons or levels of abstraction.

— Latent JEPA: Abstract Future Prediction for Latent Reasoning in Chemistry  (2610.01947 - Zhao et al., 1 Oct 2026) in Discussion, Section 5.3