DeepLoop: Depth Scaling for Looped Transformers

Abstract: Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient $κ_R$. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from $1/4$ to $1/2$ as loop count grows at fixed physical depth. The resulting method, \textbf{DeepLoop}, keeps the Post-LN DeepNorm architecture and sets $α=(2N)^{1/2}$ and $β=(8N)^{-1/2}$ for unrolled depth $N$. On GPT-style looped LLMs at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.

• The paper presents DeepLoop, a new analytic scaling rule for looped transformers that stabilizes training by adjusting residual and initialization parameters.
• It details a theoretical framework incorporating the visit-alignment coefficient to account for gradient sharing across recurrent layers, ensuring robust optimization.
• Empirical results on language modeling and downstream tasks validate that DeepLoop outperforms standard parameterizations, especially at increased recurrent depths.

DeepLoop: A Loop-Aware Residual Parameterization for Looped Transformers

Motivation and Problem Formulation

Looped Transformers extend standard architectures by decoupling depth from parameter count, repeatedly applying a compact set of $K$ physical blocks for $R$ rounds and achieving an effective depth of $N=KR$. This paradigm enables parameter-efficient scaling through sequential computation, relating closely to ideas in Universal Transformers, ALBERT, and recent recurrent-depth or test-time-compute models. However, scaling recurrent depth in this manner necessitates a reconsideration of residual-scaling mechanisms, as parameter sharing introduces novel coupling between forward and backward passes that are inapplicable to traditional untied depth analyses.

Standard residual-scaling theory (e.g., DeepNorm) ensures optimization stability in deep Transformers by selecting skip connection and residual-branch initialization scales $(\alpha,\beta)$, given $M=2N$ residual sublayer applications. The untied assumption, however, presumes mutually independent updates across all depthwise sublayers. In looped models, each parameter is shared across $R$ visits, aggregating gradients and being read multiple times in subsequent forward passes. Consequently, the first-order effect of parameter updates is governed not simply by nominal layer count, but by both the structure of parameter sharing and the degree of alignment between per-visit gradient directions.

The paper formalizes this with a visit-alignment coefficient $\kappa_R$, refining the first-order stability condition from $M(\beta/\alpha)^2=O(1)$ to $M\kappa_R(\beta/\alpha)^2=O(1)$. In the aligned regime ($\kappa_R=O(R)$), the exponent in the scaling rule must increase from $R$0 (DeepNorm) to $R$1 as $R$2 increases at fixed $R$3. The resulting prescription, DeepLoop, proposes a universal, analytic scaling for looped Transformers: $R$4, $R$5.

(Figure 1)

Figure 1: Overview of the DeepLoop framework. (a) The model shares $R$6 physical blocks, (b) which are unrolled for $R$7 rounds, giving an effective depth of $R$8, (c) with each residual sublayer applying RMSNorm and scaled skip/branch paths, (d) and DeepLoop specifying loop-aware analytic scaling.

Formal Analysis of Looped Depth Effects

Let $R$9 and $N=KR$0 represent, respectively, the per-visit gradient and linearized output sensitivity for visit $N=KR$1 to physical sublayer $N=KR$2. For untied architectures, the cumulative effect sums over $N=KR$3 independent terms, yielding stability when $N=KR$4 under local $N=KR$5 sensitivities [wang2024deepnet].

When parameters are tied across looped visits, each parameter accumulates gradients from all visits, and all $N=KR$6 visits in subsequent steps are influenced by the same shared update. The perturbation therefore scales with $N=KR$7 (capturing alignment across visits): fully decorrelated visits recover the original untied scaling ($N=KR$8), while fully aligned visits, the conservative regime considered here, require $N=KR$9, amplifying the effect of shared updates.

This yields a loop-aware stability threshold that, for the scaling family $(\alpha,\beta)$0, $(\alpha,\beta)$1, is satisfied when $(\alpha,\beta)$2, where $(\alpha,\beta)$3 controls cross-round correlation ($(\alpha,\beta)$4: decorrelated; $(\alpha,\beta)$5: fully aligned). DeepLoop sets $(\alpha,\beta)$6, ensuring stability in the worst-case aligned regime at any practical loop count.

Empirical Validation

Language Modeling: Validation Loss

DeepLoop and a baseline (tied-only, $(\alpha,\beta)$7, $(\alpha,\beta)$8) are tested on GPT-2 small and medium variants, trained on FineWeb-Edu for 50B tokens. At $(\alpha,\beta)$9, both configurations perform identically—demonstrating parameterization neutrality when no sharing occurs. For all $M=2N$0, DeepLoop consistently outperforms baseline in final validation loss, with the gap increasing with $M=2N$1, confirming theoretical predictions.

Figure 2: Final validation loss at step 100,000 on FineWeb-Edu 50BT (GPT-2 small/medium) for various loop counts. DeepLoop's advantage increases with recurrent depth.

Downstream Generalization: LM Evaluation Suite

On the lm-evaluation-harness suite (arc_challenge, arc_easy, hellaswag, etc.), DeepLoop achieves superior or equal average zero- and one-shot accuracies at $M=2N$2, with per-task improvements on seven out of eight tasks at $M=2N$3. These results, robust across scales, demonstrate that loop-aware scaling not only boosts language modeling but robustly transfers to downstream tasks.

Figure 3: Per-task 1-shot accuracy versus loop count $M=2N$4 for base vs DeepLoop (GPT-2 medium). DeepLoop achieves consistently higher accuracy at high $M=2N$5 on most tasks.

Figure 4: Eight-task 1-shot average accuracy versus $M=2N$6 (GPT-2 small), confirming DeepLoop's monotonic performance gains with increased recurrent depth.

Figure 5: Per-task 1-shot accuracy vs $M=2N$7 for base and DeepLoop (GPT-2 small); DeepLoop outperforms or matches across tasks as depth increases.

$M=2N$8-Sweep: Empirical Stability Threshold

A direct grid search over exponent $M=2N$9 in the DeepLoop scaling rule (at fixed $R$0) confirms a sharp training-stability phase transition near $R$1. All seeds diverge for $R$2; training is reliable only for $R$3. Thus, $R$4 is the empirically minimal stable exponent, matching the theoretical bound.

Figure 6: Validation loss snapshot at step 2000 on FineWeb-Edu 50BT (GPT-2 small, $R$5), sweeping exponent $R$6. Training is consistently stable for $R$7.

Hierarchical Looped Transformers / Reasoners

Extending the analysis, DeepLoop is successfully applied to hierarchical recurrent architectures such as the Hierarchical Reasoning Model (HRM) on ARC-AGI-1. Here, the same loop-aware scaling improves the headline two-vote accuracy by $R$8 points over vanilla HRM, with gains spanning all vote budgets. This supports the claim that $R$9 is required for aligned inner modules operating at fixed physical depth, even with truncated training gradients.

Practical and Theoretical Implications

DeepLoop demonstrates that correct residual and initialization scaling is essential when parameter sharing is employed across circuitous or recurrent depth in Transformers. The analytic scaling rule

$\kappa_R$0

is plug-and-play for any Post-LN looped backbone and robustly stabilizes very deep unrolled computation without architectural modification or new hyperparameters.

Theoretically, the visit-alignment coefficient $\kappa_R$1 elegantly captures the risk of over-aggressive parameterization in the presence of parameter tying. Empirically, the conservative choice $\kappa_R$2 is safe for a wide range of architectures, loop depths, and downstream evaluations, supporting its adoption as a default, loop-aware parameterization. The analysis exposes the importance of not only counting nominal layers but critically modeling how effective depth is realized.

Future Directions

• Measuring Alignment: Directly monitoring $\kappa_R$3 or empirically quantifying inter-visit gradient alignment would provide further guidance for adaptive or less conservative scaling.
• Optimizing for Decorrelation: If visit decorrelation can be induced or detected, it may be possible to interpolate exponents $\kappa_R$4 between the DeepNorm ($\kappa_R$5) and DeepLoop ($\kappa_R$6) regimes, trading off stability and learning signal.
• Scale and Architecture Extensions: Investigation into the behavior at larger $\kappa_R$7, scale, varying normalization strategies, and implications for hyperparameter transfer (as in $\kappa_R$8P/Depth-$\kappa_R$9P) would further elucidate the universality of the approach.

Conclusion

DeepLoop offers a principled and empirically validated residual parameterization for looped Transformers, ensuring robust training and improved generalization wherever residual-branch weight sharing is employed. The loop-aware analysis and its sharp stability boundary exemplify how careful theoretical modeling of parameter interactions is critical to advancing efficient and scalable deep sequence models (2607.13491).