Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers

Abstract: We study implicit reasoning, i.e. the ability to combine knowledge or rules within a single forward pass. While transformer-based LLMs store substantial factual knowledge and rules, they often fail to compose this knowledge for implicit multi-hop reasoning, suggesting a lack of compositional generalization over their parametric knowledge. To address this limitation, we study recurrent-depth transformers, which enables iterative computation over the same transformer layers. We investigate two compositional generalization challenges under the implicit reasoning scenario: systematic generalization, i.e. combining knowledge that is never used for compositions during training, and depth extrapolation, i.e. generalizing from limited reasoning depth (e.g. training on up to 5-hop) to deeper compositions (e.g. 10-hop). Through controlled studies with models trained from scratch, we show that while vanilla transformers struggle with both generalization challenges, recurrent-depth transformers can effectively make such generalization. For systematic generalization, we find that this ability emerges through a three-stage grokking process, transitioning from memorization to in-distribution generalization and finally to systematic generalization, supported by mechanistic analysis. For depth extrapolation, we show that generalization beyond training depth can be unlocked by scaling inference-time recurrence, with more iterations enabling deeper reasoning. We further study how training strategies affect extrapolation, providing guidance on training recurrent-depth transformers, and identify a key limitation, overthinking, where excessive recurrence degrades predictions and limits generalization to very deep compositions.

• The paper introduces recurrent-depth transformers that address limitations of vanilla models by enabling multi-hop compositional reasoning through looped depth recurrence.
• The methodology employs both fixed and dynamic iteration strategies, along with a zero-initialization scheme, to stabilize training and promote robust out-of-distribution generalization.
• Inference-time recurrence scaling with adaptive halting is shown to boost reasoning capacity while mitigating overthinking, offering a cost-efficient path for complex tasks.

Implicit Compositional Reasoning in Recurrent-Depth Transformers

Introduction

The paper "Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers" (2604.07822) systematically investigates the capacity of recurrent-depth (looped) transformers to perform implicit multi-hop reasoning by internally composing factual and parametric knowledge. The core motivation stems from the persistent failure of standard transformers to generalize compositionally in implicit (single forward pass, no chain-of-thought) settings, despite their parametric storage of extensive factual knowledge. The study articulates and evaluates two out-of-distribution (OOD) reasoning challenges—systematic generalization and depth extrapolation—through a series of controlled synthetic knowledge-graph-based experiments, contrasting recurrent-depth transformers to their vanilla counterparts.

Model Architecture and Training Protocols

The architecture is a decoder-only transformer where a block of $L$ layers is unrolled $R$ times in depth recurrence, yielding an effective depth of $L \times R$ (Figure 1). The embedding layer and LLM head share weights, ensuring parameter efficiency and enabling scaling of computation at inference without retraining. Crucially, a zero-initialization scheme is employed for multi-head attention and feedforward output projections, stabilizing learning dynamics under large recurrent depths—an essential modification observed to improve training stability and gradient flow compared to conventional initializations.

Figure 1: Recurrent-depth transformer architecture; the same stack of transformer layers is looped $R$ times for increased effective depth at fixed parameter count.

Training uses two stopping strategies for recurrent iterations: fixed iteration, where $R$ is held constant, and dynamic iteration, sampling $R$ per example. Dynamic iteration is shown to promote better extrapolation and robustness, as it prevents the model from specializing to a fixed computational budget and aligns more closely with uncertainties in real-world task complexity.

Systematic Generalization and Grokking Dynamics

The systematic generalization task requires the model to combine atomic facts never seen jointly during training—representing pure composition of knowledge rather than memorization or trivial interpolation. Experiments use large knowledge graphs partitioned such that some atomic facts (the OOD set) are unavailable in compositional contexts during training, thus enforcing genuine compositional generalization.

Recurrent-depth transformers substantially outperform vanilla transformers in this setting (Figure 2). Even for minimal recurrence ($R=2$), the model demonstrates non-trivial OOD accuracy, with convergence and generalization improving with additional iterations. A distinctive training dynamic emerges: systematic generalization arises only after a prolonged "grokking" period—starting with memorization, followed by in-distribution (ID) generalization, and culminating in robust OOD systematic generalization. The vanilla transformer fails to reach the third stage, highlighting the necessity of depth recurrence for systematic composition.

Figure 2: OOD accuracy curves for recurrent-depth models; systematic generalization emerges abruptly after the ID generalization phase, uniquely for looped architectures.

Internal representation analysis via logit lens reveals that, for recurrent-depth transformers, bridge and target entity representations responsible for multi-hop chaining become decodable only in the final grokking stage. In contrast, vanilla models display no such mechanistic transition on OOD inputs, corroborating prior work on the limitations of non-recurrent transformers for compositionality.

Depth Extrapolation: Scaling Reasoning Complexity

Depth extrapolation evaluates the model's ability to solve more complex reasoning chains at test time than were encountered during training—i.e., performing $k'$-hop inference after training only on up to $k$-hop compositions. With a curriculum learning protocol and permutation-based knowledge graph construction (to preclude shortcut-based solutions), the authors meticulously track the maximal compositional complexity generalized by each model.

Increasing training-time recurrent iterations $R$ directly elevates the model's learnable recursion depth, corroborating theoretical predictions that reasoning capacity grows with depth but not merely with parameter count (supported by Figure 3). Importantly, fixed $R$0 caps reasoning capacity, while dynamic iteration yields superior extrapolation, particularly when the curriculum covers high hop counts.

Figure 3: Evaluation accuracy across hop complexities and inference-time recurrence; vertical and horizontal markers denote limits of training-time generalization and inference-time computation, respectively.

Inference-time recurrence scaling is shown to unlock further generalization: applying more recurrent iterations at test time than at training extends the model's multi-hop capability, as long as the training regime was sufficiently rich. Adaptive halting, based jointly on KL-divergence and output entropy, further increases the compute-efficiency and prevents unnecessary or detrimental iterations.

Limits: Overthinking and Shortcut Solutions

Despite their extrapolative potential, recurrent-depth transformers are observed to suffer from overthinking—performance degradation with excessive inference-time iteration beyond a complexity-dependent optimum (Figure 4). The phenomenon is linked to declining prediction confidence (logit margin) after reaching the correct answer, indicating model predictions become less robust at deeper recurrences. This effect is less pronounced, but not eliminated, with dynamic iteration strategies.

Figure 4: Average logit margin trajectories across iterations reveal peak confidence is task complexity-dependent and declines with overthinking.

A further essential methodological contribution is the design of permutation-based datasets to eliminate spurious shortcut solutions, which, if not controlled, would artificially inflate the observed compositional generalization by enabling shallow sequence mappings. Activation-patching causal analyses highlight the risks: apparent deep compositionality can result from models exploiting short suffixes of relation sequences (Figure 5). The permutation construction enforces strict one-to-one relation entity mappings, ensuring only genuine multi-step composition leads to successful reasoning.

Figure 5: Causal activation-patching shows prediction heavily depends on trailing relation positions, confirming shortcut-based solutions in non-permuted settings.

Theoretical and Practical Implications

These results have notable implications:

• Architectural Inductive Bias: Recurrent-depth transformers possess an inductive bias for compositional rule induction and systematization, in contrast to the mere memorization and shallow pattern matching of vanilla transformers.
• Parameter-Efficient Scaling: Reasoning capacity can be incremented with inference-time compute rather than additional parameters, establishing a promising direction for cost-efficient model design.
• Robustness to Task Complexity: Dynamic training strategies and adaptive halting mechanisms offer resilience to task heterogeneity, potentially translating to real-world settings where reasoning complexity is not known a priori.
• Limitations: The overthinking phenomenon underlines the need for effective computation allocation strategies, suggesting a critical research avenue for integrating better stopping rules and confidence estimation.

Future Research Directions

Open questions remain regarding practical transfer to large-scale, heterogeneous corpora. The controlled synthetic setting abstracts away challenges such as variation, distractors, and partial supervision inherent in natural text. Investigation at LLM scales is required, especially for integrating recurrence-based architectures with pretraining and fine-tuning in realistic data regimes. Additionally, interface mechanisms between recurrence and latent representation modularization are crucial for optimizing depth allocation without degradation.

Furthermore, the findings motivate analysis into architecture-agnostic generalization measures and the potential synergies between explicit multi-step prompting (e.g., CoT) and implicit recurrent computation.

Conclusion

This study provides compelling mechanistic and empirical evidence that recurrent-depth transformers can achieve systematic and extrapolative implicit reasoning, overcoming the compositional generalization boundaries observed in conventional transformer architectures. The requirement for architectural weight sharing and controlled recurrence for systematic grokking and reasoning depth suggests a paradigm shift for implicit reasoning in future LLMs and symbolic-reasoning applications. While open challenges remain with scaling and practical deployment, the insights laid out in this work robustly chart the path for advancing compositionally robust neural reasoning models.