T^2MLR: Transformer with Temporal Middle-Layer Recurrence

Abstract: Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We in troduce Transformers with Temporal Middle-Layer Recurrence (T2MLR), a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position, enabling abstract intermediate computation to persist across decoding steps with little inference overhead. Across natural-language pretraining and multi-hop reasoning finetuning, T2MLR consistently outperforms data- and parameter-matched Transformer base lines. Moreover, applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence. Im portantly, T2MLR does not require pretraining from scratch: retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption. These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.

• The paper introduces a temporal shortcut that recycles middle-layer representations, enabling persistent latent reasoning without additional inference overhead.
• By fusing a recurrent cache through a gated module, T²MLR outperforms standard Transformers and LSTM baselines on both reasoning and retrieval tasks.
• The method can be retrofitted to pretrained models, yielding significant improvements on GSM8K and MATH benchmarks while maintaining training and inference efficiency.

T$^2$MLR: Transformer with Temporal Middle-Layer Recurrence

Introduction and Motivation

The Transformer architecture's reliance on strictly autoregressive decoding introduces a significant constraint for multi-step reasoning: at each step, rich latent representations are projected into a sparse token space, creating a temporal bottleneck that impedes the persistence of intermediate reasoning. T$^2$MLR (Transformer with Temporal Middle-Layer Recurrence) proposes a principled solution: a temporal shortcut that recycles middle-layer feature representations from the previous token into a shallow layer at the current position, thus facilitating persistent latent computation without augmenting inference cost substantially.

Figure 1: T$^2$MLR fuses a deep-layer representation from the previous token position into a shallow layer of the current token, yielding improved pretraining perplexity and downstream reasoning performance.

Empirical and mechanistic analyses have repeatedly established that abstract reasoning predominantly occurs in the middle layers of modern Transformers, while the early and late layers focus predominantly on lexical processing and output projection, respectively. Prior attempts at latent reasoning or recurrence have only superficially addressed this by integrating recurrence via the embedding or output layers, or by relooping over full layer stacks, the latter incurring substantial test-time cost.

Architecture: Temporal Middle-Layer Recurrence

T$^2$MLR modifies the standard decoder-only Transformer by incorporating a recurrence mechanism connecting the middle of the network across autoregressive steps. It maintains a constant-size recurrent cache $R_t$, constructed from the output of layer $\ell_2$ at time $t$, which is fused into the input to an earlier layer $\ell_1$ at time $t+1$. The fusion is implemented with a gated module, combining the standard residual stream and the recurrent cache, and controlled by dynamic and static gating parameters.

Figure 2: T$^2$MLR architecture and its representation fusion module; the recurrent cache brings layer $^2$0 representations from time $^2$1 to an earlier layer $^2$2 at time $^2$3.

The gating design is critical for stability and expressivity, enabling the model to interpolate between pure Transformer and RNN behavior. Importantly, the recurrent pathway is initialized to be inactive, ensuring equivalence with a standard Transformer at initialization and allowing stable training.

For efficient training, T$^2$4MLR adopts a Jacobi-style approximation to compute the recurrent cache in parallel across tokens, crucial for scalability on long sequences with teacher forcing. Multiple fixed-point refinement passes are performed over the recurrent block, balancing approximation fidelity and computational efficiency.

Figure 3: The T$^2$5MLR training scheme uses Jacobi iterations through middle layers to approximate the ground truth recurrent cache in a temporally parallel manner.

Empirical Analysis: Synthetic and Real-world Reasoning Tasks

To elucidate the model's inductive biases, T$^2$6MLR is first evaluated on the S$^2$7-Retrieval task, designed to decouple the capabilities of in-context retrieval (favored by Transformers) and latent state tracking (favored by recurrent architectures). T$^2$8MLR, with only four Transformer layers and shallow recurrence, matches the expressive power of deep Transformers for group computations, while retaining retrieval capacity far superior to LSTM baselines.

Figure 4: T$^2$9MLR outperforms LSTM and Transformer baselines on S$^2$0-Retrieval, sustaining exact-match and per-token accuracy at larger sequence lengths than permitted by standard parallel-scan solutions.

When applied to large-scale autoregressive language modeling and multi-hop reasoning (using benchmarks such as GSM-Aug, HotPotQA, ProsQA, and Variable Assignment), T$^2$1MLR consistently surpasses parameter- and data-matched Transformer baselines. A strong and unexpected result is that localizing the recurrence to only 20% of the middle layers, rather than looping over all layers, outperforms full-model recurrence—demonstrating the primacy of reasoning in middle network strata.

Figure 5: Validation cross-entropy loss during pretraining, highlighting favorable perplexity trends for T$^2$2MLR, particularly with middle-layer recurrence.

Figure 6: Fine-tuning results on multi-hop reasoning and grade-school math tasks, with middle-layer recurrence variants ($^2$3) dominating full-loop ($^2$4) and Transformer baselines.

Additionally, T$^2$5MLR can be retrofitted onto pretrained models: inserting the recurrent fusion module into a SmolLM2-1.7B-Instruct backbone and briefly fine-tuning yields GSM8K accuracy improvements from 35.8 to 39.9 and MATH500 from 12.8 to 18.0, demonstrating ease of adoption and transferability.

Training Scalability and Computational Tradeoffs

The Jacobi-based recurrent cache approximation matches exact sequential training in downstream perplexity and gradient directionality (as shown via gradient similarity and $^2$6 metrics in the supplementary analysis).

Figure 7: Cosine similarity and relative $^2$7 distance between gradients under different Jacobi approximation depths versus full-depth anchor, justifying the chosen approximation hyperparameters.

T$^2$8MLR introduces minimal inference overhead—around $^2$9 per generated token—compared to both full-loop and pause-token baselines whose costs scale linearly with loop depth. During prefill (prompt processing), the Jacobi scheme can further accelerate caching without performance loss.

Architectural Insights and Mechanistic Interpretability

Dynamic analysis of the learned gates reveals T$^2$0MLR consistently leans toward positive recurrent gates and negative input gates, aligning with its design rationale: the residual stream is preserved as the default, and the recurrent cache offers predictive refinement where beneficial, rather than overwhelming or competing with local processing.

Figure 8: Evolution of gating parameters $^2$1 and $^2$2 throughout pretraining, illustrating learned gate asymmetry.

A probing study indicates that T$^2$3MLR’s middle-layer representations are more predictive of future tokens than the baseline architecture, further corroborating the utility of temporal recurrence in latent computation.

Relationship to Prior Work

T$^2$4MLR differentiates itself from previous latent reasoning (e.g., COCONUT, CODI) and looped Transformer approaches by targeting temporal recurrence at the locus of abstract reasoning, rather than indiscriminately looping full stacks or routing recurrence via input embeddings. Unlike state-space or memory-augmented architectures, T$^2$5MLR leaves the attention mechanism and overall interface untouched, acting solely as an architectural refinement for improved temporal propagation within the latent space.

Practical and Theoretical Implications

Practically, T$^2$6MLR offers a drop-in architectural improvement for reasoning-heavy applications, especially where multi-step latent computation is critical and inference efficiency must be preserved. Retrofitting is feasible; no full retraining is required. Theoretically, the results suggest that the position and specificity of recurrence within the model, not mere presence or depth, are the primary determinants of reasoning capacity in modern autoregressive networks.

This work sharpens the perspective on hybrid architectures, demonstrating that effective reasoning does not mandate full-stack recurrence or departure from the Transformer backbone, but rather smartly engineered architectural routes—specifically, temporal middle-layer recurrence.

Conclusion

T$^2$7MLR establishes that temporally recurring middle-layer representations in Transformers enables persistent, abstract latent computation, resulting in consistent improvements for both pretraining and reasoning-centric tasks. Middle-layer recurrence outperforms full-stack recurrence, offering a hardware-efficient pathway for amplifying reasoning in contemporary LLMs. The method is readily transferable to existing Transformer deployments, and future work should explore further mechanistic interpretability and applications in on-policy RL or retrieval-augmented systems.

(2607.15178)