Recirculation

Abstract: We describe an inference-time architectural enhancement for off-the-shelf foundation models that markedly reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation---which is better reserved for complex inferences rather than basic state tracking---as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves remarkable gains on the Gemma3 family, including a 23% reduction in perplexity on a suite of datasets, a 21% increase in accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.

• The paper introduces a lightweight recurrent pathway that transfers normalized deep-layer activations to shallower layers across token steps, separating persistent state updates from simple depth expansion.
• Recirculation reduces perplexity across most tested datasets, reaching a reported 35.40% reduction for Gemma3 12B and showing measurable token-level effects over context lags of up to 256 tokens.
• Adaptive recirculation uses a small learned controller with frozen foundation-model weights to achieve a 23.0% mean perplexity reduction and improve GSM8K error rates, while remaining sensitive to model, task, and prefill costs.

Problem formulation and central claim

“Recirculation” (2608.17981) addresses a specific computational limitation of feedforward transformers: contextual information computed in deeper layers is not directly available to shallower layers when later tokens are processed. This architectural asymmetry can produce state-tracking failures. A token may be disambiguated only after several layers have integrated its context, while subsequent processing stages still rely on a shallower, ambiguous representation. The paper links this failure mode to contextualization errors such as interpreting bank as a financial institution after the preceding discourse has established a geographical meaning. The motivating mechanistic-intervention evidence is that copying a contextualized deep-layer representation back to an earlier layer reduces such errors by 60% in the setting studied by Lepori et al. [2025.naacl-long.155].

The paper’s central proposal is to introduce a lightweight recurrent connection into an already trained transformer. After processing each input token, recirculation transfers a small, normalized component of a deeper residual-stream activation to a shallower layer at the next recurrent step. Unlike conventional recurrent architectures, the model weights remain unchanged; unlike chain-of-thought, the additional computation is latent and does not require generating intermediate natural-language tokens. The authors therefore frame recirculation as an inference-time architectural intervention designed to make deep contextualized states persistently available for subsequent computation.

The claim is not merely that additional computation improves performance. The paper argues that where the additional computation is placed matters: recirculation should support temporal state updating, whereas ordinary depth recurrence or looping primarily increases effective depth. This distinction is important because a looped transformer can reuse layers but does not necessarily provide a stable location in the architecture where an evolving state can be updated indefinitely.

Architectural mechanism

A standard transformer processes a fixed input sequence in parallel during training and prefill. Each token’s activation moves upward through a stack of layers, and the residual stream carries information across layers. However, information that becomes explicit at depth $s$ is not ordinarily available at a lower destination layer $d$ for later state updates. Recirculation adds this downward pathway while preserving the original forward computation.

For a source layer $s$ and destination layer $d$, the destination residual stream is replaced by a mixture of its original activation and a normalized source activation. In the basic formulation, the source contribution is scaled by $\alpha$, the destination contribution by $\beta$, and typically $\beta = 1-\alpha$. The source is rescaled to match the destination’s $L_2$ norm. In simplified form, the intervention is:

$d' = \alpha f(s) + \beta d,$

where $f$ is a normalization operator. The normalization is operationally important because residual-stream magnitudes generally vary with depth. Without it, a deep-layer signal can dominate the destination activation or produce unstable behavior. The paper reports that normalization usually changes the robustness of the source–destination landscape more than the best attainable perplexity, making useful layer pairs easier to identify.

The recurrence is organized across both layer depth and token position. At each recurrent update, the model processes the current token through the stack while propagating the previous step’s deep activation into a shallower layer. This allows the same layer to contain both a prior state and its updated successor. In contrast, a looped transformer propagates information through successive copies of the stack; arbitrary state updates therefore consume additional depth rather than reusing a fixed state location.

Figure 1: Unrolling distinguishes depth-only recurrence in looped transformers from the joint depth-and-input-step recurrence of recirculation.

This distinction leads to different computational costs. During autoregressive generation, the two stacks required by one-iteration recirculation can be executed in parallel, so the authors report essentially no additional generation latency on modern hardware. Prefill is different: because state updates depend sequentially on preceding input steps, the context cannot be processed fully in parallel. The resulting autoregressive prefill cost is the principal systems-level trade-off.

Figure 2: Recirculation permits state propagation to continue in the same layer across input steps, unlike the strictly upward state propagation of a looped transformer.

The method relies on an empirical assumption about residual-stream alignment. Because residual connections provide a shared representational space, the authors hypothesize that a feature encoded at one layer can be meaningfully added at another without an adapter or cross-attention mechanism. This assumption is plausible within the residual-stream framework, but it is not established as a general property of transformer representations. The effectiveness of normalization and the substantial variation across architectures indicate that layer alignment is only approximate and model-dependent.

Hyperparameter structure and language-modeling results

The authors first sweep the source layer, destination layer, and mixture coefficient on Gemma3 1B using arXiv text. The resulting performance surfaces are smooth rather than random, with a recurring favorable region in which the destination is several layers below the source. For Gemma3 1B, layer 4 is a particularly effective destination, with source layers approximately 5–7 layers higher. Across arXiv, PG-19, and C4, the best source–destination pair yields a mean perplexity reduction of 4.72% on the tuning data at $d$0.

Figure 3: Gemma3 1B perplexity varies systematically with the source layer, destination layer, and recirculation coefficient.

The selected layer pairs differ by model scale: $d$1 for Gemma3 1B, $d$2 for Gemma3 4B, and $d$3 for Gemma3 12B. These choices are then evaluated on ten language-modeling datasets. The reported gains are substantial, especially for the 12B model.

Model	Largest reported reduction	Representative results
Gemma3 1B PT	15.95%	Booksumm/book: 32.48 to 27.30; PG-19: 22.27 to 19.06
Gemma3 4B PT	15.95%	Booksumm/book: 29.09 to 24.45; PG-19: 19.49 to 16.43
Gemma3 12B PT	35.40%	PG-19: 52.86 to 34.15; Booksumm/book: 77.02 to 51.67

For nine of the ten datasets, recirculation improves perplexity across model scales. The exception is LAMBADA, where the 1B and 12B models slightly deteriorate. The authors attribute this anomaly to short sequences and tokenization artifacts, consistent with their later finding that recirculation is more useful when enough preceding context exists to form a persistent state. The strongest 12B result, a 35.40% reduction on PG-19, is notable but should not be interpreted as a uniform scaling law: the paper explicitly notes that the 12B baseline is comparatively weak as a LLM, which inflates the possible relative improvement.

The improvement is not equivalent to temperature calibration. For Gemma3 1B on PG-19, temperature adjustment alone reduces perplexity by 8.48%, whereas recirculation reduces it by 14.21%. Combining both produces a 19.55% reduction. The near-additivity of the effects indicates that recirculation changes token-conditional representations rather than merely sharpening or flattening the output distribution.

Figure 4: Source–destination sweeps exhibit a cross-dataset region of reduced perplexity, with a mean reduction of 4.72% across arXiv, PG-19, and C4 for the best Gemma3 1B configuration.

Architectural specificity and comparison with looping

The paper evaluates recirculation on Ministral3, Pythia, Qwen3, and Phi2. All four model families display a region of improved perplexity, suggesting that the qualitative effect is not unique to Gemma. However, the magnitude is approximately 5% for Gemma3 and below 0.5% for the other tested families under the authors’ unoptimized settings. The comparison is therefore evidence for broad receptivity, not for architecture-independent effectiveness. The authors did not conduct comparable normalization and coefficient searches for these models, so the lower gains may reflect suboptimal intervention parameters.

Figure 5: Diverse model families show favorable source–destination regions, although Gemma models exhibit substantially larger gains under the tested settings.

Older and newer Gemma generations retain strong compatibility with recirculation. The authors hypothesize that Gemma’s Peri-LN design, which normalizes both layer inputs and outputs, may preserve useful activation magnitudes across depth. A second possibility is that the optimization procedure used to train Gemma produces unusually aligned residual representations. Neither explanation is directly tested, leaving the architectural cause unresolved.

The comparison with looped transformers is conceptually and empirically important. The authors insert a copied layer range after its original occurrence, thereby increasing effective depth without changing weights, and perform the same source–destination sweep. For Gemma3, training-free looping does not produce a comparably robust improvement region. Recirculation helps across 1B, 4B, and 12B models, whereas looping appears beneficial primarily at larger scales.

Figure 6: Recirculation and training-free looping produce qualitatively different layer landscapes, supporting the claim that they implement different computational mechanisms.

This result contradicts a simple “more depth is better” interpretation. Recirculation’s benefit appears to depend on feedback from contextualized deep states into a shallower processing site, not solely on executing additional transformer blocks. The comparison is limited, however, by the particular implementation of looping and by the absence of task-specific or normalization tuning for all alternatives.

Token-level evidence for persistent state

The token analyses provide the paper’s strongest mechanistic support for the state-tracking interpretation. When only one token is recirculated, its influence on later-token log likelihood is largest at short lags but remains measurable out to a lag of 256 tokens. Early context positions, especially the first approximately ten tokens in Gemma3 1B, can be harmful, presumably because insufficient state has yet accumulated. Positions roughly 20–200 show the most persistent effects.

Figure 7: Recirculation produces short-lag and long-tail improvements, with effects varying by token position and grammatical category.

The benefits are content-dependent. Adverbs, adjectives, and verbs show the largest reductions in perplexity, whereas numerals, determiners, and pronouns show smaller effects. Plural nouns benefit more reliably than singular nouns. These patterns are difficult to reconcile with a token-independent calibration effect and are consistent with the hypothesis that recirculation selectively reinforces contextual information whose interpretation evolves over the sequence.

The authors also report approximately additive effects in log likelihood when multiple tokens are recirculated. This supports a distributed state-accumulation account, although additivity does not establish that the transported representation corresponds to an explicit belief state. It may instead reflect a more general alteration of residual-stream trajectories that happens to benefit context-sensitive prediction.

Downstream generation and reasoning

Recirculation improves several generative tasks, but the gains are heterogeneous. On a simple instruction-following task, pretrained hyperparameters reduce the error rate by approximately 25% for Gemma3 4B IT and 75% for Gemma3 12B IT. Task-specific layer tuning yields larger improvements. This result supports the idea that recirculation can improve executive control over recently specified rules, but the use of task-tuned hyperparameters also demonstrates the method’s dependence on evaluation criteria.

On the Racing Thoughts contextualization benchmark, recirculation improves two of three question types for the 1B and 4B instruction-tuned models. For the 12B model, two question types become worse, while the remaining type is at ceiling. Thus, the overall contextualization result is positive but not monotonic in model scale. The paper’s own interpretation is appropriately qualified: perplexity-selected hyperparameters came from pretrained models, and instruction-tuned-model sweeps produced larger gains. This means that the intervention is sensitive not only to architecture and scale but also to post-training.

For eight single-token or multiple-choice benchmarks, basic recirculation improves six datasets, but the differences are small and inconsistent. For example, Gemma3 4B accuracy increases from 57.90% to 58.28% on MMLU, from 81.78% to 82.07% on ARC Easy, and from 79.98% to 80.52% on PiQA, while it decreases slightly on WinoGrande and HellaSwag. These results suggest that recirculation is more reliable for reducing language-modeling loss than for changing discrete benchmark decisions.

GSM8K shows a stronger effect. Recirculation improves both pass@1 and pass@128, indicating gains in both capability sharpening and capability expansion under the paper’s interpretation. The method therefore appears to support not only the processing of the problem statement but also extended chain-of-thought generation. The precise numerical accuracies are presented graphically in the paper, but the stronger result concerns adaptive recirculation: it reduces GSM8K error by 8.8% for pass@1 and 20.9% for pass@128 while leaving the Gemma3 4B model weights unchanged.

Figure 8: Recirculation improves both greedy and sampled GSM8K performance, while adaptive recirculation yields the largest gains.

Adaptive recirculation

Adaptive recirculation replaces fixed scalar coefficients with token-conditional, vector-valued coefficients generated by a small MLP from the source and destination activations. The base model remains frozen. The MLP is trained on only 250 documents each from arXiv, C4, and PG-19, using 100 optimization steps. This design tests whether a small learned controller can determine when and along which residual dimensions deep activation should be transferred.

The ablation results identify two necessary ingredients: coefficient vectors outperform scalar coefficients, and token conditioning outperforms static coefficients. The best conditional-vector variant reduces mean perplexity by 23.0% across nine datasets, compared with 8.5% for fixed recirculation. It also slightly exceeds full fine-tuning of the recirculated model, which achieves a 21.6% reduction.

Figure 9: Conditional vector-valued coefficient prediction outperforms fixed, scalar, and unconditional variants, and slightly exceeds full model fine-tuning on the reported perplexity suite.

This comparison is strong but requires careful interpretation. Adaptive recirculation is not training-free: it introduces a learned MLP and uses supervised optimization on text distributions. Its principal efficiency advantage is that the foundation-model weights remain frozen, reducing the number of trainable parameters and potentially limiting catastrophic overfitting. However, downstream generalization depends critically on the adaptation dataset. MMLU-based adaptation improves several unrelated benchmarks, whereas ARC-specific adaptation can substantially reduce performance. Moreover, one reported MMLU adaptation condition trains on part of the MMLU test split, creating overlap that limits the evidential value of the corresponding MMLU result.

The distinction between language-modeling and task adaptation is therefore central. Adaptive recirculation offers a powerful mechanism for improving perplexity with limited optimization, but it does not establish that one learned controller transfers uniformly across tasks, instruction formats, or model families.

Limitations and open questions

The paper identifies several unresolved practical and scientific issues. First, optimal source and destination layers, mixture coefficients, and normalization schemes are domain- and model-dependent. Although perplexity-selected settings often transfer to downstream tasks, the heterogeneous contextualization and benchmark results show that this transfer is not guaranteed.

Second, the prefill bottleneck may dominate deployment cost for long contexts. The proposed method requires sequential state updates even when the entire prompt is available. Blockwise recirculation could reduce this cost, but the paper does not measure the trade-off between block size and state-tracking accuracy.

Third, only one additional recirculation iteration is evaluated. The architecture can support multiple iterations, and unlimited iterations would approach a recurrent neural network, but stability, computational scaling, and diminishing returns remain unknown. Similarly, the experiments use one source–destination path, despite evidence that multiple favorable regions sometimes occur in the layer sweeps.

Finally, the mechanistic interpretation remains underdetermined. The observed lag structure, part-of-speech dependence, and distinction from temperature tuning support a state-persistence hypothesis, but they do not prove that recirculation implements an explicit belief-state update. A key open question is whether the transported activation can be causally mapped to identifiable semantic or task-relevant state variables, rather than merely serving as a beneficial residual perturbation.

Conclusion

“Recirculation” (2608.17981) proposes a recurrent feedback pathway for pretrained transformers that transfers normalized deep-layer activations to shallower layers across token-processing steps. Its principal contribution is the separation of temporal state recurrence from depth recurrence: unlike looping, recirculation allows state to remain in a fixed layer while being updated over time. On Gemma3, the method produces large reductions in perplexity—up to 35.40% in the reported language-modeling evaluation—and improves instruction following, contextualization, and GSM8K performance. Adaptive recirculation further reaches a 23.0% mean perplexity reduction and improves GSM8K error rates without modifying the foundation-model weights.

The empirical pattern is substantial but architecture-sensitive, task-dependent, and computationally constrained by sequential prefill. The paper’s most consequential claim is therefore methodological rather than universal: a trained transformer can reveal useful architectural feedback paths through carefully controlled inference-time interventions, and these paths may provide a lower-cost alternative to redesigning or fully retraining the model.