Decoding Looped Transformers Better for (Almost) Free

Abstract: Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. We introduce LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting the final prediction with an earlier recurrent pass, operating either in logit space with one extra output pass (LoopCD-Logits) or in hidden-state space with zero output overhead (LoopCD-Hidden). Across four looped Transformer families, LoopCD delivers substantial, consistent gains at full recurrent depth: LoopCD-Logits raises Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. Crucially, these performance gains enable halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%. By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.

• The paper presents the LoopCD method, which uses earlier intermediate states as weak references for contrastive decoding, without the need for additional training, improving performance and reducing computation.
• LoopCD improves model accuracy, notably in mathematical reasoning (+11.45 points on one benchmark) and code generation (+9.15 points on one metric) while reducing computational expense.
• The method can compensate for reduced recurrent depth, halving the number of iterations while maintaining or exceeding the accuracy of full-depth inference.”] ,
• follow_up_questions
• Find recent papers about alternative decode methods for Looped Transformers skills in Transformer models.
• Is the hidden-state approach applicable to all Transformer models, and if not, why?
• Can this method be adapted for other sequence-to-sequence tasks beyond reasoning and code generation, such as text summarization?
• How does the adaptive guidance mechanism contribute to the stability of the model's predictions?

Problem setting and central contribution

Looped Transformers increase effective depth by repeatedly applying a shared Transformer block while keeping parameter count fixed. Each recurrent iteration produces an intermediate representation for the same input prefix, but conventional decoding uses only the final state. “Decoding Looped Transformers Better for (Almost) Free” (2610.02185) argues that these discarded intermediate states are not merely partial computations: they constitute naturally aligned weak predictors that can guide the final prediction through contrastive decoding.

The proposed method, LoopCD, contrasts a final recurrent prediction with an earlier one. The earlier state is weaker because it has undergone less recurrent computation, while remaining aligned with the final state in vocabulary, input context, and model parameters. This removes the principal engineering burden in conventional contrastive decoding: acquiring a suitable weak model or perturbing the input. The method requires no training, auxiliary checkpoint, modified prompt, or additional recurrent iteration.

The paper evaluates LoopCD on four looped-model families—Ouro, Huginn, Parcae, and Looped-Qwen3—covering mathematical reasoning, code generation, multiple-choice likelihood scoring, and generated-answer evaluation. Its strongest claim is computational as well as statistical: using half as many recurrent iterations, LoopCD matches or exceeds the accuracy of unguided full-depth inference while reducing theoretical forward FLOPs by 22.5% to 48.2%.

Architecture and LoopCD formulation

The evaluated architectures differ in where recurrence is inserted. Ouro repeats its complete decoder stack. Huginn, Parcae, and Looped-Qwen3 place a shared recurrent block between fixed prelude and coda layers, with different initialization and update mechanisms. This distinction matters because LoopCD can be applied either before or after the coda.

Figure 1: The evaluated looped architectures differ in whether recurrence covers the full stack or a shared middle block surrounded by fixed layers.

For a recurrent trajectory with states $h_1,\ldots,h_R$, the final state is normally mapped to logits through the coda and language-model head. LoopCD uses an earlier state, usually $h_1$, as the weak reference. Two variants are defined.

LoopCD-Logits independently decodes the reference and final states, producing $z_1$ and $z_R$, and applies

$z' = z_R + \omega (z_R-z_1).$

The resulting distribution is equivalent, up to normalization, to

$$p'(v) \propto p_R(v)\left(\frac{p_R(v)}{p_1(v)}\right)^\omega.$$

This variant requires a second pass through the post-loop coda and language-model head. The contrast explicitly amplifies token preferences that emerge between the early and final recurrent states.

LoopCD-Hidden applies the same extrapolation before the coda:

$h' = h_R+\omega(h_R-h_b),$

where $h_b$ is the selected reference state. The modified state is then passed through the coda and output head once. This produces effectively zero additional output computation, although its behavior depends on how the coda transforms or attenuates the hidden-state contrast.

Figure 2: LoopCD-Logits contrasts two output distributions, whereas LoopCD-Hidden combines recurrent states before a single coda and language-modeling pass.

The guidance coefficient $\omega$ is either fixed or adaptive. For multiple-choice scoring, the paper generally selects $\omega=0.5$. For autoregressive generation, smaller values, typically $h_1$0 to $h_1$1, are required because early token changes alter subsequent prefixes. The adaptive rule scales guidance using the margin between the two most probable tokens: it applies stronger guidance when the final model is uncertain and suppresses it when one token is already dominant.

This design is not equivalent to simply increasing temperature or sharpening the final distribution. The analysis decomposes the logit contrast into a component parallel to $h_1$2, which primarily changes calibration or temperature, and an orthogonal component, which changes token ranking. The latter is responsible for most of the accuracy gains.

Empirical gains at full recurrent depth

The most prominent results concern mathematical reasoning. On Ouro-2.6B-Thinking, adaptive LoopCD-Logits increases AIME 2024 pass@1 from 61.88% to 73.33%, an 11.45-point improvement. The same configuration raises AIME 2025 pass@1 from 49.58% to 56.88% and OlympiadBench pass@1 from 64.05% to 67.29%. Across the three mathematical benchmarks, the mean pass@1 gain is 7.33 points, while pass@10 improves by 3.71 points.

Configuration	Baseline	LoopCD	Change
Ouro-2.6B, AIME 2024 pass@1	61.88	73.33	+11.45
Ouro-2.6B, AIME 2025 pass@1	49.58	56.88	+7.30
Ouro-2.6B, OlympiadBench pass@1	64.05	67.29	+3.24
Huginn, HumanEval pass@1	23.17	28.66	+5.49
Huginn, HumanEval pass@1, hidden-state form	22.56	31.71	+9.15

These results imply that the recurrent trajectory contains actionable information beyond the final state, particularly for sequential reasoning where small token-level changes can compound over long solutions. However, the paper also reports an important counterexample: on Looped-Qwen3, fixed LoopCD lowers AIME 2024 pass@1 from 64.79% to 61.88% while increasing pass@10 from 82.91% to 86.17%. Thus, improved coverage across sampled trajectories does not necessarily translate into higher single-sample reliability. The contrastive signal can diversify or redirect solutions without uniformly improving every trajectory.

Code generation shows similarly substantial gains, especially for Huginn. At $h_1$3, LoopCD-Hidden increases HumanEval pass@1 from 22.56% to 31.71% and the extended-test score from 19.51% to 29.27%. Its four-column mean across HumanEval and MBPP, using both base and extended tests, improves by 5.13 points. The hidden-state form exceeds both fixed and adaptive logit guidance in this setting, despite requiring no additional output pass.

The implication is architectural: hidden-state guidance is not merely a cheaper approximation to logit guidance. When the coda is shallow, or when generation benefits from avoiding additional temperature sharpening, hidden-state extrapolation can provide a better intervention.

Cross-architecture multiple-choice performance

On seven multiple-choice benchmarks—ARC-Challenge, ARC-Easy, SciQ, MMLU, HellaSwag, WinoGrande, and PIQA—every evaluated LoopCD-Logits configuration improves its aggregate mean. Gains are smaller than on mathematical reasoning and code generation, but they are broadly distributed across model families.

Ouro-2.6B obtains a mean improvement of 0.84 points with fixed guidance and 0.95 points with adaptive guidance. Huginn at $h_1$4 improves by 0.74 and 0.86 points under fixed and adaptive guidance, respectively. Parcae-1.3B improves by 1.38 and 1.59 points. Looped-Qwen3 improves by 0.32 and 0.79 points.

The aggregate consistency conceals benchmark-level regressions. WinoGrande decreases for both Ouro models under most settings, and PIQA declines in several Huginn, Parcae, and Looped-Qwen3 configurations. Looped-Qwen3 also loses on ARC-Easy and SciQ under adaptive guidance. Therefore, the statement that LoopCD improves every model’s mean should not be interpreted as uniform per-benchmark improvement.

LoopCD-Hidden produces comparable or larger gains without additional output computation. It raises the seven-benchmark mean by 0.83 points for Huginn at $h_1$5, 0.75 points for Huginn at $h_1$6, and 0.85 points for Parcae-1.3B. The exception is Parcae-1.3B in comparison with logit-space guidance: its eight-layer coda attenuates the hidden-state contrast, allowing the logit form to retain a larger effective intervention.

Figure 3: LoopCD-Hidden retains most of LoopCD-Logits’ multiple-choice gain for shallow-coda models and exceeds it on Huginn code generation.

The coda determines the relationship between the two forms. In Ouro, which has no post-recurrent Transformer coda, the hidden-state update can produce a larger and more direct output displacement. In architectures with deeper codas, the hidden contrast is transformed by additional nonlinear layers and may be weakened or redirected. This explains why LoopCD-Hidden is particularly attractive for Ouro and Huginn, while Parcae-1.3B favors logit-space guidance more often.

Reduced recurrent depth and the compute–accuracy trade-off

The paper’s main systems result is that LoopCD can compensate for reducing recurrent depth. Halving the number of iterations produces an unguided multiple-choice deficit of 0.17 to 1.29 points relative to full-depth inference. Applying LoopCD at the reduced depth recovers this deficit in all six evaluated configurations.

For Huginn, reducing the trajectory from 32 to 16 iterations produces a guided model that exceeds the full-depth unguided baseline by 1.02 points with LoopCD-Logits and by 0.51 points with LoopCD-Hidden. Looped-Qwen3 at four rather than eight damped substeps matches the full-depth unguided baseline using hidden-state guidance.

Figure 4: Half-depth LoopCD recovers or exceeds full-depth unguided accuracy while reducing forward FLOPs.

The theoretical forward-cost reduction ranges from 22.5% for Parcae-370M to 48.2% for Huginn with hidden-state guidance. The savings depend on the fraction of total computation occupied by the recurrent block and on the cost of the post-loop readout. LoopCD-Logits incurs an extra coda and head pass, so its reduced-depth benefit is smaller in models with substantial fixed tails. LoopCD-Hidden avoids this overhead and is therefore preferable when the coda is computationally large.

These FLOP estimates are arithmetic workload estimates for a 512-token prefill, not measured latency or throughput. Kernel fusion, memory movement, batching, and implementation details could alter wall-clock speedups. The result nevertheless establishes a clear inference-compute trade-off: recurrent iterations need not be treated as indivisible accuracy requirements when their intermediate states can provide directional guidance.

Mechanistic account of the gains

The paper validates the weak-to-strong interpretation directly. Across six model configurations, the first recurrent iteration trails the final prediction by 4.0 to 23.7 points on the seven-benchmark mean. As iterations proceed, standalone accuracy rises and disagreement with the final prediction falls.

For Ouro-1.4B, the Jensen–Shannon divergence between the first recurrent prediction and the final prediction is 0.71 bits, decreasing to 0.013 bits by the third iteration. For Huginn, the corresponding KL divergence decreases from 3.0 nats after the first step to 0.04 nats after the sixteenth. LoopCD gains track this disagreement: on ARC-Challenge, Huginn’s first state disagrees with the final option on 51% of questions and yields a 3.07-point gain at $h_1$7, whereas the sixteenth state disagrees on only 7% and yields 0.17 points.

Figure 5: Contrastive gains increase with the disagreement between the reference prediction and the final prediction.

This result supports a precise interpretation of the reference state. Its standalone accuracy is not the decisive criterion. What matters is whether it supplies a direction that remains aligned with the final refinement trajectory. A later state may be more accurate but less useful because it has already converged toward the final prediction.

The first state is not universally optimal in hidden space. Huginn initializes its recurrent state with Gaussian noise; its first hidden-state contrast is therefore dominated by noise removal rather than semantic refinement. LoopCD-Hidden performs best with a later reference, approximately the sixth or seventh state, after burn-in. By contrast, the coda and language-model head project away much of this initialization noise, making Huginn’s first state effective for LoopCD-Logits.

The analysis also explains why intermediate layers within a recurrent iteration are unsuitable references. Their output distributions drift substantially from the final distribution, even late in the pass. Using such states can reduce ARC-Challenge accuracy by up to 20.6 points for Ouro-1.4B and 24.9 points for Ouro-2.6B. Completed recurrent iterations are valid exit points; arbitrary internal layers are not. This sharply distinguishes LoopCD from layer-wise methods such as DoLa, whose intermediate representations are used as contrastive references in conventional dense Transformers (O'Brien et al., 2023, Lin et al., 2024).

Re-ranking uncertain decisions

LoopCD changes relatively few decisions, and its improvements are concentrated on low-confidence examples. At $h_1$8, only 5.8% to 14.9% of ARC-Challenge answers change across the analyzed models, while the least-confident fifth gains between 6.4 and 13.3 points. The most-confident fifth gains at most 0.4 points.

This behavior follows from the geometry of the update. If the final prediction has a large margin between its two leading candidates, the contrast is insufficient to change their order. If the margin is small, the recurrent difference can re-rank the candidates. Thus, the net improvement is a targeted correction of borderline decisions rather than a global reshaping of the distribution.

The orthogonal component of the logit contrast carries this re-ranking effect. On HellaSwag, the re-ranking component alone produces gains of +1.72 points for Ouro-1.4B and +2.27 points for Ouro-2.6B at $h_1$9, compared with +0.76 and +1.56 points for the full update. The parallel component can sharpen the distribution in ways that are harmful on some benchmarks. In Ouro, LoopCD-Hidden closely approximates the orthogonal re-ranking update: its update has cosine similarity 0.96 with the isolated re-ranking component, compared with only 0.25 with the full logit update.

Figure 6: LoopCD’s gains are primarily due to orthogonal re-ranking of uncertain decisions; parallel logit scaling can help or hurt depending on the benchmark.

The adaptive rule follows directly from this mechanism. It increases guidance when the top-two probability margin is narrow and decreases it when the final prediction is already settled. This prevents high-strength guidance from damaging confident decisions while retaining reach on ambiguous ones.

Guidance strength and generation stability

The optimal strength depends strongly on the evaluation protocol. Multiple-choice scoring tolerates a broad range centered near $z_1$0. Autoregressive generation is much less tolerant: useful fixed strengths typically lie between 0.2 and 0.3, and performance declines sharply beyond approximately $z_1$1.

The difference arises because multiple-choice evaluation makes one decision per item, whereas generation repeatedly feeds guided tokens back into the model. A small early error can change the entire subsequent prefix. The paper reports especially severe degradation at high strengths: for Ouro-1.4B, GSM8K loses 8 points at $z_1$2 and 20 points at $z_1$3; HumanEval shows losses of 10 and 22 points at those strengths.

Figure 7: Multiple-choice evaluation supports substantially stronger fixed guidance than autoregressive generation.

Adaptive guidance widens the stable operating range for multiple-choice scoring, maintaining positive gains across caps roughly between 0.5 and 1.0 where fixed guidance can overshoot. The paper does not establish that the same adaptive rule is sufficient for all generative settings: LoopCD-Hidden is evaluated with fixed strength because applying the margin rule would require an additional coda and head pass, eliminating its zero-output-overhead property.

Limitations and open questions

The strongest limitations concern evaluation scope, hyperparameter selection, and sensitivity to recurrent architecture.

First, the reduced-depth experiments are restricted to multiple-choice scoring. The paper establishes full-depth generation and reasoning gains, but it does not show that halving recurrent iterations preserves HumanEval, GSM8K, MMLU-Pro, or AIME performance. Since generation is much more sensitive to guidance strength and prefix compounding, the compute-reduction claim cannot yet be generalized from likelihood scoring to autoregressive reasoning.

Second, guidance strengths and reference states are selected through model- and task-specific sweeps. This is appropriate for characterizing the method, but it leaves deployment cost and transferability unresolved. In particular, the optimal hidden-state reference varies because of initialization noise, coda depth, and recurrent dynamics. A general reference-selection rule that does not require evaluation-specific tuning remains open.

Third, the benchmark improvements are not uniformly positive. GSM8K changes are mixed, with a decline of up to 1.29 points for Huginn under fixed guidance. Looped-Qwen3’s pass@1 decreases while pass@10 increases, and several multiple-choice benchmarks regress in individual configurations. The method therefore improves aggregate performance without guaranteeing monotonic gains for each task or metric.

Fourth, FLOP reductions are theoretical. The paper does not report end-to-end latency, memory bandwidth, batching behavior, or hardware measurements. The claim of “almost free” applies most directly to LoopCD-Hidden’s additional output computation, not necessarily to total inference latency.

Finally, the mechanism depends on completed recurrent iterations forming semantically aligned prediction states. The paper demonstrates this property for four families, but it remains open how it behaves under other loop parameterizations, recurrent normalization schemes, learned halting policies, or models whose intermediate states are not trained or calibrated as prediction interfaces.

Conclusion

“Decoding Looped Transformers Better for (Almost) Free” (2610.02185) presents LoopCD as an inference-only method that converts discarded recurrent states into weak references for contrastive decoding. Its central empirical finding is that early completed iterations provide aligned directional information about how the final model prediction is formed. Extrapolating along this direction improves full-depth performance, especially on mathematical reasoning and code generation, while hidden-state guidance can add no extra output pass.

The method’s most consequential systems result is the recovery of full-depth multiple-choice accuracy at half recurrent depth, with 22.5% to 48.2% lower theoretical forward FLOPs. The mechanistic analysis attributes the gains primarily to re-ranking uncertain decisions rather than indiscriminate confidence sharpening. The principal unresolved question is whether the same depth-reduction effect extends reliably from likelihood-based evaluation to long autoregressive reasoning and code generation, where guidance errors compound across tokens.