LoopCD Framework for Looped Transformers

• LoopCD is a training-free contrastive-decoding framework for Looped Transformers. It enhances final token prediction by selectively adjusting unstable decisions with intermediate recurrent state predictions.
• LoopCD integrates intermediate and final recurrent prediction states for supervised latent refinement during autoregressive decoding, enhancing reasoning tasks without additional training
• The method adjusts unstable decisions more frequently, particularly improving 'hard' tokens in reasoning eras such as numbers, punctuation, and operators

LoopCD is a training-free contrastive-decoding framework for Looped Transformers and Looped LLMs (LoopLMs). It exploits intermediate predictions produced during recurrent refinement: an earlier recurrent state supplies a weaker, temporally aligned reference, while the final recurrent state supplies the stronger prediction. By extrapolating from the earlier prediction through the final one, LoopCD selectively changes token rankings, particularly on decisions that remain unstable across recurrent iterations. The method requires no additional model, training, modified prompt, or additional recurrent computation; its logit-space form requires one additional output pass, whereas its hidden-state form operates before the output head with negligible additional FLOPs (Liu et al., 1 Oct 2026).

1. Architectural context and recurrent prediction

A Looped Transformer repeatedly applies shared recurrent block weights rather than using a distinct parameterized block at every depth. It may be denoted by $(p,k\otimes d,l)$, where $p$ is the number of untied prelude layers, $k$ is the number of recurrent layers, $d$ is the number of recurrent applications, and $l$ is the number of untied coda layers. For an input hidden-state sequence $X$, its computation is

$$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$$

The architecture contains only $p+k+l$ distinct layers’ worth of parameters but performs computation equivalent to $p+kd+l$ layers. Recurrence therefore increases test-time computation while enabling iterative latent refinement without a proportional increase in parameter count.

Let $h_k$ denote the hidden state after the $p$0-th complete recurrent iteration. Each completed state can be passed through the coda and language-model head:

$p$1

Consequently, a Looped Transformer naturally provides a sequence of next-token distributions,

$p$2

for the same prefix. The final representation $p$3 is normally used for decoding, while earlier states are discarded by standard inference.

The principal evaluated model families include Ouro, Huginn, Parcae, and Looped-Qwen3. Ouro repeats its complete decoder stack and has no separate prelude or coda Transformer layers. Huginn uses a prelude, a recurrent block, and a two-layer coda; its recurrent state is initialized from Gaussian noise. Parcae uses prelude and coda layers with learned recurrent dynamics and projections. Looped-Qwen3 wraps a frozen four-layer middle window with damped recurrent updates and has 15 layers before and 17 layers after the recurrent window.

LoopCD depends on completed recurrent iterations rather than arbitrary internal layers. Internal states within an incomplete recurrent pass can have output distributions that drift substantially before the pass has finished and are therefore poor weak predictors. The relevant contrastive axis is temporal recurrent depth, not ordinary spatial Transformer depth.

2. Contrastive principle and decoding variants

The central assumption is that earlier recurrent states are generally weaker because they have received less recurrent computation, while the final state is stronger because it has received the full recurrent depth. The difference between them encodes a direction of latent refinement. If an early state favors one token but later computation increases the relative score of another, extrapolating from the early prediction through the final prediction can reinforce that change.

Let $p$4 be an earlier reference state and $p$5 the final recurrent state. The guidance coefficient is $p$6. LoopCD has two principal variants.

LoopCD-Logits

LoopCD-Logits independently decodes the reference and final states:

$p$7

$p$8

It forms the guided logits

$p$9

or, equivalently,

$k$0

The next token can be selected greedily by

$k$1

or sampled from $k$2.

In probability space, the resulting distribution satisfies

$k$3

Tokens favored more strongly by the final state than by the weak reference receive increased scores. This is the standard contrastive-decoding interpretation, with both distributions generated by the same model and prefix.

LoopCD-Logits requires one additional coda and output-head pass for $k$4. It does not require an additional recurrent pass.

LoopCD-Hidden

LoopCD-Hidden performs the contrast before the coda and output head:

$k$5

The guided state is then decoded once:

$k$6

Its additional computation consists primarily of vector addition, subtraction, and scaling. Under the reported FLOP accounting, its multiplier is effectively $k$7 relative to ordinary decoding.

The two variants are not generally equivalent. A nonlinear or deep coda can attenuate or rotate the hidden-state difference. When the coda is absent or shallow, hidden-state guidance can preserve much of the useful contrast. For Ouro, the hidden-state update closely tracks the orthogonal logit re-ranking component, with reported cosine similarity of approximately $k$8 to that component and approximately $k$9 to the complete logit update.

Variant	Contrast location	Output passes	Additional recurrent computation
LoopCD-Logits	After coda and output head	Two	None
LoopCD-Hidden	Before coda and output head	One	None

3. Loop instability and reasoning-critical tokens

Recurrent refinement is not monotonic for every token. Easy tokens generally acquire an increasingly strong margin for the eventual prediction. Hard tokens can become confident in an early prediction and then have that prediction weakened or revised by later iterations. Their confidence trajectories therefore exhibit non-monotonic peak-drop behavior.

Hard tokens include numbers, arithmetic operators, punctuation, reasoning-step markers such as “The” and “So,” and tokens determining intermediate quantities or transitions between reasoning steps. They are not necessarily semantically difficult in isolation; they are unstable under latent refinement.

For Huginn, let $d$0 be the logit of token $d$1 at recurrent iteration $d$2, and let

$d$3

be the final-loop winner. The fixed-target margin at iteration $d$4 is

$d$5

With

$d$6

the confidence drop is

$d$7

A token is hard when

$d$8

with $d$9. Approximately $l$0 of tokens are classified as easy, leaving hard tokens at roughly 10–15% of generated tokens.

Hard tokens are disproportionately associated with reasoning errors. Incorrect trajectories contain more hard tokens than correct trajectories. Replacing hard tokens with alternative top-three candidates damages accuracy more often than replacing easy tokens: on GSM8K, originally correct trajectories became incorrect in $l$1 of hard-token replacements and $l$2 of easy-token replacements; on StrategyQA, the corresponding rates were $l$3 and $l$4.

LoopCD changes hard-token decisions much more frequently than easy-token decisions. Among 58,447 generated tokens, the margin-flip rate was $l$5 for hard tokens and $l$6 for easy tokens. Hard tokens were therefore 10.7 times more likely to be flipped, despite LoopCD being applied to all tokens. The method does not require an explicit hardness detector because the contrast between recurrent states naturally becomes larger where refinement is unstable.

This mechanism does not imply that every hard-token change is beneficial. The final recurrent state is generally, but not invariably, the best anchor, and the contrastive operation can alter correct easy-token decisions.

4. Reference states and guidance strength

For LoopCD-Logits, the first completed recurrent state is generally the most useful reference:

$l$7

Its usefulness derives from disagreement with the final prediction while preserving alignment with the same prefix, vocabulary, and prediction task. A later intermediate state may be more accurate in isolation but less useful contrastively because it has already converged toward $l$8.

Reported examples include the following:

• Huginn on ARC-Challenge: $l$9 disagreed with the final prediction on $X$0 of questions and produced a $X$1-point gain; $X$2 disagreed on $X$3 and produced a $X$4-point gain.
• Parcae-1.3B: $X$5 disagreed on $X$6 of cases and produced a $X$7-point gain; $X$8 disagreed on $X$9 and produced a $$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$$0-point gain.
• Ouro-1.4B: $$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$$1 disagreed on $$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$$2 of cases and produced a $$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$$3-point gain; $$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$$4 disagreed on $$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$$5+0.34$$-point gain.</li> </ul> <p>Hidden-state guidance requires a different reference selection when the initial state contains substantial noise. For Huginn, whose state is initialized from Gaussian noise, the first hidden state contains a noise-removal component that is partly projected away by the coda and output head. The best hidden-state references were post-burn-in states:$$H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$6 or $H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$7, depending on recurrent depth and evaluation type. Using the first hidden state could reduce performance by $H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$8 points, whereas $H=S'_l\left(\left(\underbrace{S_k\circ S_k\circ\cdots\circ S_k}_{d\text{ times}}\right)(S'_p(X))\right).$9 produced approximately a $p+k+l$0-point improvement. Parcae and Looped-Qwen3 have deterministic initial representations and generally use $p+k+l$1 for hidden guidance.

  Guidance strength is task-sensitive. Typical settings are $p+k+l$2 for multiple-choice scoring, $p+k+l$3 for Ouro and Looped-Qwen3 generation, and $p+k+l$4 for Huginn generation. Values around $p+k+l$5–$p+k+l$6 are generally effective for autoregressive generation, while values beyond approximately $p+k+l$7 can cause degradation by propagating an early mistake through later tokens. Negative guidance consistently degrades performance.

  LoopCD-Logits also supports adaptive guidance. If $p+k+l$8 and $p+k+l$9 are the largest and second-largest final-state probabilities, respectively, the adaptive coefficient is

  $p+kd+l$0

  Guidance is therefore larger when the top two candidates are close and smaller when one candidate is dominant. The rule uses only the top two final probabilities and does not require full-vocabulary entropy. Adaptive guidance cannot be applied to LoopCD-Hidden without additional output passes, because the final distribution must first be decoded to determine the uncertainty margin.

  5. Inference procedure and computational characteristics

  At each autoregressive decoding step, LoopCD performs the following operations:

  1. Run the recurrent computation through completed states $p+kd+l$1.
  2. Retain the final state $p+kd+l$2 and a selected reference state $p+kd+l$3.
  3. For LoopCD-Logits, compute $p+kd+l$4 and $p+kd+l$5 through the coda and output head.
  4. For LoopCD-Hidden, form $p+kd+l$6.
  5. Apply fixed or adaptive guidance.
  6. Select the next token greedily or sample from the guided distribution.
  7. Append the token to the prefix and continue.

  LoopCD adds no training, architectural component, auxiliary model, modified prompt, or additional recurrent loop. Its overhead is mainly output-space arithmetic for the logit variant or vector arithmetic for the hidden variant.

  For LoopCD-Logits, reported forward-FLOP multipliers relative to unguided inference are $p+kd+l$7 for Ouro-1.4B, $p+kd+l$8 for Ouro-2.6B, $p+kd+l$9 for Huginn with 32 recurrent iterations, $h_k$0 for Huginn with 16 iterations, $h_k$1 for Parcae-370M, $h_k$2 for Parcae-1.3B, and $h_k$3 for Looped-Qwen3. The larger Looped-Qwen3 overhead results from reevaluating its 17-layer post-loop tail.

  For an earlier LoopCD implementation evaluated on Huginn-0125 and Ouro-1.4B, reported throughput on MATH-500 was:

  Model	Greedy	LoopCD
  Huginn-0125	$h_k$4	$h_k$5
  Ouro-1.4B	$h_k$6	$h_k$7

  For Huginn, absolute throughput was 5.09 tokens/s for greedy decoding and 4.99 tokens/s with LoopCD (Yu et al., 21 Sep 2026). LoopCD was compatible with early exit: on Huginn, greedy decoding achieved 5.09 tokens/s, LoopCD 4.99 tokens/s, early exit 10.06 tokens/s, and early exit plus LoopCD 9.89 tokens/s in the reported experiment.

  LoopCD can also compensate for reduced recurrent depth. At half depth, guided models matched or exceeded full-depth unguided baselines in the reported settings. The theoretical forward-FLOP savings ranged from 22.5% to 48.2%, depending on architecture and guidance variant. These are analytical FLOP reductions for a 512-token prefill rather than end-to-end latency measurements; actual latency depends on memory traffic, batching, kernel fusion, caching, and hardware utilization.

  6. Empirical results, limitations, and related uses

  LoopCD has been evaluated on mathematical reasoning, code generation, commonsense reasoning, and multiple-choice benchmarks. The model families include Ouro-1.4B and Ouro-2.6B, Huginn-0125, Parcae-370M and Parcae-1.3B, and Looped-Qwen3.

  For Ouro-2.6B-Thinking, adaptive LoopCD-Logits increased AIME 2024 pass@1 from $h_k$8 to $h_k$9, AIME 2025 from $p$00 to $p$01, and OlympiadBench from $p$02 to $p$03. The mean pass@1 gain across those three reasoning benchmarks was $p$04 points, while mean pass@10 improved by $p$05 points.

  For Huginn, LoopCD-Hidden increased HumanEval pass@1 from $p$06 to $p$07 at 32 recurrent iterations, a $p$08-point improvement with no additional output pass. At 16 iterations, LoopCD-Logits increased HumanEval from $p$09 to $p$10, while LoopCD-Hidden increased it from $p$11 to $p$12. On the four HumanEval and MBPP base-and-extended columns, LoopCD-Hidden improved the mean by $p$13 points at 32 iterations and $p$14 points at 16 iterations.

  LoopCD-Logits improved the seven-benchmark multiple-choice mean for every evaluated configuration. Representative gains included $p$15 for Ouro-1.4B with fixed guidance, $p$16 for Ouro-2.6B with adaptive guidance, $p$17 for Huginn with 32 iterations, $p$18 for Parcae-1.3B with adaptive guidance, and $p$19 for Looped-Qwen3 with adaptive guidance. Individual regressions occurred, particularly on some WinoGrande and PIQA settings.

  The method remains heuristic. Its principal limitations are:

  • Reference selection: the optimal weak state varies by model, initialization, task, and guidance space.
  • Final-state fallibility: the final recurrent state is not always correct, so extrapolation can reinforce an error.
  • Guidance sensitivity: coefficients suitable for multiple-choice evaluation may be too large for long autoregressive generation.
  • Autoregressive amplification: changing an early token can redirect the entire subsequent trajectory.
  • Deep-coda distortion: nonlinear coda layers can attenuate hidden-state guidance.
  • Individual regressions: aggregate improvements do not preclude benchmark-specific degradation.
  • FLOP–latency discrepancy: analytical compute reductions do not directly establish wall-clock speedups.
  • Output-interface requirement: the method assumes that intermediate completed recurrent states can be passed through a compatible coda and language-model head.

  LoopCD should be distinguished from other meanings of the same name. In one separate usage, “LoopCD” denotes a benchmark framework for testing recursive-collapse warning claims using gain $p$20, recursive persistence $p$21, and declining diversity $p$22 under a matched false-positive contract (Mullett, 29 May 2026). In another, loop calculus is used to improve belief-propagation and linear-programming decoding of low-density-parity-check codes [0609154]. These topics are unrelated to loop-wise contrastive decoding for Looped Transformers.

  The broader significance of LoopCD is that recurrent inference exposes a sequence of aligned intermediate predictions rather than only a final hidden state. The difference between an early and final prediction supplies a model-internal guidance direction. LoopCD uses that direction to modify close token decisions, improve reasoning quality, and, in some configurations, recover the performance of deeper recurrent computation while using fewer recurrent loops. Its empirical gains are substantial in several reported settings, but its behavior remains dependent on recurrent architecture, reference-state selection, guidance strength, output-interface geometry, and task.