Memory-Efficient Looped Transformer (MELT)

• The paper introduces a learned gated mechanism that updates a per-layer, per-token cache, replacing the traditional append-only KV cache in looped models.
• MELT decouples the number of reasoning loops from the memory footprint, reducing cache complexity from O(NLTd) to O(NLd) without sacrificing latent multi-step reasoning.
• It employs a two-phase fine-tuning process with interpolated transition and attention-aligned distillation, showing competitive performance against Ouro-style recurrent models.

Memory-Efficient Looped Transformer (MELT) is a looped language-model architecture that decouples reasoning depth from Key-Value (KV) cache growth by replacing the append-only per-loop KV cache of LoopLM/Ouro-style recurrent LLMs with a single per-layer, per-token cache updated across loops by a learnable gate. It preserves the latent multi-step computation characteristic of looped models, but changes the memory mechanism so that increasing the number of reasoning loops does not increase KV-cache size; in the reported implementation, MELT is obtained by fine-tuning pretrained Ouro parameters with a dedicated two-phase post-training procedure rather than by training from scratch (Vendrell et al., 8 May 2026).

1. Conceptual setting and motivation

Looped or recurrent LLMs reuse the same stack of Transformer layers multiple times for the same token. In the formulation used for MELT, a standard causal model applies a stack once,

$$F(x)=\mathrm{lmhead}\circ T_N\circ \dots \circ T_1 \circ \mathrm{emb}(x),$$

whereas a looped model re-applies the stack $T$ times,

$$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$

This supplies additional latent computation without generating intermediate tokens, and LoopLM/Ouro-style systems are reported to match or beat non-looped Transformers nearly twice their size on reasoning benchmarks (Vendrell et al., 8 May 2026).

The difficulty is that standard looped causal models retain a standard KV cache across loop iterations. For sequence length $L$, depth $N$, hidden dimension $d$, and loop count $T$, the total cache scales as

$M_{\text{LoopLM}} = O(NLTd).$

Reasoning depth therefore increases memory usage linearly. MELT targets precisely this bottleneck: it seeks to preserve the reasoning behavior of looped models while restoring KV-memory scaling to that of a non-looped Transformer, namely $O(NLd)$ (Vendrell et al., 8 May 2026).

The broader looped-transformer literature provides the conceptual background for this move. Looped architectures were explicitly motivated as parameter-efficient iterative learners, and linear looped Transformers were shown to implement multi-step gradient descent during in-context learning with tied parameters across loop steps (Yang et al., 2023, Chen et al., 2024). MELT inherits that recurrent-depth perspective, but focuses on the memory cost of autoregressive deployment rather than on parameter count alone.

2. Architecture and KV-sharing mechanism

MELT keeps the looped/Ouro-style reuse of a shared stack, but changes how attention state is stored. For a given layer and loop index $t$, the model distinguishes the standard hidden state $T$0 from a latent state $T$1 used specifically for KV generation. The latent state is updated by an element-wise gated momentum rule: $T$2 The gate $T$3 controls whether each dimension preserves prior latent content or overwrites it with the current hidden state. Keys and values are then derived from the latent state: $T$4 Crucially, these KVs are not appended across loops. For each token and layer, MELT keeps one cache row and overwrites or updates it across the internal reasoning loops (Vendrell et al., 8 May 2026).

This change yields a fixed-size per-layer KV cache. In a standard looped model, each layer stores $T$5 KV entries; in MELT, each layer stores exactly one KV row per token, independent of loop count: $T$6 At inference for token $T$7, a query is still formed from the current hidden state,

$T$8

and attends over the current layer cache $T$9, whose rows correspond to the final latent-derived representations of earlier tokens. Earlier tokens therefore carry information integrated over their own completed loops, but the cache size does not scale with reasoning depth (Vendrell et al., 8 May 2026).

The paper contrasts this learned mechanism with simpler constant-memory heuristics. It evaluates “Last,” “Mean,” “EMA-0.2,” and “Single-gated” variants that also constrain the cache to be constant in $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$0, but finds that MELT’s element-wise gating is the strongest variant on the reported reasoning benchmarks. A central analytical claim is that when $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$1, the Jacobian of the latent update approaches the identity, so gradients through loops are preserved; the paper describes this saturated-gate regime as a “Gradient Superhighway” (Vendrell et al., 8 May 2026).

3. Adaptation and training procedure

MELT is not trained from scratch in the reported experiments. Instead, it is adapted from a pretrained Ouro model, because naively replacing Ouro’s append-only per-loop KV cache with MELT’s gated latent-state mechanism makes the model behave like an untrained network and damages the learned representations (Vendrell et al., 8 May 2026).

Training is organized around chunk-wise processing. Because token $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$2 depends on the fully updated latent state of token $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$3 after all its loops, full parallel sequence training is incompatible with MELT’s KV dynamics. The compromise is to split each sequence into chunks of size $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$4, process tokens in parallel within a chunk while holding the chunk’s initial latent states fixed, and process chunks sequentially. In the main setup, the chunk size is 320, the sequence length is 10k, training runs for 130 hours on $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$5 H100, and the data mixture includes AceReason-1.1-SFT and OpenThoughts3 (Vendrell et al., 8 May 2026).

The first phase is an interpolated transition from LoopLM/Ouro KV semantics to MELT KV semantics. For each batch, MELT computes both the original LoopLM-style cache $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$6 and its own gated cache $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$7, then interpolates: $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$8 with $$F^{(T)}(x)=\mathrm{lmhead}\circ M^{(T)}\circ \mathrm{emb}(x), \qquad M^{(T)}=M\circ M\circ \dots \circ M.$$9 ramped linearly from 0 to 1 during training. This allows the new gating parameters to learn while the model initially behaves almost like Ouro. Knowledge distillation from Ouro is applied at all loops during this phase (Vendrell et al., 8 May 2026).

The second phase is attention-aligned distillation. Once $L$0, MELT runs in pure constant-memory mode, but naive continuation causes representation drift. A frozen Ouro teacher is therefore used to align MELT’s post-attention representations at every layer and loop: $L$1 and the overall objective is

$L$2

Ablations show that removing any of the four main ingredients—chunk-wise training, interpolated transition, KD across loops, or attention-aligned distillation—degrades performance, and removing chunk-wise training causes complete failure on all reported benchmarks (Vendrell et al., 8 May 2026).

4. Complexity, memory footprint, and empirical performance

MELT’s central complexity result is a memory reduction, not a compute reduction. Relative to LoopLM/Ouro,

$L$3

The factor of $L$4 disappears from the cache, but FLOPs at inference still grow with the number of loops, so MELT decouples reasoning depth from memory, not from compute (Vendrell et al., 8 May 2026).

For 32k-token generation under vLLM, the reported memory numbers are:

Model	KV for 32k tokens	Total memory
Ouro-1.4B-Thinking	25.17 GB	27.97 GB
MELT-1.6B	6.29 GB	9.49 GB
Qwen3-1.7B	3.67 GB	7.07 GB

These figures imply that MELT uses about $L$5 less KV memory than Ouro for the same looped reasoning, and that its total memory footprint is only about 2.5 GB higher than Qwen3-1.7B, with the remaining gap attributed mainly to Qwen’s use of Multi-Query Attention rather than to looped reasoning itself (Vendrell et al., 8 May 2026).

The reported benchmark picture is similarly mixed and specific. On math and reasoning tasks, MELT generally sits slightly below Ouro but above comparably sized non-looped baselines, while occasionally surpassing Ouro. Selected results are:

Benchmark	Ouro-1.4B-Thinking	MELT-1.6B
AIME24 pass@1	50.2	46.7
AIME26 pass@1	36.7	41.0
MATH-500 accuracy	94.4	93.4
HumanEval accuracy	76.8	81.7

The average math pass@1 is reported as 62.3 for Ouro and 59.9 for MELT, while Qwen3-1.7B reaches 56.9 and Gemma4-E2B 56.0. On MMLU-Red, MELT and Ouro are tied at 74.2. The paper therefore presents MELT as preserving most of the looped-reasoning advantage over non-looped models while sharply reducing loop-induced memory usage (Vendrell et al., 8 May 2026).

A common misunderstanding is that MELT is a general long-context compression scheme. The reported results do not support that reading. MELT still stores one KV row per token and per layer, so its memory remains linear in sequence length; what it removes is the additional multiplicative factor in loop count. This is a distinct objective from architectures that reduce attention complexity or replace full per-token state with a much smaller recurrent state (Vendrell et al., 8 May 2026).

5. Relation to adjacent looped-transformer research

MELT belongs to a fast-growing family of looped architectures, but its design target is narrower than many neighboring proposals. “LT2: Linear-Time Looped Transformers” replaces quadratic full attention inside looped models with linear or sparse mixers, arguing that looping synergizes with subquadratic attention by enabling iterative memory refinement and receptive-field growth. LT2 therefore addresses the attention-complexity bottleneck of looped models, whereas MELT addresses the per-loop KV-cache bottleneck while still using attention over full per-token $L$6 for the sequence (Deng et al., 20 May 2026).

“Parallel Loop Transformer for Efficient Test-Time Computation Scaling” introduces Cross-Loop Parallelism and shared-KV Gated Sliding-Window Attention to reduce the latency and memory penalties of sequential loops, and “LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling” reports a strongly non-monotonic loop-count effect in that setting, with two loops outperforming both one-loop and deeper-loop variants on code and agentic benchmarks (Wu et al., 28 Oct 2025, Yang et al., 16 Jun 2026). These works are architecturally adjacent because they also share KV information across loops, but they target near-baseline latency and parallel scheduling rather than MELT’s learned per-token latent-state overwrite.

Parcae addresses a different problem again: stable training and scaling laws for looped LLMs. It treats looping as a nonlinear dynamical system over the residual stream, constrains the spectral norm of injection parameters via discretization of a negative diagonal parameterization, and derives training and test-time scaling laws for stable recurrent depth (Prairie et al., 14 Apr 2026). Hyperloop Transformers, by contrast, pursue parameter efficiency through begin–middle–end looping and hyper-connections, reporting approximately 50% fewer parameters than depth-matched Transformers and robustness under post-training 4-bit quantization (Zeitoun et al., 23 Apr 2026).

Theoretical work has also split looped models into distinct memory regimes. “Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation” argues that compressed latent loops are limited by the size of their recurrent state, whereas full sequence-state loops and chain-of-thought scratchpads live in a memory-rich regime with qualitatively different algorithmic capacity (Zhang, 29 May 2026). This clarifies that “memory-efficient” looped modeling can refer to different things: reduced parameter count, reduced attention complexity, reduced latency, reduced loop-induced KV growth, or reduced recurrent-state size. MELT is specifically a reduction of loop-induced KV growth.

6. Limitations, misconceptions, and future directions

The paper is explicit about several limitations. First, MELT inherits Ouro’s fixed number of loops at inference. Different inputs likely require different reasoning depth, and the authors identify adaptive loop depth as a natural extension because MELT’s constant-memory cache means only compute, not KV size, would change with additional loops (Vendrell et al., 8 May 2026).

Second, MELT does not yet incorporate Multi-Query Attention. This is important because the strongest non-looped baseline in the memory comparison, Qwen3-1.7B, is more memory-efficient largely because it shares $L$7 across heads. The paper therefore treats MQA as an obvious next step for narrowing the remaining deployment gap (Vendrell et al., 8 May 2026).

Third, chunk-wise training is essential but reduces training parallelism relative to standard Transformers. This is not a cosmetic issue: the sequential dependence of MELT’s cache semantics is what makes fully parallel sequence training invalid, and more parallelizable training strategies remain open. Fourth, MELT usually remains slightly below Ouro on math benchmarks, so the memory reduction is not free; it is a concrete trade-off against some reasoning performance (Vendrell et al., 8 May 2026).

A final misconception concerns heuristic cache sharing. Earlier looped-model work suggested averaging or reusing selected loop caches, but MELT reports that such untrained heuristics collapse on long reasoning tasks in strong Ouro models, often producing coherent beginnings followed by repetitive or incoherent degeneration. The learned gated latent state is therefore not merely an engineering refinement; within the paper’s evidence, it is what makes constant-memory looping viable at all (Vendrell et al., 8 May 2026).

In that sense, MELT occupies a specific point in the design space of efficient looped LLMs: it preserves full-sequence per-token attention and latent multi-step reasoning, but rewrites the cache semantics so that deeper internal reasoning no longer implies deeper KV storage.