Looped Language Models (LoopLMs)

• Looped Language Models (LoopLMs) are transformer architectures that reuse a block of layers through multiple passes to achieve greater effective depth with fewer parameters.
• They incorporate sparse Mixture-of-Experts blocks in place of dense feed-forward networks, enhancing compute-quality trade-offs and maintaining memory efficiency.
• LoopLMs support early-exit mechanisms at loop boundaries, allowing significant inference savings with minimal impact on perplexity.

Looped LLMs (LoopLMs) are a class of sequence models that achieve greater effective computational depth by reusing a block of transformer layers multiple times through depth. This approach substantially reduces the number of unique parameters and memory requirements while maintaining or improving reasoning ability and compute–quality trade-offs. LoopLMs support parameter-efficient scaling and provide natural locations for efficient early exits by design, but require careful architectural decisions—particularly regarding layer sparsity—to avoid quality degradation compared to standard transformers. Recent advances show that when implemented with sparse Mixture-of-Experts (MoE) layers, LoopLMs can match or exceed the performance of standard transformers under fixed compute, parameter, and memory budgets, while also enabling meaningful inference savings through early-exit mechanisms at loop boundaries.

1. Architectural Principles and Looping Mechanism

LoopLMs are primarily instantiated as decoder-only transformer architectures composed of multi-head self-attention, pre-layer RMSNorm, SwiGLU feed-forward networks, and rope positional embeddings. Rather than stacking $D$ unique layers once (“Base”), LoopLMs partition the depth into a block of $L$ unique layers, which is then reapplied $R$ times. The effective depth is $D = L \cdot R$.

Formally, denoting $f_\ell(\cdot)$ as the $\ell$th transformer layer, the input tokens $x_{\mathrm{tok}}$ are processed as follows: $$\begin{aligned} x_0^{(1)} &= \mathrm{Embed}(x_\mathrm{tok}) \ x_\ell^{(r)} &= f_\ell(x_{\ell-1}^{(r)}), \quad \ell = 1,\dots,L,\; r = 1,\dots,R \ x_0^{(r+1)} &:= x_L^{(r)}\quad(r < R) \ \mathrm{Output} &= \mathrm{Head}(x_L^{(R)}) \end{aligned}$$

This mechanism supports parameter reuse and reduces memory consumption by maintaining only $L$ unique sets of weights for a depth of $D$ transformer blocks. In pseudocode: $R$8 (Lee et al., 9 May 2026)

2. Sparse Layers and the Critical Role of Mixture-of-Experts

Dense LoopLMs alone do not scale favorably: iso-compute scaling experiments show strictly worse test cross-entropy and downstream accuracy relative to standard transformers. The key innovation is replacing each dense SwiGLU FFN with a sparse top-$L$0 Mixture-of-Experts (MoE) block ($L$1 experts, $L$2 active per token), each governed by a learned router. For input $L$3: $L$4

Empirically, the routing assignments significantly diverge between loops: 25–53% of tokens have completely disjoint expert sets in passes 1 vs 2, showing that looped layers realize distinct computations on each pass (routing “divergence”). This recovers much of the expressivity otherwise sacrificed by parameter sharing.

Comparative scaling law fits for compute-optimal models yield similar power-law exponents for Base ($L$5) and Looped-MoE ($L$6), with Looped-MoE often surpassing both dense looped and standard baselines in downstream tasks (Lee et al., 9 May 2026).

3. Compute–Quality Trade-Offs and Early Exits

LoopLMs naturally support early-exit mechanisms. Because loop boundaries correspond to full passes through the shared block, they present superior points for halting computation. The early-exit criterion is based on token-wise entropy $L$7: for tokens where $L$8, inference halts, and remaining layers’ FLOPs are saved.

An analysis of output convergence using Jensen–Shannon divergence to final output distributions reveals that a majority of tokens are “converged” by the end of loop 1, and sharp decreases in divergence occur exactly at loop boundaries. This enables highly efficient inference schedules, often saving 10–30% FLOPs with negligible degradation in perplexity. Increasing the number of loops (higher $L$9, shallower $R$0) further improves granularity for early exits (Lee et al., 9 May 2026).

4. Memory and Inference Efficiency

LoopLMs yield considerable memory savings due to aggressive parameter reuse: for example, a Looped-MoE with 216M weights matches or exceeds the performance of a 246M parameter Base, and pure looped models can shrink to 168M parameters at some quality cost. Sparse MoE composition is essential for maintaining high expressivity per FLOP and thus high memory–quality Pareto efficiency.

Inference savings are substantial when early exits are deployed only at loop boundaries. Relative increases in perplexity for a given drop in FLOPs are lower than in standard models: e.g., at 10% FLOP savings, Looped-MoE’s perplexity rises negligibly (from 34.8 to 51.0), and can drop further with more, finer-grained loops (Lee et al., 9 May 2026).

5. Theoretical Underpinnings and Scaling Behavior

Looped architectures exploit the equivalence between iterative depth, parameter sharing, and expressive acting via MoE or sparsity. In these models, individual loop passes select different subsets of experts, and diverging routing indices ensure a rich set of computations not possible in purely dense, shared-weight stacks.

At fixed compute, Looped-MoE and iso-FLOP Base models achieve similar, superior scaling. Comparative downstream evaluations indicate that Looped-MoE architectures not only scale as favorably as standard transformers but can dominate under strict memory or parameter constraints. Dense LoopLMs, in contrast, systematically underperform unless equipped with MoE (Lee et al., 9 May 2026).

6. Implementation Recommendations and Best Practices

Empirical and theoretical analysis recommend the following:

• Always substitute dense FFNs in looped models with sparse, top-$R$1 MoE (e.g., $R$2, $R$3).
• Use $R$4P scaling for width transfer and zero-shot hyperparameter adaptation.
• Place early exits only at loop boundaries, tuning entropy thresholds for required compute/perplexity.
• Prefer increasing the number of loops $R$5 (with smaller $R$6, keeping $R$7 constant) to maximize early-exit opportunities.
• Do not deploy dense LoopLMs at scale without a sparse or expert-based augmentation, as parameter sharing alone cannot recover lost expressivity.

7. Impact and Future Directions

Looped LLMs with sparse MoE layers define a leading approach for parameter- and compute-efficient scaling of transformers. Their design allows for models which, under robust compute and memory constraints, can outperform standard transformers in both language modeling and downstream tasks. The natural compatibility with early-exit policies at loop boundaries enables further inference efficiency gains. Ongoing research investigates hybrid looped–expert architectures, finer granularity in loop scheduling, and further improvements in routing divergence for next-generation scaling (Lee et al., 9 May 2026).

---

Key references:

• “Sparse Layers are Critical to Scaling Looped LLMs” (Lee et al., 9 May 2026)