Looped-MoE Models Overview

• The paper demonstrates that Looped-MoE models reuse transformer blocks with sparse MoE feed-forward networks, achieving increased effective depth and diversified routing.
• Looped-MoE models are defined by repeated application of a shared transformer block paired with top-k expert selection, enabling parameter sharing while scaling capacity.
• These architectures optimize compute efficiency with adaptive early-exit strategies and memory savings, showing improved performance on reasoning and math tasks compared to dense models.

Looped-MoE models are sparse LLMs that combine repeated computation through depth with Mixture-of-Experts (MoE) feed-forward sublayers. In the strongest training-time formulations, a shared stack or block of transformer layers is reused across loop passes while the FFN position is replaced by sparse top-$k$ experts, so the same physical layer can realize different effective computations on different passes through expert selection (Lee et al., 9 May 2026). Closely related lines of work extend the category in two directions: a training-free inference-time wrapper that loops a contiguous mid-stack window of a frozen sparse MoE checkpoint without architectural changes (Chen et al., 22 May 2026), and “Expert Tying,” which shares only expert tensors across consecutive layers while keeping attention and routing layer-specific as a “partial loop” (Jaggi, 15 Jun 2026). The underlying motivation is aligned with the broader looped-transformer thesis that many reasoning problems need depth more than distinct parameters, so effective depth can be supplied by iteration rather than by fresh layers (Saunshi et al., 24 Feb 2025). At the same time, not every sparse model with variable compute is looped: Grove MoE is explicitly a single-pass sparse MoE Transformer layer with heterogeneous grouped augmentation, not a looped-MoE architecture (Wu et al., 11 Aug 2025).

1. Conceptual scope and taxonomy

A looped LLM reuses a stack of physical layers repeatedly through depth rather than allocating a fresh set of parameters at each depth position. In the notation used for looped transformers, if a model has $L$ unique physical layers and $R$ loops, then its effective depth is $L \times R$, and the model computes repeated self-composition of the shared block (Lee et al., 9 May 2026). Looped-MoE models inherit this parameter-sharing-over-depth structure but place sparse MoE FFNs in the feed-forward position, which creates a second scaling axis: total parameter capacity can increase through experts while active per-token compute remains controlled by sparse routing (Lee et al., 9 May 2026).

A useful reading of the recent literature is that “Looped-MoE” is no longer a single architecture but a family of depth-sharing regimes. One branch trains looped sparse models end to end; another retrofits loops onto already trained sparse MoE checkpoints at inference time; a third shares only experts across neighboring layers rather than looping the entire block. This suggests that the field is organized less by one canonical implementation than by a common question: how much effective depth can be recovered from weight reuse before expressivity, routing stability, or systems efficiency breaks down.

Regime	Mechanism	Representative paper
Training-time Looped-MoE	Shared transformer layers reused across depth with sparse MoE FFNs	(Lee et al., 9 May 2026, Chen et al., 3 Jun 2026)
Training-free looping	Frozen checkpoint; contiguous mid-stack window re-applied at inference	(Chen et al., 22 May 2026)
Partial loop / Expert Tying	Experts shared across consecutive layers; attention and routing remain layer-specific	(Jaggi, 15 Jun 2026)
Non-looped adaptive sparse MoE	Single-pass grouped adjugate experts with token-dependent active parameters	(Wu et al., 11 Aug 2025)

The historical motivation predates sparse looping itself. A dense looped-transformer study argued that many reasoning problems need depth more than parameters, showed that a $k$-layer transformer looped $L$ times can nearly match a $kL$-layer non-looped model on addition, $p$-hop induction, and synthetic math, and connected looping to latent internal thought steps (Saunshi et al., 24 Feb 2025). Later sparse work effectively splits the dense argument into two orthogonal resources: MoE contributes parameter capacity, while looping contributes sequential computation.

2. Architectural forms of looping with sparse experts

In the canonical Looped-MoE construction, all compared models use a decoder-only transformer with multi-head self-attention, RoPE positional embeddings, pre-RMSNorm, residual stream, and SwiGLU feed-forward blocks; the decisive variation is dense FFN versus sparse MoE FFN, and unique layers versus looped/shared layers (Lee et al., 9 May 2026). The Looped-MoE variant is the looped transformer in which the dense FFN in each transformer block is replaced by a sparse MoE FFN. The physical layer remains RMSNorm, MHSA, residual, RMSNorm, MoE FFN, residual, and the same physical block is reused across loops (Lee et al., 9 May 2026).

The formal looped recurrence is written as

$$h^{(0)} = x,\qquad h^{(r,\ell)} = f_\ell\!\left(h^{(r,\ell-1)}\right),$$

with $r=1,\dots,R$, $L$0, and equivalently

$L$1

where $L$2 (Lee et al., 9 May 2026). In the main scaling study, the standard dense baseline “Base” has 16 unique layers, “Looped” has $L$3, “MoE” has 16 unique layers with sparse FFNs, and “Looped-MoE” has $L$4 with sparse FFNs, all in an isoFLOP regime (Lee et al., 9 May 2026).

LoopMoE generalizes the same idea into a sandwich-style looped sparse transformer with non-shared prefix layers $L$5, a shared loop block of $L$6 layers executed $L$7 times, and non-shared suffix layers $L$8, giving effective depth

$L$9

and unique depth

$R$0

Its loop is pure recursion rather than a skip-reinjected sandwich. If $R$1 is the hidden state after the prefix, the recurrent core starts at $R$2 and iterates

$R$3

with iteration- and token-dependent conditioning $R$4 (Chen et al., 3 Jun 2026). At 3B scale the model uses $R$5, $R$6, $R$7, $R$8, so it has 6 unique layers but 12 effective layers; at 9B it uses $R$9, $L \times R$0, $L \times R$1, $L \times R$2, giving 9 unique layers and 18 effective layers (Chen et al., 3 Jun 2026).

Expert Tying occupies a distinct point in this design space. Instead of looping an entire block, it aliases expert FFN weights across consecutive layers while keeping self-attention, routing, and normalization layer-specific. If layers are partitioned into groups of size $L \times R$3, then the gate, up, and down projections of the $L \times R$4 experts are aliased across all $L \times R$5 layers, reducing unique FFN parameters by a factor of $L \times R$6; each layer still computes its own routing distribution and its own attention output (Jaggi, 15 Jun 2026). The authors describe this explicitly as a “partial loop,” which is similar to recurrent parameterization over depth in the sense of reuse, but not a recurrent core executed multiple times at inference.

3. Routing dynamics, symmetry breaking, and stability

Routing behavior is the central mechanistic issue in looped sparse models. In the Looped-MoE scaling study, the paper uses top-$L \times R$7 token-choice routing with total experts $L \times R$8 and active experts per token $L \times R$9. Its main claim is that dense looped transformers underperform because a dense FFN applies the same transformation to every token each time the physical layer is reused, whereas in Looped-MoE the router can choose different experts on each loop pass and thereby recover expressivity without additional parameters (Lee et al., 9 May 2026). The direct evidence is a routing-overlap analysis on the compute-optimal $k$0 Looped-MoE at $k$1 FLOPs: across physical layers 1–6 and 8, 25–53% of tokens have zero overlap between pass-1 and pass-2 top-2 experts, only 4–14% have exact match, and the majority often show partial overlap. Layer 7 is an outlier with 37% exact match and only 3% zero overlap (Lee et al., 9 May 2026). The authors interpret this as routing divergence across loops, with some layers more loop-invariant than others.

LoopMoE reaches a similar objective by a different route. It identifies two problems in naively looped sparse models: weight-sharing symmetry and asymmetric active-capacity growth. To break symmetry, it introduces IterAdaLN, which jointly conditions on iteration index and per-token hidden state, separately for attention and MoE branches. The iteration stream comes from sinusoidal encoding of the loop index plus a learned MLP, while the token stream comes from branch-specific linear projections of the current hidden state (Chen et al., 3 Jun 2026). The paper further adds a residual gate $k$2 only on the attention branch, arguing that an MoE-side residual gate adds no real expressive power because it can be absorbed into tokenwise routing weights and could interfere with load-balancing calibration (Chen et al., 3 Jun 2026). The routing analysis then finds a three-phase structure: the entry phase is distinct, the recurrent core shows a relatively stable expert coalition, and the final iteration keeps representations close in cosine space but lowers expert-set Jaccard similarity, suggesting reformatting for the suffix (Chen et al., 3 Jun 2026).

Training-free looping exposes a different stability pathology. Reapplying a block naively as

$k$3

typically degrades performance because it advances the hidden state farther along a trajectory the post-loop layers were never trained to consume. The proposed correction treats the looped block as a residual field $k$4 and replaces one large update with smaller damped sub-steps,

$k$5

which the paper interprets as forward Euler refinement of the same endpoint rather than repeated full advancement (Chen et al., 22 May 2026). For sparse MoE backbones, the more important distinction is between block-mode and layer-mode. In block-mode, each reapplication perturbs the hidden state seen by every MoE gate in the window, so token-to-expert assignments change across iterations; the paper calls this “routing thrash.” In layer-mode, by contrast, the gating decision is computed once per layer and the same expert mixture is reused across the $k$6 repeats of that layer, making layer-mode “the correct default for MoE backbones” (Chen et al., 22 May 2026).

Expert Tying sharpens the question of what actually differentiates repeated sparse computation. Its routing diagnostic is cross-loop agreement, defined as average top-1 expert agreement across layers within the same tying group. Untying the router drastically reduces agreement, meaning routers choose more different experts across tied layers, but this barely changes validation loss; untying attention yields the much larger loss improvement (Jaggi, 15 Jun 2026). In the representative $k$7 ablation, all-tie has loss 3.511 and agreement 0.733, attn-tie has loss 3.512 and agreement 0.160, and expert-tie has loss 3.469 and agreement 0.121 (Jaggi, 15 Jun 2026). The conclusion is explicit: per-layer attention, not routing, is what differentiates layers. This creates a productive tension with the full-loop Looped-MoE result: routing divergence clearly matters when the entire block is reused, but under partial expert-only reuse, attention carries most of the depth-specific computation.

4. Scaling behavior, memory economy, and adaptive inference

The principal empirical claim of the scaling-law study is that sparse layers are critical to scaling looped models. Under matched active parameter count and matched compute, dense looped transformers scale worse than standard dense transformers, standard non-looped MoE remains best in pure test loss, but Looped-MoE beats the dense Base model and preserves the practical benefits of looping: lower stored parameters and natural early-exit points (Lee et al., 9 May 2026). Across budgets $k$8 FLOPs, the paper fits power laws $k$9 and reports $L$0 for Base and $L$1 for Looped-MoE, so the advantage is mostly an offset improvement rather than a dramatically different exponent (Lee et al., 9 May 2026).

The stored-versus-active distinction is fundamental in all sparse looped work. The scaling study uses $L$2 for compute accounting, with

$L$3

and $L$4, not stored parameters (Lee et al., 9 May 2026). This is why Looped-MoE can match active compute while storing fewer weights than the dense Base. At the largest budget, the AI2 OLMES Core 9 downstream table gives stored parameters of 246M for Base and 216M for Looped-MoE, while Looped-MoE also achieves the higher downstream average (Lee et al., 9 May 2026). LoopMoE makes the accounting caveat even more explicit: its reported active-parameter number is FLOP-matched rather than the number of physically distinct active parameters. At 3B both models are reported as 3B total / 0.8B active, but LoopMoE’s actual physical active parameter count is 0.6B; at 9B both are reported as 9B total / 1.9B active, while LoopMoE’s actual physical active count is 1.3B (Chen et al., 3 Jun 2026).

Expert Tying focuses directly on parameter memory. In scaled-down production models with $L$5, total parameters fall from 523M to 296M for DeepSeekMoE and OLMoE, a 43% saving, and from 459M to 220M for Qwen3-MoE, a 52% saving. In the full-scale OLMoE-1B-7B experiment, the baseline 7.12B model becomes 3.50B under $L$6, a 50.9% saving (Jaggi, 15 Jun 2026). These reductions are accompanied by throughput gains: 15.7% in a small configuration, and at large scale 51,777 tokens/sec versus 41,859 tokens/sec, a 23.7% speed-up (Jaggi, 15 Jun 2026). The paper attributes this to less weight-loading bandwidth, smaller optimizer-state updates, reduced DDP communication, and lower memory pressure (Jaggi, 15 Jun 2026).

Looping also changes adaptive inference. The Looped-MoE study argues that loop boundaries are natural early-exit points because each loop ends with the same layers that produce the final output, unlike an arbitrary intermediate layer in a non-looped transformer (Lee et al., 9 May 2026). Under a training-free entropy-based early-exit rule, looped architectures have better compute-quality trade-offs than non-looped ones. At 10% FLOPs saved, perplexities are 55.4 for Base, 75.7 for MoE, 50.2 for Looped, and 51.0 for Looped-MoE; at 20% FLOPs saved, they are 112.6, 167.5, 84.3, and 89.7 respectively (Lee et al., 9 May 2026). More aggressive looping improves the early-exit frontier further: for Looped-MoE, $L$7 is better than $L$8, and $L$9 is better than both, although the paper does not extend the full training-time scaling-law study to those higher loop counts (Lee et al., 9 May 2026).

Training-free looping pays a localized compute surcharge rather than whole-model re-execution. The added cost is $kL$0 extra forward passes through the loop window, and in a wall-clock profile on Qwen3-4B-Instruct with $kL$1 and a 4-layer window, full decode-time looping adds about +21.6% wall-clock (Chen et al., 22 May 2026). The paper reports approximate single-GPU bf16 memory footprints of $kL$2 GB for Qwen1.5-MoE-A2.7B, $kL$3 GB for DeepSeek-V2-Lite and Moonlight-16B-A3B, and $kL$4 GB for Qwen3-30B-A3B-Instruct, but it does not isolate expert-parallel communication overhead from looping overhead (Chen et al., 22 May 2026).

5. Empirical results and task profile

On downstream quality, Looped-MoE is not merely a memory-saving trick. In the $kL$5-FLOP scaling experiment, AI2 OLMES Core 9 averages are 37.4 for Looped, 38.7 for Base, 39.6 for Looped-MoE, and 36.4 for standard MoE (Lee et al., 9 May 2026). The especially important comparison is that Looped-MoE beats Base while storing fewer unique parameters. The same paper notes a curious asymmetry: standard MoE has the best test loss but the weakest downstream average in the full benchmark table, and speculates that Looped-MoE may benefit from seeing more unique experts across loops because routing changes between passes (Lee et al., 9 May 2026).

LoopMoE reports a stricter head-to-head comparison against a matched Vanilla MoE under identical total parameters, per-token FLOPs, active sublayer ratios, training data, optimizer, schedule, document ordering, and evaluation protocol (Chen et al., 3 Jun 2026). At 3B scale, Vanilla MoE obtains MMLU 27.86, HellaSwag 58.68, PIQA 74.12, WinoGrande 49.92, TriviaQA 36.19, RACE 35.00, BBH 26.12, GSM8K 4.12, MATH 1.53, average 34.84; LoopMoE obtains 26.81, 60.98, 75.22, 51.42, 37.63, 35.77, 27.94, 5.26, 1.88, average 35.89, winning on 8 of 9 benchmarks with an average gain of 1.05 (Chen et al., 3 Jun 2026). At 9B and an early-training checkpoint of 100B tokens, Vanilla MoE averages 35.02 and LoopMoE averages 36.17, again winning on 8 of 9 with a 1.15-point gain; MMLU is the only regression in both scales (Chen et al., 3 Jun 2026). The largest gains are concentrated on reasoning and mathematics, especially GSM8K at 9B (Chen et al., 3 Jun 2026).

The ablation structure in LoopMoE is notable because it separates the contribution of looping itself from symmetry breaking and capacity balancing. On the subset $kL$6MMLU, HellaSwag, BBH, GSM8K$kL$7, the bare loop already improves BBH from 26.12 to 27.33 and GSM8K from 4.12 to 5.34. IterAdaLN adds a modest gain on HellaSwag but hurts MMLU in that snapshot, and the final balancing step gives the largest HellaSwag boost while recovering MMLU and BBH relative to the prior row (Chen et al., 3 Jun 2026). The paper interprets this as evidence that repeated computation itself captures much of the gain on reasoning and math, while active attention-to-FFN ratio matching is important for overall quality.

The training-free wrapper produces smaller but practically relevant gains on released sparse checkpoints. For Qwen1.5-MoE-A2.7B, the paper reports ARC-Challenge $kL$8 (+2.30 pp), CommonsenseQA $kL$9 (+1.72 pp), and OpenBookQA $p$0 (+1.60 pp). For Moonlight-16B-A3B it reports MMLU $p$1 (+0.74 pp) and OpenBookQA $p$2 (+1.20 pp). For DeepSeek-V2-Lite-Chat it reports ARC-Challenge $p$3 (+0.85 pp). For Qwen3-30B-A3B-Instruct it reports CommonsenseQA $p$4 (+1.14 pp), plus a broader untuned sweep at $p$5 with positive changes on ARC-Easy, HellaSwag, SciQ, TruthfulQA, MMLU-flex, GPQA-Main, and SuperGPQA while preserving or slightly improving LAMBADA PPL (Chen et al., 22 May 2026). The authors summarize that the strongest and most reliable gains overall are on knowledge-heavy multiple-choice tasks and that MLA-based MoE models move in the same direction but with smaller magnitudes than the strongest dense MHA models (Chen et al., 22 May 2026).

The broader dense-loop background helps interpret this task profile. A dense looped-transformer study found that looping does not fully recover perplexity or memorization relative to an iso-FLOP dense baseline, but it disproportionately improves reasoning-heavy downstream tasks and can exceed the deeper dense baseline on reasoning primitives (Saunshi et al., 24 Feb 2025). This suggests a coherent division of labor across looped-MoE research: sparsity is repeatedly used to restore or expand capacity under shared depth, while looping is repeatedly associated with reasoning, iterative refinement, or improved early exits.

6. Non-examples, misconceptions, and open problems

A recurring misconception is to equate any adaptive sparse MoE with a looped-MoE. Grove MoE is the clearest counterexample. Its novelty is heterogeneous expert augmentation via grouped shared “adjugate experts” and token-dependent variation in activated parameters, but routing still happens once per MoE layer, top-$p$6 experts are selected once, and selected experts are evaluated in parallel (Wu et al., 11 Aug 2025). There is no iterative refinement step, no recurrence over experts, no repeated application of the same expert block within a layer, no adaptive halting, and no recursive or staged expert hierarchy (Wu et al., 11 Aug 2025). It is therefore best described as a non-looping adaptive sparse MoE baseline.

A second misconception is that one can obtain looped-MoE behavior by simply reapplying blocks more times. The training-free study is explicit that naive $p$7-fold looping degrades performance almost universally, and that wide windows or whole-model looping can collapse badly (Chen et al., 22 May 2026). In sparse backbones, the dominant failure mode is routing thrash, which is why layer-mode is the safe default for MoE checkpoints (Chen et al., 22 May 2026). Expert Tying presents an analogous architectural caution from the training side: full-stack tying in a MoEUT-style extreme is much worse than middle-stack tying with an untied $p$8 prelude/coda, leading the authors to recommend preserving at least a thin untied beginning and end (Jaggi, 15 Jun 2026).

The present evidence base is substantial but still bounded. The Looped-MoE scaling-law work reaches active parameter counts up to roughly 305M and stored parameter counts up to 711M for non-looped MoE, then extrapolates larger-scale behavior through scaling laws (Lee et al., 9 May 2026). LoopMoE directly evaluates 3B and 9B scales, with the 9B comparison reported only at a 100B-token early checkpoint (Chen et al., 3 Jun 2026). Expert Tying reaches full-scale OLMoE-1B-7B and shows that $p$9 tying can be almost free there, but explicitly leaves frontier-scale and longer-horizon pretraining open (Jaggi, 15 Jun 2026). The training-free wrapper is deliberately inference-only and therefore does not analyze MoE load-balancing losses, expert-capacity constraints, routing regularizers under looping, or expert-parallel communication effects in detail (Chen et al., 22 May 2026).

The most important open technical question is which component should carry iteration-specific distinctness. Full-loop studies emphasize routing divergence across passes or explicit iteration conditioning; partial-loop studies argue that untied attention already provides most of the required differentiation and that routing diversity alone matters relatively little for loss (Lee et al., 9 May 2026). This suggests that “looped-MoE” is not a single settled recipe but a design space with at least three viable answers: full shared sparse blocks with loop-dependent routing, inference-time looping with pinned routing for stability, and expert-only partial loops with per-layer attention and routers. A plausible implication is that future progress will depend less on whether depth is shared at all than on where the model places its depth-specific degrees of freedom.