Loop the Loopies!

Abstract: We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

• The paper introduces layer-loop recurrence to improve compute efficiency and parameter sharing in MoE Transformers.
• It proposes the Loopie Recipe, a compute-matched scaling framework that boosts throughput while maintaining fixed pre-training budgets.
• Empirical benchmarks reveal that Loopie-20B-A2B achieves competitive scores (e.g., 81.28 on MMLU) and outperforms non-recurrent baselines.

Analysis of "Loop the Loopies!" (2607.16051)

Motivation and Background

Looped Transformers, originating from the Universal Transformer paradigm, have recently gained interest due to their capacity for efficient parameter sharing and recurrent computation. Historically, the main challenge for looped Transformers has been justifying their use not in terms of parameter efficiency—which is often clear—but in terms of compute efficiency under fixed pre-training budgets. Prior work established that, given $N$-fold more compute, simply increasing parameter count typically outperforms looping a shallower network $N$ times. As a consequence, looped computation has yet to become a widely accepted scaling axis in practical large-scale LLM training.

"Loop the Loopies!" addresses this gap directly. The Loopie series consists of two large-scale Mixture-of-Experts (MoE) looped Transformer models: Loopie-20B-A2B (20B total, 2B active parameters) and Loopie-6B-A0.6B (6B total, 0.6B active). The novel contributions include the Loopie Recipe for compute-matched scaling, layer-loop recurrence (as opposed to traditional model-loop), and a high-throughput post-training pipeline for competitive mathematical and scientific reasoning abilities.

Looped Transformer Architecture and the Layer-Loop Paradigm

The key innovation is the layer-loop recurrence pattern.

Model-loop (classic): Recursively applies the entire layer stack; i.e., all layers are traversed, then the stack is repeated for $R$ steps.

Layer-loop (Loopie): Applies each layer multiple times in succession before moving to the next layer, enhancing execution locality and more coherently sharing parameters across adjacent depths.

Empirically, the authors demonstrate that model-loop offers initial gains but is ultimately outperformed by layer-loop as pre-training advances. The improvement, attributed to layer-loop, results from a more local and coherent sharing of parameters—a property consistent with the nonuniform distribution of linguistic abstraction across transformer depth, as emphasized in recent probing studies.

Figure 1: Average downstream benchmark scores comparing layer-loop and model-loop in Loopie-6B-A0.6B; layer-loop overtakes model-loop for large-token budgets.

The practical advantage of layer-loop also extends to training efficiency. The shorter reuse distance for parameters and gradients under layer-loop naturally aligns with pipeline parallelism and memory sharding, maintaining high device utilization and minimizing communication overhead.

Compute-Matched Scaling: The Loopie Recipe

To address the core issue—compute-optimal scaling—the paper introduces the Loopie Recipe. Three stages are delineated:

1. Recurrent seed reduction: Halve the stored depth of the non-recurrent MoE baseline and set two layer-loops ($R=2$).
2. Loop unrolling: Double the per-device microbatch size using the reduced activation memory (as recurrent applications share activations in checkpointed blocks), which increases throughput.
3. Capacity reinvestment: Use the resultant microbatch efficiency gain to increase model width/depth, yielding a model whose optimizer-step time matches the original baseline under the same hardware.

Crucially, models are not matched by analytical FLOPs but by measured optimizer-step wall-clock time in Megatron-LM. This hardware-aware matching accounts for real-world bottlenecks such as kernel launch overhead, communication, and activation storage.

Figure 2: Compute-matched comparison of Loopie-20B-A2B and Qwen3-30B-A3B; Loopie achieves higher downstream performance and throughput under matched per-step wall time.

The Loopie Recipe therefore enables looped models to be compared fairly to non-recurrent baselines at matched compute, not just for fixed parameter budgets.

Empirical Results and Scaling Behavior

Loopie is evaluated using strong baselines (e.g., Qwen3-30B-A3B). Initially, looped models lag; after a certain token threshold, however, they consistently overtake non-looped counterparts in downstream benchmarks.

A scaling ladder experiment shows that, over model sizes from sub-billion to multi-billion parameters, compute-matched looped models consistently outperform non-recurrent models with identical pre-training wall time.

Figure 3: Scaling ladder comparison; at every run, compute-matched Loopie models (sized per the Loopie Recipe) surpass non-recurrent baselines in downstream accuracy.

Ablations isolate the benefit of layer-loop: removing layer-loop from Loopie-6B-A0.6B substantially hurts performance, even when active parameter count and compute budget are held constant.

Figure 4: Effect of ablating layer-loop in Loopie-6B-A0.6B; clear degradation in average downstream benchmark performance.

Loop Depth and Marginal Returns

The study systematically examines the marginal benefit of increasing loop steps $R$ under a fixed compute budget. Their findings indicate diminishing returns beyond $R=2$ in large-scale pre-training: additional recurrence gives lower gains than simply investing compute in additional non-shared capacity.

Figure 5: Comparison between $N\times$ layer-loop and $N\times$ stored-layer scaling; recurrence gains saturate at $R=2$.

Consequently, $R=2$ is chosen for all Loopie models, positioning recurrence as a practical architecture-level refinement rather than an open-ended loop-depth scaling axis.

Data, Pre-Training, and Post-Training

Pre-training is executed in two stages: multi-epoch high-quality pre-training on Nemotron-CC-v2-HQ, followed by an annealing phase using a high-quality data mixture emphasizing STEM, math, code, and code-reasoning. Figure 6 details the Stage-2 mixture composition.

Figure 6: Token distribution of Stage-2 high-quality data annealing pool across various synthetic, SFT, code, web, and math sources.

Large-scale post-training comprises "Supervised Pre-Training" (SPT) and mathematical/coding RL fine-tuning:

• SPT aligns with SFT-like loss criteria but operates over multi-trillion token budgets typical of LLM pre-training, with large batch and context.
• RL leverages advanced sequence-level policy optimization with dynamic filtering and reward normalization (GSPO/DAPO), further increasing mathematical and programmatic reasoning capabilities.

The models show smooth loss convergence in pre-training and SPT (Figure 7), and monotonic improvement on reasoning and pre-training benchmarks across an unprecedented data volume (Figures 10, 11).

Figure 7: Supervised pre-training loss curve for Loopie-6B-A0.6B on 2T tokens; loss decreases smoothly with no overfitting.

Figure 8: Trends in reasoning metrics during SPT; continued improvement through 2T tokens without signs of saturation.

Figure 9: ARC-Challenge and MMLU performance during SPT; both reasoning and general knowledge metrics improve during extended supervised training.

Benchmark Results

Loopie-20B-A2B attains SOTA/gold-medal-level scores on challenging benchmarks:

• $N$0 on MMLU, $N$1 on ARC-Challenge, $N$2 on AIME 2024, and $N$3 on AMC.
• Achieves gold-medal thresholds at the 2025 International Mathematical Olympiad (IMO) and International Physics Olympiad (IPhO) using only raw model output, not external tools.

Despite being pre-trained on only $N$4T tokens—far less than Nemotron 3 Nano and Cascade 2—Loopie matches or exceeds these models in nearly all core knowledge, general, code, and math tasks. This empirical result robustly supports the assertion that recurrent scaling, properly implemented, can yield compute-superior models relative to conventional best practices.

Theoretical and Practical Implications

The main theoretical advance is the demonstration that, under a proper hardware-aware design and compute-matching protocol, looped computation can be made competitive—or superior—to scaling model parameter count. This challenges the prevailing view that recurrence is suboptimal under realistic compute budgets, and establishes a practical blueprint for integrating looped architectures into large-scale training pipelines.

Notably, this result is only achieved within a specific regime: $N$5, large-batch SPT, and carefully matched memory/batch scaling. Moreover, the success depends on the synergy between activation memory management, pipeline parallelism, and parameter sharing achievable with the layer-loop pattern.

On the practical side, the Loopie method provides an immediate path for improving token efficiency and test-time generalization on reasoning tasks using open-source MoE architectures. The code is available, and model checkpoints can be used in research and deployment with standard toolchains.

Future Directions

Prospective research avenues include:

• Extending the looped compute-matched scaling protocol to more exotic backbones (dynamic recurrence, adaptive computation, etc.).
• Matching and optimizing inference-time compute in addition to pre-training compute.
• Integrating additional architectural advances alongside looped recurrence (e.g., improved memory tokens, explicit chain-of-thought supervision, hierarchical recurrence).
• Broadening post-training to further domains, including open-ended instruction following, scientific question answering, and robust alignment.

Conclusion

"Loop the Loopies!" provides compelling empirical and methodological evidence that layer-looped MoE Transformers, when carefully compute-matched, outperform non-recurrent baselines across a wide range of reasoning and general-language metrics, with robust improvements scaling to large model and data regimes. This challenges long-standing skepticism about the practical value of looped computation under constrained compute resources, positions layer-loop recurrence as a viable frontier-scaling axis for next-generation LLMs, and suggests new directions for both theory and large-scale model training practice.