Persistence of the LoopMoE performance gap through full training

Determine whether the performance advantage of the 3B LoopMoE model over its matched vanilla Mixture-of-Experts baseline persists through full training rather than only at the reported 200-billion-token checkpoint.

Background

The related-work discussion describes LoopMoE as a closely matched feedforward comparison: its 3B model uses a prelude–loop–coda structure and is compared with a vanilla Mixture-of-Experts model under matched parameter and compute conditions. Although LoopMoE leads on eight of nine benchmarks at 200 billion training tokens, the cited 9B comparison is only an early checkpoint.

The authors explicitly state that persistence of the larger-model performance gap through full training has not been established.

References

The persistence of the 9B gap through full training remains unresolved.

— Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design  (2610.23036 - Huang et al., 16 Oct 2026) in Appendix D, paragraph “LoopMoE provides the closest feedforward comparison”