Establish the effect of high repetition on router z-loss

Determine whether high data-repetition rates systematically increase Mixture-of-Experts router z-loss, independently of the auxiliary-loss dynamics associated with extremely low training loss.

Background

The paper reports that higher repetition rates are associated with larger router z-loss values early in training and with repetition-driven cycles at the 200M- and 1B-active-parameter scales. However, the observed z-loss eventually falls at very high repetition rates. The authors identify an unresolved ambiguity: high repetition may directly increase router z-loss, or the unusually low training loss under high repetition may amplify auxiliary-loss optimization and subsequently drive z-loss downward.

References

It is possible that high repetition typically results in higher z-loss, but that the extremely low training loss at high $R$ results in a relatively strong optimization signal from auxiliary losses, eventually driving z-loss to fall.

— Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data  (2609.11917 - Jha et al., 10 Sep 2026) in Appendix, Section “Additional Results,” subsection “Routing Load Balance and Stability,” caption of the router z-loss figure