Preconditioner design for slow momentum

Develop and evaluate preconditioners specifically designed for the slow-momentum component of flat-direction multiscale momentum, with the aim of further accelerating large-language-model pretraining.

Background

MuonM applies Muon preconditioning to the fast momentum and uses row-wise normalization for the slow momentum rather than a specialized preconditioner. The paper motivates this choice as computationally efficient and reports substantial empirical gains, but it does not investigate whether a more suitable preconditioner for the slow-momentum component would yield additional acceleration.

The unresolved problem concerns the design of such a preconditioner while preserving the stability benefits of restricting slow momentum to estimated flat directions and using sphere constraints.

References

Second, we have not investigated preconditioner design for the slow momentum, which we posit could further accelerate training.

— Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining  (2608.28442 - Zhu et al., 28 Aug 2026) in Section 8, Conclusion and Discussion, p. 14