Characterize Muon’s inductive biases and training trajectory

Determine the inductive biases of the Muon optimizer when training deep neural networks, characterize the trajectory it follows through the loss landscape that yields rapid convergence, and ascertain the implications of this trajectory for the properties of the final solution to which Muon-optimized models converge.

Background

The paper emphasizes that optimizer choice can fundamentally alter learning trajectories and the functional properties of solutions, beyond affecting training speed. Muon—an optimizer based on orthogonalizing gradient updates—has gained popularity due to strong empirical performance and faster convergence, yet its inductive biases and the path it takes in the loss landscape remain insufficiently understood.

The authors analyze a simplified variant, Spectral Gradient Descent (Spectral GD), to provide initial theoretical insights. Their results suggest Spectral GD removes the simplicity bias seen with gradient descent, offering speed benefits but potentially harming structural generalization in some settings. Despite these initial steps, fully characterizing Muon’s biases, its precise training trajectory, and the consequences for converged solutions is explicitly noted as unknown.