Phases of Muon: When Muon Eclipses SignSGD

Abstract: Recently, Muon and related spectral optimizers have demonstrated strong empirical performance as scalable stochastic methods, often outperforming Adam. Yet their behaviour remains poorly understood. We analyze stochastic spectral optimizers, including Muon, on a high-dimensional matrix-valued least squares problem. We derive explicit deterministic dynamics that provide a tractable framework for studying learning behaviour with a focus on (stochastic) SignSVD, which Muon approximates, and (stochastic) SignSGD, the latter serving as a proxy for Adam. Our analysis shows that for large batch size, SignSVD performs a square-root preconditioning with respect to the data covariance spectrum, while for small batch size smaller eigenmodes behave like SGD, slowing down convergence. We contrast with SignSGD which for generic covariance performs no preconditioning and has no transition, leading to different optimal learning rates and convergence characteristics. The two methods match up to a constant factor with isotropic data, but behave differently with anisotropic data. An analysis of a power law covariance model with data exponent $α$ and target exponent $β$ shows there are three phases in the $(α,β)$ plane: one where SignSGD is uniformly favored, one where SignSVD is uniformly favored, and a third where the two methods exhibit a trade-off in performance.

• The paper presents an RMT-based framework to derive deterministic risk trajectories and optimal learning rates for both spectral and diagonal optimizers.
• It reveals phase transitions in high-dimensional regimes, delineating 'easy', 'hard', and 'mixed' learning phases based on covariance anisotropy and batch size.
• Numerical experiments validate that while spectral preconditioning excels in anisotropic settings, diagonal methods perform optimally in isotropic regimes.

Analysis of "Phases of Muon: When Muon Eclipses SignSGD" (2605.09552)

Context and Motivation

The study addresses a central open problem in scalable stochastic optimization for deep learning: the relative behavior and performance of spectral (non-diagonal) optimizers, exemplified by the MUON algorithm and its theoretical limit SIGNSVD, versus diagonal methods such as ADAM and its proxy SIGNSGD. Although MUON and spectral methods have demonstrated empirical advantages, particularly in large-scale matrix optimization problems, the mechanisms underlying this performance remain theoretically unquantified, especially in high-dimensional and mini-batch regimes with realistic, potentially anisotropic data. The paper provides a unified and rigorous framework based on random matrix theory (RMT) to characterize optimizer dynamics, learning rates, and their dependence on problem and data statistics, with an explicit focus on transition phenomena and phase diagrams distinguishing "easy", "hard", and "mixed" learning regimes.

Model Setup and Algorithmic Classes

The analysis is grounded in a matrix-valued (multi-output) linear regression problem that retains the essential outer-product structure of neural network layer gradients. The global risk is a quadratic function of a $N_{\text{out}} \times N_{\text{in}}$ weight matrix $W$, with independent Gaussian data $x_\text{in}$ and $x_\text{out}$ and covariances $\Sigma_\text{in}, \Sigma_\text{out}$. This setting is sufficiently expressive to capture eigenstructure-sensitive effects in optimization.

The primary algorithms under rigorous analysis are:

• SIGNSVD: The idealized spectral optimizer, performing spectral whitening using the matrix sign (polar) function, $y(z) = z^{-1/2}$, at each step. This can be interpreted as a matrix generalization of sign-based gradient methods, but with non-diagonal preconditioning.
• MUON: A scalable, practical approximation to SIGNSVD, employing a truncated Newton–Schulz iteration to estimate the matrix sign transform. Heavy-ball momentum is considered but not included in the main theoretical results.
• SIGNSGD: The diagonal, entrywise sign-based optimization used as a tractable surrogate for ADAM (with $\beta_1 = \beta_2 = 0$).

Deterministic Dynamics and Random Matrix Theory

A central technical contribution is the derivation, via high-dimensional RMT, of explicit deterministic risk trajectories $R(t)$ that tightly track the stochastic risk of each algorithm in the proportional asymptotic regime (fixed aspect ratios $n/B$ as $n,B \to \infty$). This is achieved by reducing the problem to a system of closed drift–volatility recursions characterized by spectral integrals over the input/output data covariance spectra and batch dimension. The analysis carefully distinguishes regimes where the drift/volatility kernels are eigenmode-independent (isotropic case) or sensitive to covariance anisotropy (non-isotropic, e.g., power-law spectrum).

Crucially, the approach enables prediction of:

• Per-iteration contraction rates.
• Optimal learning rate scaling with $W$0, $W$1, and spectrum.
• Noise floor and time-to-threshold ($W$2-approximate solution) as explicit functions of algorithm/data regime.

Isotropic Regime

Under isotropic Gaussian data, drift and volatility kernels for both algorithms become simple, and the risk dynamics reduce to a scalar ODE. The key outcomes are:

• Learning Rate Scaling: The optimal learning rate for SIGNSVD exceeds that for SIGNSGD by a factor of $W$3 for $W$4 and $W$5 for $W$6, matching the previously observed "MOONSHOT rule" in large-scale neural network training.
• Convergence Rate: Both optimization methods achieve identical geometric convergence rates (up to constants) when properly tuned, and no spectral preconditioning speedup is possible in this case.
• Noise Floor: The noise-induced residual risk limit is algorithm-independent up to absolute constants.

This demonstrates that–contrary to some prior beliefs–matrix-valued spectral preconditioning does not universally improve performance in isotropic regimes; instead, proper adjustment of the learning rate suffices.

Anisotropic and Power-law Covariance Regimes

When the output covariance is anisotropic—especially with a power-law spectral decay—substantially richer phenomena emerge:

• Critical Batch Effect for SIGNSVD: The fraction of spectrum "resolved" by a batch is $W$7 for fast-decaying ($W$8) spectra; as $W$9 increases, more eigenmodes are whitened, with a full transition to optimal preconditioning at $x_\text{in}$0. For $x_\text{in}$1, only top modes are efficiently preconditioned, the remainder behave as in unpreconditioned SGD.
• Spectral Acceleration: In the resolved regime ($x_\text{in}$2), SIGNSVD acts (up to scaling) as a square-root preconditioner, matching the optimal acceleration for strongly convex quadratics.
• Comparison with SIGNSGD: In the presence of output anisotropy, SIGNSGD is blind to the spectrum up to random Haar bases, and does not gain from non-diagonal structure, unless the basis is aligned with the data.

A detailed analysis for power-law output spectra ($x_\text{in}$3) reveals three distinct phases in the $x_\text{in}$4 parameter space (where $x_\text{in}$5 is the target exponent):

• Phase A ($x_\text{in}$6): SIGNSVD is strictly superior; time-to-threshold for SIGNSGD is polynomially slower in $x_\text{in}$7.
• Phase B ($x_\text{in}$8): A mixed regime where, at large $x_\text{in}$9, SIGNSGD is initially faster, but as lower thresholds are targeted, SIGNSVD overtakes (crossover).
• Phase C ($x_\text{out}$0): Both methods saturate to the same asymptotic scaling in $x_\text{out}$1, but with a performance advantage to SIGNSGD.

These transitions are governed by how the algorithms’ drift kernels interact with the eigenvalue distribution, leading to sharp exponents in time-to-$x_\text{out}$2 and clarifying the regimes where spectral methods are essential.

Numerical and Theoretical Validation

The RMT-predicted curves and phase boundaries are validated through extensive numerical experiments, showing quantitative agreement with both risk trajectories and scaling exponents, both for idealized SIGNSVD and for stochastic MUON with finite Newton–Schulz depth.

Notably, the deterministic dynamics accurately predict:

• The scaling of optimal learning rates.
• The risk convergence curves across different regimes (matching numerics at $x_\text{out}$3 as small as $x_\text{out}$4).
• The crossovers in dominance between SIGNSVD and SIGNSGD in different spectral and batch-size regimes.

Practical and Theoretical Implications

This work provides a mathematically principled, fine-grained blueprint for understanding when spectral preconditioning (e.g., via MUON) can outpace diagonal adaptivity (e.g., ADAM/SIGNSGD). The results show that:

• There is no universal dominance of spectral methods; practitioners must account for data spectrum, batch size, and alignment.
• Properly tuned, SIGNSGD (and hence ADAM-like methods) is essentially optimal in isotropic and easy regimes; however, non-diagonal methods are indispensable for high-rank, bias-dominated, or strongly anisotropic data.
• The possibility of crossovers in performance—with diagonal methods dominating initially, but spectral methods winning for tighter optimization thresholds—suggests nuanced algorithmic scheduling.

Future directions proposed include the extension to momentum, adaptive learning rates, more realistic neural networks with feature learning, and direct incorporation of edge-of-stability phenomena. Furthermore, the explicit link between spectral preconditioning and phase transitions in learning (as flagged by the sharp exponents in the scaling laws) opens a path toward principled compute-optimal training strategies in LLMs and high-dimensional inference.

Conclusion

This paper substantially advances the quantitative theory of stochastic optimization dynamics for matrix-valued problems, moving beyond single-step or worst-case analyses. By leveraging explicit RMT-based deterministic equivalents, it unearths intricate phase behavior, precise optimizer dependence on spectrum, and concrete conditions for the practical efficacy of non-diagonal methods. The resulting framework not only demystifies when and why MUON (or similar spectral optimizers) outpace ADAM-like adaptivity in deep learning, but also points toward a deeper principled understanding of optimization and scaling in overparameterized regimes.