The Spectral Dynamics and Noise Geometry of Muon

Abstract: Muon replaces a matrix gradient $G=UΣV^\top$ by its polar factor $UV^\top$. This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. Under explicit alignment assumptions, we prove that the polar update is the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, we derive exact singular-value dynamics for continuous-time Muon and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. This geometry also rules out a common low-rank interpretation: at fixed Frobenius norm, Muon's distinguished state has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.

• The paper introduces the Spectral-Bias Theorem, rigorously characterizing Muon’s flat-spectrum update dynamic and differentiating it from classical minimization methods.
• It presents a flattening criterion and variational analysis through projected polar flows, linking singular value dynamics with noise sensitivity across matrix sensing and transformer pretraining.
• Empirical results validate that Muon preserves spectral diversity, informs optimizer scaling, and exhibits regime-dependent performance in both matrix regression and deep network training.

Spectral Dynamics and Optimization Bias in Muon

Overview

"The Spectral Dynamics and Noise Geometry of Muon" (2606.08388) rigorously investigates the theoretical and empirical properties of the Muon optimizer, focusing on its spectral bias, implicit regularization, and noise sensitivity. The study characterizes Muon's distinct flat-spectrum update dynamic in matrix regression, differentiating it from classical gradient descent, gradient normalization, and nuclear-norm minimization. The work leverages mathematical analysis of projected polar flows and introduces new criteria for spectral flattening, variational characterization, and noise geometry, supported by extensive controlled experiments across matrix sensing and transformer pretraining.

Theoretical Foundations

Muon replaces the gradient matrix $G=UEV^T$ with its polar factor $UV^T$, preserving the singular directions while enforcing uniform update amplitude. The paper formalizes the projected polar flow on the interpolation manifold $\mathcal{M} = \{W : XW = Y\}$, yielding exact singular-value dynamics and spectral flattening conditions. Key contributions include:

• Spectral-Bias Theorem: For bounded updates in shared singular-frame alignment, the polar profile maximizes first-order spectral entropy gain. This is proved without global assumptions, offering a geometric explanation for Muon's bias toward flat spectra (equal nonzero singular values).
• Flattening Criterion and Variational State: The spectral flattening criterion $dR_{pw}/dt < 0$ holds if and only if the update direction's alignment is concentrated on leading singular vectors. The unique minimizer of the pairwise spectral functional $R_{pw}$ at fixed Frobenius norm is the flat singular-value spectrum, distinctly incompatible with nuclear-norm minimization, which concentrates singular values.
• Robustness under Approximate Frame Alignment: The deviation of the projected gradient/momentum polar flow from the self-polar flow is bounded by the gauge-invariant polar misalignment, with precise decomposition in terms of singular frames.
• Discrete-Step Analysis: The paper provides finite-step bounds, showing that Muon's discrete dynamics match continuous-time predictions up to $O(\epsilon^2)$, provided spectral gaps are maintained.

Experimental Regimes and Numerical Results

Matrix Sensing

Muon is empirically falsified as a nuclear-norm minimizer: across random Gaussian matrix sensing instances, Muon's converged nuclear norm consistently exceeds that of convex solvers (mean gap $1.59\times$, supporting the theoretical separation). Muon solutions show greater spectral entropy and flatter singular-value profiles than nuclear-norm minima, ruling out late-stage convergence to low-rank solutions and confirming the flat-spectrum variational characterization.

Noise Geometry and Critical Batch Size

Theoretical derivation of the polar-map spectral sensitivity (using the Mathias formula) yields critical batch size formulas for Muon. Controlled experiments reproduce the predicted sensitivity scaling and AR(1) momentum correction factor for noise variance, confirming that noise geometry in Muon is governed by spectral sensitivity and not uniformly by flatness. For isotropic noise and square full-rank gradients, experimental verification matches the theoretical formula $S(p)=r(r-1)/(4\sigma_0^2)$ exactly.

Transformer Pretraining (NanoGPT)

In NanoGPT pretraining on OpenWebText, Muon consistently achieves:

• Higher per-layer spectral entropy ($\Delta H \approx 0.38$ nats vs. AdamW)
• Stable effective rank and greater nuclear norm, indicating retention of spectral diversity

However, in a matched small-ViT vision regime, Muon does not outperform AdamW, underscoring regime dependence. Muon's flat-spectrum bias is beneficial in "bulk" spectral regimes (language), less so in well-conditioned/low-rank regimes (vision).

Practical and Theoretical Implications

• Optimizer Regime Dependence: Muon is not universally superior; its flat-spectrum bias is advantageous when multiple spectral directions must remain active—primarily in high-rank transformer pretraining or tasks where diversity in singular values is desirable.
• Inductive Bias and Generalization: The study clarifies that Muon's implicit bias is toward spectral flattening but not low rank. This geometric bias fundamentally differs from classical implicit biases under SGD or nuclear-norm minimization, impacting solution selection in underdetermined regression and deep network training.
• Noise Geometry and Training Scaling: The analytical connection between spectral sensitivity and critical batch size informs tuning protocols for Muon in large-scale training, with direct implications for optimizer benchmarking and learning rate scaling.
• Weight Decay and Spectral Rank: Decoupled weight decay preserves spectral dynamics and active threshold spectral rank, reinforcing current best practices in Muon usage.

Future Directions

The paper identifies several open problems:

• Full characterization of measurement geometries guaranteeing persistent spectral flattening under Muon.
• Global Lyapunov-style theory bridging discrete Muon iterates to projected polar flows.
• Extension of critical batch size analysis to rectangular/rank-deficient regimes, where polar sensitivity diverges.
• Empirical validation of transformer reduction hypotheses, particularly activation rank measurements and layerwise regime interventions.

Conclusion

This work establishes Muon as a flat-spectrum optimizer governed by explicit spectral dynamics and regime-dependent implicit bias, theoretically and empirically distinguishing it from gradient normalization and nuclear-norm regularization. The optimization bias and noise geometry insights refine the understanding of Muon’s inductive bias, scaling laws, and practical utility in both matrix regression and deep neural network training, particularly for LLMs. The theoretical results, empirical falsifications, and regime-dependent analyses collectively advance the spectral optimization literature and inform future developments in spectral optimizer design and benchmarking.