Muon is Not That Special: Random or Inverted Spectra Work Just as Well

Abstract: The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonstrating that precise geometric structure is not the key factor affecting optimization performance. First, we introduce Freon, a family of optimizers based on Schatten (quasi-)norms, powered by a novel, provably optimal QDWH-based iterative approximation. Freon naturally interpolates between SGD and Muon, while smoothly extrapolating into the quasi-norm regime. Empirically, the best-performing Schatten parameters for GPT-2 lie strictly within the quasi-norm regime, and thus cannot be represented by any unitarily invariant LMO. Second, noting that Freon performs well across a wide range of exponents, we introduce Kaon, an absurd optimizer that replaces singular values with random noise. Despite lacking any coherent geometric structure, Kaon matches Muon's performance and retains classical convergence guarantees, proving that strict adherence to a precise geometry is practically irrelevant. Third, having shown that geometry is not the primary driver of performance, we demonstrate it is instead controlled by two local quantities: alignment and descent potential. Ultimately, each optimizer must tune its step size around these two quantities. While their dynamics are difficult to predict a-priori, evaluating them within a stochastic random feature model yields a precise insight: Muon succeeds not by tracking an ideal global geometry, but by guaranteeing step-size optimality.

• The paper demonstrates that local optimization quantities, rather than rigid geometric structures, are the primary drivers of convergence and performance in spectral optimizers.
• The paper reveals that randomized methods like the Kaon optimizer and quasi-norm updates in Freon achieve validation losses comparable to Muon despite abandoning strict norm constraints.
• The paper introduces efficient rational iteration techniques for computing fractional updates, emphasizing step-size stability and local gradient alignment in deep learning.

Challenging the Geometric Narrative of Spectral Optimizers

Introduction: Geometric Preconditioning and Its Limitations

Spectral optimizers such as Muon and Shampoo have been widely adopted at scale for training large neural networks, with theoretical justification traditionally rooted in non-Euclidean geometry and Linear Minimization Oracle (LMO) frameworks. Muon's celebrated performance is typically attributed to its complete spectrum whitening, ensuring precise adherence to a target geometry, resulting in descent directions optimal under specific matrix norms. However, recent empirical and theoretical analyses show that these justifications are largely insufficient. The paper systematically deconstructs geometric arguments and demonstrates that convergence and performance are governed not by the underlying matrix geometry, but by local optimization quantities independent of strict geometric structure.

A quintessential example illustrating the irrelevance of geometric structure is the Kaon optimizer, which replaces singular values with random noise yet matches Muon's empirical performance and retains classical convergence guarantees. Similarly, the Freon family of Schatten (quasi-)norm-based updates reveals optimal hyperparameter regimes that lie strictly outside the normed region, contravening LMO theory. The main claim is that the prevailing geometric explanations for spectral optimizers are incomplete, and that fundamental optimization dynamics are instead determined by local batch gradient alignment and directional descent potential.

Figure 1: Transformations of the gradient singular value spectrum under various spectral optimizers for a GPT-2 layer.

Spectral Optimization Beyond Geometry: Freon and Kaon

Freon generalizes spectral updates by interpolating between SGD ($c=0$) and Muon ($c=1/2$) via $(GG^\top)^{-c}G$, extending into the quasi-norm regime ($c > 1/2$), which fundamentally breaks the norm-based LMO framework. The introduction of an efficient QDWH-based iterative approximation allows for stable computation of these updates, leveraging theory from rational minimax approximation and polar decomposition techniques [amsel2025polarexpressoptimalmatrix, zolopd]. Empirically, in a GPT-2 training setup, the optimal Schatten exponents consistently fall in the quasi-norm region, and the best-performing updates are not representable by any unitarily invariant norm.

Complementing this, the Kaon optimizer exploits chaotic polynomial spectral reshaping, replacing the gradient's singular values with noise via the third-order logistic recurrence. This statistically redistributes the spectrum, yet is demonstrably equivalent in convergence and performance to Muon. The classical assumptions underlying LMO and geometric descent are thus unnecessary for practical optimizer design. The suppression of large singular values is necessary for stabilization, but precise target spectra are largely irrelevant.

Figure 2: NanoGPT validation-loss curves for Muon, Kaon, and Freon at different Schatten exponents; all variants achieve comparable performance.

Empirical Findings: Learning Rate Sensitivity and Loss Dynamics

Learning rate sweeps across NanoGPT and WikiText-2 reveal that, once large singular values are suppressed, all spectral optimizers (including Muon, Freon, Kaon, TruncatedSGD) converge to similar validation loss, provided step sizes are tuned appropriately. TruncatedSGD improves over SGD but cannot match Muon, indicating that clipping noise is insufficient—gradient direction alignment is not always meaningful. Freon with $c > 1/2$ matches Muon and Kaon on validation loss, even though it operates outside any norm-defined geometry, underscoring the flexibility of spectral updates.

Figure 3: Final validation loss versus learning rate for all optimizer families on WikiText-2; Kaon matches Muon despite its stochastic spectral redistribution.

Theoretical Framework: Local Quantities Govern Optimization

The paper proposes local optimization quantities as more fundamental drivers. Specifically, for an update direction $D_k$ and batch gradient $G_k$, loss decrease is governed by:

• Batch gradient alignment: $$\gamma_k = \frac{\langle G_k, D_k \rangle}{\langle \widetilde{G}_k, D_k \rangle}$$
• Directional descent potential: $$\Phi_k = \frac{\langle \widetilde{G}_k, D_k \rangle^2}{\langle D_k, \nabla^2 f(Z_k)[D_k] \rangle}$$

Empirical exploration shows that optimizers deliberately trade gradient alignment ($\gamma$) for descent potential ($c=1/2$0), and that the largest improvements in loss are realized by step-size tuning, not by adherence to global geometry. Importantly, optimizers that operate in the quasi-norm regime (Freon, Kaon) can vastly outperform norm-constrained spectral methods by amplifying $c=1/2$1 even at the cost of reducing $c=1/2$2.

Figure 4: Per-layer curvature alignment and batch-gradient alignment in GPT-2 training at checkpoints for various spectral update directions.

Efficient Rational Iterations and Numerical Stability

Computation of $c=1/2$3 for fractional exponents requires stable and efficient methods, as naive polynomial iterations are numerically unstable for $c=1/2$4, especially in low precision. The paper establishes that rational Remez-type iterations, generalizing QDWH polar decomposition, uniquely achieve forward stability across all tested exponents and matrices, outperforming classical polynomial schemes [zolopd, Trefethen2025]. Analytical and empirical results provide doubly exponential convergence bounds and closed-form schedules for hyperparameters, allowing robust implementation of Freon across quasi-norm regimes.

Figure 5: Convergence of polynomial (left) and rational (right) Remez iterations for $c=1/2$5; only rational iterations converge universally.

Random Feature Model Analysis and Asymptotics

In random-feature regression models, detailed asymptotic analysis reveals that spectral descent has a mechanistic advantage due to step-size stability: the optimal step size for spectral updates remains constant throughout training, whereas optimal SGD step sizes are highly oscillatory and impractical. The optimal Schatten exponents vary dynamically by layer and batch, and full-batch quadratic models poorly approximate global non-quadratic landscapes, especially in transformer architectures. The local curvature landscape is distorted in real models, favoring exponents beyond the norm regime.

Figure 6: Training loss and step-size dynamics in random-feature regression with ReLU activation; spectral descent exhibits step-size constancy.

Practical and Theoretical Implications

Theoretical implications include a demonstration that norm-constrained LMOs are not necessary for optimizer convergence or optimality, and that batch gradient alignment and descent potential are mechanistic quantities underlying training progress. Practically, this suggests that practitioners may design spectral optimizers by focusing on local signal-to-noise properties and step-size stability, rather than geometric fidelity. Random or chaotic spectral reshaping (as in Kaon) is equally effective, reducing reliance on complex geometric analysis. The findings challenge future developments to focus on locally tuned, potentially stochastic update schemes that harness the primary descent potential $c=1/2$6, and motivate abandoning strict geometric arguments in favor of empirically driven diagnostics.

Figure 7: Validation loss curves for various spectral optimizers at their best learning rates; Freon variants operate outside norm regime yet match Muon.

Conclusion

The paper systematically refutes geometric explanations for Muon's and other spectral optimizers' success, showing that randomized and quasi-norm spectral updates are equally effective. It provides a framework where local alignment and descent potential, mediated by step-size adaptation, are the true determinants of optimization. Theoretical and empirical evidence broadens the design space for optimizers and motivates further research into stochastic, locally adaptive spectral schemes devoid of strict norm constraints. Geometry, while elegant, is practically irrelevant for optimizer efficacy in deep learning (2605.11181).