Spectral Scaling Laws of Muon

Abstract: Orthonormalized update rules have rapidly become a leading choice of optimizer for training LLMs, with recent open-source state-of-the-art models adopting Muon. To keep these updates tractable, Muon performs the orthonormalization with the Newton--Schulz (NS) iteration. Since NS is only approximate, directions with small singular values fail to be orthonormalized. In Muon, NS is applied to the momentum matrix at every step, yet little is known about how the singular value spectrum of these momentum matrices behaves during training, or how that behavior changes with model size. We present the first systematic study of this question. Tracking singular value quantiles of the momentum buffer across layers in models ranging from 77M to 2.8B parameters, we observe a consistent picture: after a short burn-in, the quantiles stabilize at a value determined by the layer type and model size. These stabilization values follow remarkably clean power laws in model size, with layer-dependent exponents. Layers up to mid-late depth scale very mildly with model size $M$ (around $M^{-0.25}$), so the standard 5-step NS configuration used at academic scale will continue to orthonormalize them at much larger scales. Some of the late layers, however, scale much more aggressively (up to $M^{-0.96}$) and will fall into the NS failure regime at frontier scale unless one uses more NS iterations or better-tuned coefficients. NS iterations are computationally expensive at scale; our laws give practitioners a principled, layer-aware recipe for choosing the minimum NS configuration that still orthonormalizes the directions that matter -- avoiding unnecessary computation without sacrificing update quality.

• The paper demonstrates that momentum singular value quantiles stabilize and adhere to layer-dependent power laws across model scales.
• It employs rank-p ablation studies to evaluate NS accuracy, showing that orthonormalizing the top 50% of directions largely preserves performance.
• The work provides a principled, layer-aware recipe for NS configuration, optimizing computational efficiency in large-scale training.

Spectral Scaling Laws of Muon: Systematic Characterization and Practical Implications

Introduction

The paper "Spectral Scaling Laws of Muon" (2606.04058) presents a rigorous empirical investigation into the spectral dynamics of the Muon optimizer’s momentum matrices during large-scale LLM training. Muon, an orthonormalized-update optimizer, utilizes a Newton-Schulz (NS) iteration to orthonormalize update directions. The work systematically tracks how singular value quantiles of these matrices evolve during pre-training across model scales ranging from 77M to 2.8B parameters. The authors demonstrate that, post-burn-in, these quantiles stabilize at layer- and size-dependent values, which obey remarkably clean power laws in model size with layer-specific exponents. This spectral behavior is exploited to recommend layer-aware NS configurations for efficient large-scale training, addressing a previously opaque optimization hyperparameter choice.

Muon and Newton-Schulz Approximation Mechanisms

Muon replaces standard gradient updates in weight matrices with approximate orthonormalized updates, realized via NS iteration. For each step, Muon maintains a momentum buffer $M_t$, to which NS is applied, yielding updates of the form $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$, where $O_{t+1}$ is an orthonormalized update direction.

The NS iteration acts as a sequence of odd polynomials applied to the normalized momentum matrix, pushing each singular value toward unity. However, directions corresponding to sufficiently small singular values are not properly orthonormalized due to NS’s inherent limitation: the polynomial mapping $f(\sigma)$ fails to approximate the sign function near zero.

Figure 1: NS map $f(\sigma) = p^{\circ 5}$ shows strong approximation for moderate $\sigma$ but fails for small singular values.

Optimally tuning NS coefficient sequences is critical, with typical configurations using five polynomials (as in NanoGPT) and more accurate compositions (up to ten steps) preferred at frontier scale (e.g., DeepSeek-V4). Each iteration incurs a substantial computational overhead, motivating analysis of which layers and quantiles require precision.

Empirical Characterization: Rank-p Updates and Quantile Dynamics

To clarify which fraction of directions must be accurately orthonormalized, the paper performs controlled ablations using rank-$p$ orthonormal updates. Only the top $p$ fraction of singular vectors are preserved in the update; performance is tracked across multiple model scales.

Figure 2: Pre-training loss curves for 77M, 160M, and 354M models under rank-$p$ updates, quantifying performance degradation as $p$ decreases.

Results reveal that orthonormalizing the top $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$0 of directions essentially retains Muon’s efficacy, whereas restricting to just $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$1 results in substantial loss inefficiency (approx. $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$2 increase in iterations required). These results are robust to initial conditions and not an artifact of early training dynamics.

Figure 3: Persistent performance gap for $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$3 updates even after warm-starting from full Muon.

This empirical threshold reveals that the practical regime of NS accuracy must cover at least the median quantile of the momentum singular value spectrum for Muon’s benefits to be realized.

Quantile Stabilization and Spectral Distribution Evolution

Quantile tracking ($\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$4) of normalized singular values across transformer block layers shows a consistent phenomenon: after burn-in, quantiles stabilize at fixed values specific to layer type and model scale. These stabilization values decrease monotonically with model size, and are highly dependent on layer depth, especially in the final blocks.

Figure 4: Stabilization of the 50\% normalized singular value quantile across three representative layer types and model sizes.

The stabilized spectra are dominated by a single large singular value, with the bulk of directions clustered near zero, especially in late layers.

Figure 5: Full and bulk singular value spectra for 2.8B model momentum matrices at step 1500, illustrating exponential decay and single-value dominance.

Quantile evolution under rank-$\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$5 updates mirrors validation loss: $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$6 closely track Muon’s spectrum, while lower $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$7 diverges, confirming the correspondence between spectral dynamics and update quality.

Figure 6: Quantile trajectories for the 50\% quantile under rank-$\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$8 updates, supporting empirical threshold for NS accuracy.

Spectral Scaling Laws and Layer-Dependent Exponents

By averaging stabilizing quantile values post-burn-in and fitting log-log models against size, the authors establish clean power laws of the form $\Theta_{t+1} \leftarrow \Theta_t - \eta O_{t+1}$9, where $O_{t+1}$0 is layer-dependent.

Figure 7: Scaling laws for stabilization values of momentum singular value quantiles across depth and model sizes, highlighting layer variation in scaling exponents.

Exponents range from $O_{t+1}$1 in mid-early/mid-late layers (mild scaling) to $O_{t+1}$2 for the final MLP blocks (aggressive scaling). This variance means a uniform NS configuration is inefficient: final layers rapidly fall below NS’s effective regime as $O_{t+1}$3 increases, while earlier layers remain well-served with standard NS settings.

Figure 8: Comprehensive scaling laws across quantiles and layer types, reinforcing the necessity of layer-specific NS tuning.

Practical Implications and Frontier-Scale Extrapolation

The scaling laws enable direct extrapolation to frontier-scale training. For example, in a hypothetical 300B-parameter run, mid-late $O_{t+1}$4 projection stabilization remains outside the NanoGPT NS failure regime, while the final $O_{t+1}$5 projection drops below, necessitating the adoption of more accurate NS or custom-tuned coefficient sequences for the latter. This approach yields a principled, layer-aware recipe for optimizer configuration, enabling practitioners to minimize computation without sacrificing update fidelity.

Conclusion

This manuscript provides the first comprehensive analysis of Muon’s momentum spectra scaling behavior. It establishes the stabilization of singular value quantiles, fits layer-dependent power laws, and thereby turns NS configuration from a heuristic into a principled, quantitative hyperparameter. The results demonstrate that mild scaling in early layers permits standard NS settings across several orders of magnitude in model size, but late layers require more precise orthonormalization for effective training at scale. Methodologically, spectral scaling laws could be extended to alternative architectures and optimizers relying on iterative matrix polynomial approximations. Future work includes designing layer-specific NS polynomials and generalizing scaling law analysis to diverse model/optimizer classes.

For comprehensive citations, please see the full reference list in (2606.04058).