When do spectral gradient updates help in deep learning? (2512.04299v1)

Abstract: Spectral gradient methods, such as the recently popularized Muon optimizer, are a promising alternative to standard Euclidean gradient descent for training deep neural networks and transformers, but it is still unclear in which regimes they are expected to perform better. We propose a simple layerwise condition that predicts when a spectral update yields a larger decrease in the loss than a Euclidean gradient step. This condition compares, for each parameter block, the squared nuclear-to-Frobenius ratio of the gradient to the stable rank of the incoming activations. To understand when this condition may be satisfied, we first prove that post-activation matrices have low stable rank at Gaussian initialization in random feature regression, feedforward networks, and transformer blocks. In spiked random feature models we then show that, after a short burn-in, the Euclidean gradient's nuclear-to-Frobenius ratio grows with the data dimension while the stable rank of the activations remains bounded, so the predicted advantage of spectral updates scales with dimension. We validate these predictions in synthetic regression experiments and in NanoGPT-scale LLM training, where we find that intermediate activations have low-stable-rank throughout training and the corresponding gradients maintain large nuclear-to-Frobenius ratios. Together, these results identify conditions for spectral gradient methods, such as Muon, to be effective in training deep networks and transformers.

• The paper introduces a criterion based on the gradient's nuclear rank versus the activation's stable rank to predict when spectral updates yield greater loss reduction.
• Empirical results on transformers and MLPs demonstrate that low stable rank activations combined with high gradient nuclear rank lead to a dimension-dependent spectral advantage.
• The study also identifies cases where architectural choices, such as gated activations, diminish the benefits of spectral updates compared to traditional Euclidean methods.

When Do Spectral Gradient Updates Help in Deep Learning? — (2512.04299)

Introduction and Motivation

The efficacy of first-order methods, particularly SGD and its adaptive variants (Adam, RMSprop), in deep learning is well established. Recent developments in spectral gradient methods, notably the Muon optimizer, have demonstrated strong empirical performance, especially in transformer and LLM training. Despite promising results, the underlying regime where spectral updates surpass Euclidean (Frobenius-based) updates was not systematically characterized. This paper provides a precise analysis and predictive criterion for when spectral steps yield larger loss reduction than their Euclidean counterparts, introducing a unified theoretical and empirical framework for understanding these effects.

Key Theoretical Criterion: Nuclear Rank vs. Stable Rank

A central contribution is the derivation of a rigorous, layerwise condition that calibrates the efficacy of spectral updates:

$$\frac{\|G\|_*^2}{\|G\|_F^2} \ge \frac{\|A\|_F^2}{\|A\|_{\mathrm{op}}^2}$$

where $G$ is the gradient block for a weight matrix $W$ acting on activations $A$, $\|\cdot\|_*$ denotes the nuclear norm, $\|\cdot\|_F$ the Frobenius norm, and $\|\cdot\|_{\mathrm{op}}$ the spectral norm. The term on the right is the stable rank $\mathrm{st}(A)$ of the activations, while the left is the "nuclear rank" $nr(G)$. In multi-layer networks and transformers, the criterion becomes blockwise. Spectral updates are favored if, for each block, $nr(G_\ell) \ge \mathrm{st}(A_{\ell-1})$.

This result enables direct diagnostic evaluation in realistic networks. Empirically, typical neural network activations are highly degenerate—having low stable rank—while the corresponding gradients often have high nuclear rank, particularly after a short period of training.

Figure 1: Stable rank of MLP post activations while running a snapshot of modded-NanoGPT, showing the consistently low stable rank relative to the maximal matrix rank.

Empirical Analysis: Stable Rank in Deep Networks

The paper provides both theoretical proofs and empirical data demonstrating that post-activations in common architectures (MLP, Transformers) have stable rank bounded by a small constant, independent of width, depth, or sequence length, under mild initialization conditions. This low stable rank is observed:

• At initialization for Gaussian-model networks
• Throughout training, including in LLM settings

Concretely, in transformer blocks, the RMS-normalized representations entering attention and MLP projections, as well as the MLP post-activations themselves, maintain low stable rank over the entire training trajectory.

Figure 2: Stable rank of RMS-normalized activations $A^{\mathrm{rms}}$ across training, indicating persistent low stable rank in transformer blocks.

Figure 3: Embeddings matrix post-activation and token-indicator stable rank throughout training reveals modest but non-trivial stable rank in embeddings compared to internal layers.

The mechanism driving low rank is the presence of mean-induced "spikes" in the Gram matrix of the post-activations, most pronounced in non-centered nonlinearities such as ReLU.

Spectral Updates: Scaling Advantage with Dimension

Beyond the one-step descent analysis, the paper provides results for random feature models, showing that after modest "burn-in," the nuclear rank of the gradient can grow linearly with the data or feature dimension, while the activation stable rank remains $O(1)$. This yields a dimension-dependent advantage for spectral methods that persists across a significant window of training.

Figure 4: Evolution of the nuclear rank of the gradient in the realizable random-feature model, evidencing rapid growth and large dynamic range in early training.

Figure 5: Teacher–student random-feature model illustrating the scaling of gradient nuclear rank with dimensionality after the first step.

Numerical experiments, both synthetic (e.g., random feature regression) and on models such as NanoGPT, validate the predicted advantage and demonstrate strong empirical alignment with theoretical predictions.

Regimes in Which Spectral Updates Are Not Superior

The paper also critically addresses cases where spectral updates are less effective or even worse than Euclidean methods. For example, in random feature models with centered (gated) activations such as SwiGLU, mean-induced spikes in activation covariances are suppressed, resulting in substantially higher stable rank and loss of spectral update advantage. These findings are empirically validated, including ablations on modded-NanoGPT, indicating that Muon's benefit is block-dependent and can be mitigated by architectural or activation choices.

Theoretical Implications for Architecture and Optimization

This work unifies a range of empirical observations under explicit, calculation-ready criteria, offering several substantial implications:

1. Optimization Geometry: Spectral steps adapt to the underlying degeneracy in the propagated data, providing improved conditioning and steeper descent when activations are low-stable-rank.
2. Architecture Design: Non-centered activations and normalization designs strongly influence the prevalence of the low-stable-rank regime, suggesting that in “mean-spiked” settings, spectral methods should be standard on interior (hidden) blocks.
3. Practical Guidance: The blockwise nuclear rank versus stable rank condition can be monitored online during training, enabling per-block optimizer selection and hyperparameter adjustment.
4. Limitations: In non-degenerate cases with relatively high stable rank post-activations, Euclidean updates retain or exceed the spectral variant’s efficiency. Gated MLPs, as used in some recent LLMs, exemplify such scenarios.

Experimental Findings in Large-Scale LLMs

• MLP post-activation, RMS-normalized, and many attention-related matrices in transformer blocks are always low-stable-rank, and their gradients exhibit consistently high nuclear rank.
• In contrast, the input embedding and language-model head weights interact with matrices of somewhat higher (but still moderate) stable rank, making spectral benefit less pronounced.

Extensions and Related Work

The paper aligns its criterion with and sharpens recent convergence analyses (e.g., [shen2025convergence], [pethick2025scion]) where the gradient nuclear rank emerges as the critical quantity. It also complements structural studies of low-rank representations and gradients in deep networks ([gurari2018tiny_subspace], [huh2021low_rank_simplicity], [zhao2024galore]), providing explicit operational meaning to observed spectral phenomena in modern deep learning.

Conclusion

This analysis introduces a precise, theoretically-justified regime for the expected superiority of spectral gradient methods over standard Euclidean updates in deep network and transformer optimization. The key is the prevalent, persistent low-stable-rank structure of internal activations, which, when combined with large gradient nuclear ranks, predicts strong spectral descent benefits—especially in wide, deep settings typical of current LLMs. The paper’s diagnostic criterion supports practical optimizer selection and deepens the understanding of training geometry in neural networks, while also indicating architectural and normalization factors that determine spectral geometry’s effectiveness.

Spectral updates are most effective where both low activation stable rank and high gradient nuclear rank coincide. Activation and training regimes that disrupt this degeneracy—such as gating and strong normalization—reduce or eliminate the spectral advantage. These insights provide a new lever for both practical optimizer scheduling and theoretical analysis of optimization dynamics in deep learning.

---

References:

• “When do spectral gradient updates help in deep learning?” (2512.04299)
• Related works: [shen2025convergence], [pethick2025scion], [huh2021low_rank_simplicity], [zhao2024galore], [gurari2018tiny_subspace]