When to use what Schatten-$p$ norm in deep learning?

Abstract: Schatten-$\infty$ based optimizers such as Muon have shown promising empirical performance, but there remains seemingly conflicting observations regarding whether they are beneficial. We resolve this conflict by showing that the conclusion is regime dependent. Even when the objective is smooth in the Schatten-$\infty$ geometry, smaller Schatten-$p$ geometries can be optimal, specifically in the low-dimensional regime, which we show includes Chinchilla scaling. This conclusion follows from a new noise-robust acceleration result for the SODA framework for $p>2$. The same analysis explains why Muon-like methods do not require warmup, why they naturally favor large batches, and yields a batch size scaling rule for arbitrary $p$.

• The paper provides a regime-dependent theoretical framework that reconciles conflicting behaviors in Schatten-p norm based optimizers by unifying Muon, HTMuon, and finite-p methods.
• It introduces explicit update recursions for vector and matrix parameters under p-uniform convexity, deriving complexity bounds and batch size scaling laws.
• The analysis offers practical guidance for optimizer tuning by prescribing optimal Schatten-p norms based on model dimension, compute budget, and gradient noise characteristics.

Schatten-$p$ Norm Selection in Deep Learning Optimization

Introduction and Motivation

The optimization geometry induced by Schatten-$p$ norms has become a focal point in adaptive methods for deep learning, largely driven by the empirical success of optimizers like Muon, which employs the Schatten-$\infty$ (spectral) norm. Despite successes, practitioners have observed conflicting phenomena regarding when spectral geometry actually elicits improvements. This paper provides a rigorous and regime-dependent theoretical reconciliation, supplying explicit complexity bounds and design rules for choosing the value of $p$ in Schatten-$p$ norm-based optimizers, covering both vector and matrix parameterizations.

The work leverages the SODA (Schedule-free Optimistic Dual Averaging) framework, extending the theoretical analysis to $p > 2$ via uniform convexity, which was not encompassed by previous analyses. This more general approach explains several empirical findings, including the conditions in which Muon is optimal, when warmup is necessary, the batch size scaling required for acceleration, and critically, why finite-$p$ methods (e.g., HTMuon, Soft-Muon) often outperform the pure spectral approach in settings typical of current large-scale training.

Theoretical Contributions

The analysis hinges upon the geometry-noise duality in stochastic optimization: the optimal norm depends on both the geometry in which the loss is smooth and the tail behavior of gradient noise. The SODA template is parameterized by a $p$-uniformly convex regularizer, yielding explicit update recursions for vector and matrix variables:

• Vector variables ($\ell_p$): The dual update combines the sign and $(q-1)$-power (where $p$0) operations componentwise.
• Matrix variables (Schatten-$p$1): The dual update involves an SVD and applies the power operation to singular values, precisely matching the logic of HTMuon and Soft-Muon when $p$2 is finite.

This explicit parameterization unifies Muon \cite{jordan2024muon}, HTMuon, and other recent variants, enabling coverage of $p$3 regimes and connecting optimism, momentum, and weight decay in modern optimizers within the same framework.

The core complexity result under $p$4-uniform smoothness and possibly heavy-tailed statistical noise is: $p$5 for the accelerated setting, where $p$6 is the number of iterations, $p$7 the $p$8-geometry smoothness constant, $p$9 the optimization radius, and $\infty$0 the noise scale in the dual norm. This formulation highlights how the geometry ($\infty$1) directly modulates the attainable rates, and that in the deterministic (noise-free) limit, lower $\infty$2 always yields stronger acceleration.

Figure 1: Non-accelerated stepsize scaling as a function of $\infty$3 illustrates how the effective learning rate adapts across geometries and regimes.

Figure 2: The accelerated batch size scaling rule under bounded variance demonstrates the growth of the optimal batch size with $\infty$4 and the approach to the spectral norm.

A further contribution is the noise-robust extension: the analysis tracks the heavy-tailed moment assumptions, deriving batch size scaling rules. Under bounded variance ($\infty$5), the batch size $\infty$6 (where $\infty$7 is the total oracle budget), showing that as $\infty$8 (i.e., the spectral norm), ever-larger batches are required, matching empirical batch size tuning heuristics observed for Muon-like schemes.

Regime Dependence and Geometry Selection

A key advancement is the explicit identification of when a Schatten-$\infty$9 geometry is optimal. If the objective is smooth in the spectral norm (i.e., $p$0 is small), intuition and prior works suggest aligning the optimizer to match, i.e., choose $p$1. This analysis rigorously demonstrates that this is true only in the high-dimensional or compute-constrained regime ($p$2, with $p$3 the effective parameter dimension), where the number of parameter updates is small relative to the model.

However, as soon as training steps become abundant relative to model size (the "low-dimensional" or "overtraining" regime, $p$4), smaller $p$5 (down to $p$6 or $p$7 determined by the noise moment) can be strictly optimal even when the loss is spectrally smooth. This is due to the $p$8 and $p$9 scaling in the convergence rates. The practical implication is that for modern, well-scaled training regimes satisfying Chinchilla-style recipe ($p$0), the low-dimensional regime prevails and finite-$p$1 geometries (e.g., $p$2) provide superior theoretical guarantees and empirical results.

This regime distinction directly explains the recent empirical transition: Muon and its spectral-norm cousins display strength in short runs and speedrun settings, but are outperformed by SGD, Adam, or Lion-like optimizers—and by finite-$p$3 variants of Muon—in large, full-batch or overparameterized settings.

Batch Size Scaling and Noise Robustness

The analysis provides an explicit batch size scaling law for arbitrary $p$4 (in both classical and accelerated schedules):

$p$5

Thus, maximal effective learning in Schatten-$p$6 optimization (especially spectral norm) necessitates large batch sizes, precisely mirroring empirical findings that Muon needs large $p$7 to outperform other methods. Moreover, under acceleration and bounded variance, the batch size exponent is even higher, emphasizing the need for substantial compute allocation in these regimes.

The framework also ties the preferred geometry $p$8 to the noise moment ($p$9) in mini-batched gradients, leading to the selection of the minimal admissible geometry $p > 2$0 as the budget $p > 2$1 grows relative to dimension.

Geometry Scheduling and Practical Prescriptions

The regime analysis admits an adaptive prescription:

• High dimension/low budget ($p > 2$2): Use geometry matching the smoothness; e.g., spectral norm ($p > 2$3) if smooth in $p > 2$4.
• Low dimension/high budget ($p > 2$5): Use geometry matching the tail of the stochastic noise; e.g., Euclidean or lighter norms. If the noise is heavy-tailed, select $p > 2$6 per moment bound.
• Unknown budget/anytime setting: A switching schedule for $p > 2$7 is proposed, shifting from high $p > 2$8 to minimal admissible $p > 2$9 once $p$0 crosses the critical boundary; this is consistent with empirical schedules applied in recent finite-$p$1 Muon variants.

Implications for Scaling Laws and Future Directions

Given that Chinchilla scaling regimes ($p$2) dominate large-scale practice, this theory explains the observed reduction in Muon's advantage and the rise of finite-$p$3 norm methods as top-performing optimizers in large model runs. It also supplies a first-principles justification for batch size annealing and optimizer schedule tuning beyond grid search, based on geometric and stochastic characteristics of the loss and the minibatch oracle.

Future work should investigate dynamically adaptive geometry schedules and further refine geometry selection based on empirical noise distributions in deep models, advancing toward optimizers that adapt both learning schedule and geometry automatically.

Conclusion

This paper provides the first regime- and geometry-aware analysis of Schatten-$p$4 norm selection for deep learning optimization, unifying and explaining much of the divergent empirical evidence around spectral norm methods. The theoretical results systematically connect optimizer geometry, batch size scaling, and statistical noise, offering actionable guidance for optimizer and training recipe design. These results heavily influence how practitioners should choose and tune optimization geometry, particularly in large-scale training and in domains increasingly demanding both high performance and robust, automatic adaptivity.