Convergence of Spectral Descent for Non-smooth Optimization

Abstract: The Muon optimizer has recently demonstrated remarkable empirical success in training LLMs. However, the theoretical understanding of its mechanisms remains limited. Current convergence guarantees for Muon rely heavily on smoothness assumptions, leaving its non-smooth convergence behavior largely unexplored. In this work, we take a step toward bridging this gap by investigating Spectral Descent (SD), a simplified variant of Muon, together with its truncated counterpart, Truncated Spectral Descent (TSD). Under convexity, Lipschitz continuity, and sharpness conditions, we establish global linear convergence for both SD and TSD in non-smooth convex formulations. We also study regularized variants equipped with decoupled weight decay and derive sublinear convergence guarantees through their connection with Frank-Wolfe methods. Finally, we apply our theoretical framework to robust low-rank matrix recovery under mixed sparse and dense noise regimes and provide rigorous recovery guarantees. Numerical experiments support the theoretical findings and demonstrate the effectiveness of Muon-type methods for non-smooth optimization.

• The paper provides global linear convergence guarantees for both SD and its truncated variant under non-smooth convex assumptions.
• It introduces rank-dependent contraction mechanisms and truncation relaxations that reduce computational load while preserving convergence.
• Empirical validations on ReLU networks and low-rank recovery demonstrate robustness and accelerated descent compared to traditional optimizers.

Convergence Analysis of Spectral Descent for Non-smooth Optimization

Motivation and Context

Matrix-wise optimization has become foundational in large-scale deep learning, especially in training LLMs. Recent advances such as Muon and Spectral Descent (SD) leverage spectral geometry for descent directions, offering computational and generalization benefits compared to classical optimizers like SGD and Adam. However, rigorous convergence guarantees for Muon-type methods outside of smooth settings are scarce. This paper addresses that gap by providing global convergence analysis for SD and its truncated variant (TSD) under non-smooth convex regimes, and further, for regularized variants equipped with decoupled weight decay. The theoretical framework is validated on robust low-rank matrix recovery, and numerical experiments corroborate the analytic predictions.

Theoretical Contributions

The core theoretical advances are:

• Linear Convergence for SD and TSD: The authors rigorously prove global linear convergence rates for SD and TSD assuming convexity, Lipschitz continuity, and sharpness. For SD, the contraction rate is explicitly governed by the subgradient rank, yielding a rank-dependent descent mechanism. The step-size schedule follows geometric decay, and the condition parameter $\kappa = \mu/L$ must satisfy $\kappa > \sqrt{1 - 1/\bar{r}}$, with $\bar{r}$ an upper bound on trajectory ranks.
• Truncation Relaxations in TSD: TSD, which employs rank-$s$ truncation of the descent direction, reduces computational overhead and relaxes the worst-case condition parameter requirement. The convergence bound is controlled by truncation-dependent constants, and for practical purposes, small $s$ (e.g., $s=1$) suffices, avoiding expensive full SVDs.
• Regularization and Decoupled Weight Decay: The paper analyzes regularized SD and TSD updates that incorporate decoupled weight decay, connecting the dynamics to spectral-norm-constrained Frank-Wolfe methods. A neighborhood-based subgradient selection (spatial smoothing) provides theoretical guarantees of $\mathcal{O}(1/\sqrt{T})$ sublinear convergence, circumventing the condition parameter requirement.
• Application in Robust Low-Rank Matrix Recovery: Using a Least Absolute Deviation (LAD) formulation, the framework yields recovery guarantees under mixed sparse and dense noise. The connection to Ky Fan norms enables computational tractability by selecting minimal truncation ranks.

Empirical Validation

Empirical tests demonstrate that SD, TSD, and Muon-type optimizers achieve robust and accelerated convergence across a variety of non-smooth settings:

• ReLU Network Training: SD and Muon consistently outperform GD and Adam in convergence speed and robustness to learning rate schedules for two-layer ReLU networks on MNIST and CIFAR-10.

Figure 1: Empirical comparison on MNIST and CIFAR-10 showing SD and Muon achieving faster loss decay than GD and Adam under both constant and decaying step sizes.

• Low-rank Matrix Recovery: RTSD-WD exhibits rapid initial descent and robustness to noise regimes (sparse, dense, mixed), converging to the optimal solution neighborhood for the noiseless and sparse cases, while remaining stable under mixed noise.

  Figure 2: RTSD-WD convergence for low-rank matrix recovery, illustrating robustness and rapid initial descent under different noise settings.

• Sensitivity to Truncation Parameter: Empirically, all tested truncation parameters ($s=1,2,3$) are robust, with $s=1$ both computationally efficient and yielding strong results.

  Figure 3: RTSD-WD convergence across different truncation parameters demonstrating robustness and computational efficiency for small $s$.

• Linear Programming and Matrix Classification: SD and TSD reliably minimize non-smooth polyhedral convex objectives and recover optimal solutions, with observed alignment metrics indicating that empirical behavior far exceeds worst-case theoretical bounds.

Figure 4: Optimization dynamics for SD/TSD on linear programming, tracking loss, gradient norm, direction alignment, and parameter error.

• Hinge-loss Matrix Classification: SD achieves rapid reduction in loss and parameter error, even in linearly inseparable noise-perturbed settings.

Figure 5: SD convergence for matrix classification, with rapid descent in loss and parameter error in noisy, non-separable conditions.

Practical and Theoretical Implications

The results extend convergence theory for matrix-aware optimizers to non-smooth convex landscapes. Practically, Muon-type methods can be confidently applied to large-scale, non-smooth settings such as ReLU networks, robust matrix recovery, and classification with polyhedral losses. The use of truncated spectral directions significantly improves scalability.

Theoretically, the paper formalizes the role of rank in contraction rates and demonstrates that spatial regularization via weight decay guarantees convergence independently of the condition parameter. Empirical evidence suggests that worst-case bounds can be substantially tightened for real-world problems, motivating future work toward problem-specific and probabilistic alignment analysis. The connection of spectral descent to Frank-Wolfe and spatial smoothing opens new avenues for structured optimization algorithms in non-smooth machine learning.

Future work will address extending these results to highly non-convex landscapes, such as deep neural network architectures (including transformers and ReLU MLPs), where sharpness and convexity assumptions do not strictly hold.

Conclusion

This paper establishes rigorous convergence guarantees for Spectral Descent, Truncated Spectral Descent, and regularized Muon-type optimizers in non-smooth convex settings. The analysis holds under realistic Lipschitz and sharpness assumptions, and demonstrates both theoretically and empirically that these methods achieve either global linear or sublinear convergence, depending on regularization regime. The framework provides practical algorithms for robust matrix recovery and deep learning, scaling efficiently via truncation and spatial smoothing. Empirical validation confirms robustness and rapid descent, even when theoretical condition parameter requirements are violated. Tightening deterministic bounds through problem-specific analysis and extending to non-convex landscapes constitute promising directions for future research in spectral convex optimization and its applications in modern AI.