MuCon: Clipped Muon Updates for LLM Training

Abstract: Muon-style optimizers take a matrix-valued momentum or preconditioned update $B = U \operatorname{diag}(σ_1,\ldots,σ_r) V^\top$ and replace it with its canonical partial polar factor $\operatorname{Pol}(B) = U V^\top$. This maps every nonzero singular value to one. MuCon is the clipped-Muon variant studied here: it applies singular-value clipping to the same Muon matrix, $D^{\mathrm{MuCon}}_τ(B) = \operatorname{MClip}_τ(B) = U \operatorname{diag}\bigl(\min{σ_i,τ}\bigr) V^\top, \qquad τ> 0$. Thus, $\operatorname{MClip}_τ$ denotes the mathematical clipping operator, while MuCon denotes the optimizer primitive that substitutes this clipped direction for Muon's polar direction. The Muon/MuCon scaling parameterization used in this work is called $\text{SpectralP}$: it is the hidden-matrix scaling recipe under which polar Muon or clipped MuCon directions are applied. The map $\operatorname{MClip}_τ$ is the Frobenius projection onto the spectral-norm ball ${X : |X|_2 \le τ}$: it leaves singular values at or below $τ$ unchanged and modifies only the violating singular directions. This paper asks when the MuCon clipping step can be approximated without a full dense SVD. We record two exact identities, a polar/absolute-value formula and a scalar-root formulation leading to a rational Newton filter for the clipped positive-semidefinite factor, and identify the numerical obstruction common to both: singular values near the threshold make sign decisions and rational solves ill-conditioned. Matrix-function methods are therefore useful only when paired with stable polar/square-root primitives or explicit regularization near the clipping boundary.

• The paper presents MuCon as a spectral control primitive that replaces full orthogonalization with singular-value clipping to maintain moderate update directions.
• It rigorously formulates the clipping operator as a projection onto spectral-norm balls, employing partial SVD, polar decomposition, and rational Newton iterations for numerical robustness.
• The study integrates MuCon with scaling recipes like SpectralP and CompleteP, demonstrating its potential to balance update stability with effective learning in deep transformer models.

MuCon: Clipped Muon Updates for LLM Training

Overview

The paper "MuCon: Clipped Muon Updates for LLM Training" (2605.26459) systematically analyzes a matrix optimizer primitive termed MuCon, which adapts Muon's matrix update by applying singular-value clipping. MuCon replaces the aggressive orthogonalization—where updates become isometries—with clipping, yielding projections onto spectral-norm balls. The study investigates both the mathematical properties and numerical aspects of MuCon, focusing on scalability, approximation techniques, and integration with robust scaling conventions such as SpectralP and CompleteP.

Motivation and Scaling Context

Large neural network training, and particularly LLMs, increasingly relies on sophisticated optimizer parameterizations and spectral methods to preserve stability, enable hyperparameter transfer, and avoid lazy learning regimes. CompleteP, as described in [dey2025completep], provides width-depth scaling rules, emphasizing joint scaling that keeps feature learning nontrivial as both width ($N$) and depth ($L$) grow. SpectralP, as referenced in [zheng2026spectral], motivates spectral control in matrix updates—specifically operator-norm normalization.

MuCon is introduced as not a new parameterization, but as a projection-based spectral control primitive layered atop SpectralP's scaling and RMS-matching calibration. Where Muon's polar step applies a full isometry transformation ($\Pol(B) = UV^\top$), MuCon applies clipped singular values ($\MClip_\tau(B) = U(\min\{\sigma_i,\tau\})V^\top$), preserving moderate directions while mitigating high-gain instability.

This distinction is crucial: full orthogonalization can excessively distort update geometry, whereas singular-value clipping retains moderate directions and only suppresses extreme ones. The spectral norm constraint is enforced precisely—$\MClip_\tau(B)$ is the Frobenius projection onto $\|X\|_2 \le \tau$ [golub2013matrix].

Mathematical Formulation and Identities

The paper formalizes the clipping operator, its projection interpretation, and the exact low-rank correction identity. Given $B = U \Sigma V^\top$, clipping is defined as:

$\MClip_\tau(B) = U(\min\{\sigma_i, \tau\}) V^\top$

If $k_>$ singular values exceed $\tau$, the clipped matrix is a rank-$L$0 correction:

$L$1

where $L$2 correspond to violating singular directions.

The projection onto the spectral-norm ball $L$3 is unique due to unitary invariance:

$L$4

This framework also yields a polar/absolute-value formulation for clipping:

$L$5

with $L$6 applied by functional calculus.

Algorithmic Regimes and Numerical Challenges

Low-Rank Correction

When only a small subset of singular values violate the threshold, partial SVD, Lanczos, or randomized subspace methods (e.g., [halko2011finding]) offer efficient baselines. The computational cost can remain sublinear in problem size, and corrections are focused exclusively on outlier directions.

Matrix-Function Approximations

For situations with many violating directions, the paper examines matrix-function iterations, including polar/absolute-value and rational Newton approaches:

• The polar/absolute-value method reduces clipping to classical matrix functions (polar decomposition and matrix absolute value [higham2008functions]). However, directions near the threshold $L$7 introduce numerical sensitivity—eigenvalues close to $L$8 make sign decisions and rational solves ill-conditioned. Iterative polar decomposition and matrix absolute value computations are susceptible to finite precision artifacts, requiring regularization or smoothing near the boundary.
• Rational Newton iteration targets the clipped PSD factor directly, using spectral functional calculus to converge to $L$9. While locally quadratic convergence is achieved away from the threshold, directions near $\Pol(B) = UV^\top$0 deteriorate conditioning and slow convergence. The method relies on stable square-root/polar routines and may require explicit regularization to address ill-conditioned solves.

Both approaches are exact in spectral filter terms but depend critically on polar or square-root primitive stability. Dense SVD cost remains prohibitive for large-scale repeated optimizer applications unless matrix structure or approximation schemes are exploited.

Scaling Recipes and Practical Integration

The paper precisely details SpectralP scaling recipes for hidden matrix groups, aligning Muon/MuCon optimizer steps with RMS-matching calibration. SpectralP prescribes group-learning-rate conventions and operator-norm bookkeeping, ensuring update scales remain order one under joint width-depth scaling. It distinguishes companion AdamW groups, embedding/unembedding hyperparameter multipliers, and explicit post-processing for spectral control.

Notably, MuCon's direction is not forced to be an isometry; it only enforces the spectral-norm bound, making the optimizer primitive robust under the scaling conventions proposed by SpectralP and CompleteP.

Implications and Future Directions

The theoretical implications are highly relevant for optimizer design in deep, large-scale neural architectures. Projection-based spectral control primitives—such as MuCon—strike a balance between stability and preservation of update geometry, potentially improving learning dynamics and compute efficiency in LLM training.

Practically, the low-rank correction regime suggests partial SVD and randomized methods as preferred approaches when only a few directions require clipping. Global matrix-function methods may gain utility for more uniform spectra but necessitate stable matrix-function primitives and regularization.

Future research may address:

• SVD-free approximation schemes tailored to real-time optimizer use in LLMs
• Regularization strategies for eigenvalues near the clipping threshold
• Robust implementation of spectral clipping in distributed or hardware-accelerated training environments
• Empirical analysis contrasting MuCon with full polar orthogonalization in large transformer models, quantifying trade-offs in stability and feature learning preservation

Conclusion

MuCon introduces a mathematically rigorous, numerically tractable spectral clipping primitive for optimizer updates in LLM training, differentiating itself from Muon's polar orthogonalization by applying selective singular-value clipping. The study rigorously characterizes the mathematical structure and computational regime splits, advocating low-rank correction identities and matrix-function approaches as complements to dense SVD. Under SpectralP scaling, MuCon presents a promising direction for spectral control in deep transformer optimizers, contingent on robust matrix-function approximations and scalable implementation.