Low-Rank Muon Optimization

• Low-Rank Muon is a family of optimization methods that leverages low-rank approximations to modify matrix updates, thereby reducing computational cost and enhancing stability.
• The approach integrates low-rank manipulations in orthogonalization, parameter manifold updates, and optimizer state compression to achieve faster convergence and memory efficiency.
• Empirical studies show that low-rank variants outperform standard optimizers in large-scale language model pretraining and fine-tuning, while maintaining favorable spectral properties.

Low-Rank Muon designates a family of Muon-derived optimization methods in which low-rank structure is introduced into the orthogonalization step, the parameter manifold, the optimizer state, or the update constraint. Muon itself replaces a matrix gradient or momentum $M=U\Sigma V^\top$ by its polar factor $UV^\top$, preserving singular directions while flattening the update spectrum. Low-rank variants modify this basic operation in distinct ways: some approximate the matrix sign on a rank-$r$ sketch, some derive Muon-style spectral steepest descent directly on a low-rank manifold such as LoRA, and others use low-rank factorization or projection to reduce memory, improve compressibility, or control update geometry. The literature also emphasizes that Muon is not, in itself, a low-rank regularizer: its characteristic bias is toward flat spectra rather than nuclear-norm concentration (He et al., 15 Sep 2025, Cesista et al., 11 Jun 2026, Beneventano et al., 7 Jun 2026).

1. Muon as a matrix optimizer and the loci of low rank

Muon is a matrix-aware optimizer. In the notation used across this literature, a momentum buffer is formed and then orthogonalized, for example

$$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$

with $\operatorname{Orth}(X)=UV^\top$ when $X=U\Sigma V^\top$. Equivalent formulations write the update as $X^{k+1}=X^k-\eta_k\,\mathrm{msgn}(M^k)$. In all cases, the defining step is spectral orthogonalization: the update keeps singular directions and flattens singular values.

Variant	Low-rank locus	Reported purpose
Low-rank Muon (He et al., 15 Sep 2025)	Orthogonalization on a rank-$r$ sketch $QQ^\top M$	Faster and more robust matrix optimization; promising GPT-2 and LLaMA pretraining
LoRA-Muon (Cesista et al., 11 Jun 2026)	Low-rank manifold $W=AB^\top$ and tangent-space spectral step	A good low-rank proxy for full-rank Muon and Shampoo-family optimizers
LoRA-Pre with Muon (Wang et al., 27 Feb 2026)	Low-rank factorization of momentum state	Reduced optimizer memory footprint with maintained optimization performance
NuMuon (Dolatabadi et al., 4 Mar 2026)	Nuclear-norm-constrained top-$UV^\top$0 Muon update	Increased weight compressibility and improved post-compression quality
GUM (Pan et al., 20 Oct 2025)	Low-rank gradient projection with unbiased layerwise sampling	Convergence guarantees matching base Muon with low-rank memory efficiency

These variants differ in where low rank enters the algorithmic pipeline. In low-rank orthogonalization, it is an approximation to the matrix sign itself. In LoRA-Muon, it is the geometry of the parameter space. In LoRA-Pre and GUM, it is imposed on the optimizer state or gradient storage. In NuMuon, it is enforced directly on the update direction. This suggests that low-rank Muon is best understood as a design space around Muon’s spectral step rather than as a single optimizer.

2. Low-rank orthogonalization as the narrow sense of Low-Rank Muon

In the narrow sense introduced by the paper on low-rank orthogonalization, Low-Rank Muon replaces full-matrix orthogonalization by a sketched procedure. Given $UV^\top$1 and a target rank $UV^\top$2, the algorithm draws a random Gaussian matrix $UV^\top$3, computes $UV^\top$4, projects to $UV^\top$5, computes its matrix sign, and lifts the result back: $UV^\top$6 Equivalently, $UV^\top$7. The same paper gives the approximation bound

$UV^\top$8

and uses the momentum update

$UV^\top$9

The stated motivation is that gradient matrices in modern deep networks are approximately low-rank in practice, while full-rank orthogonalization can be computationally heavy and unstable in noisy or ill-conditioned settings. The paper reports that low-rank Muon achieves promising results in GPT-2 and LLaMA pretraining and surpasses the performance of carefully tuned vanilla Muon. For large models, specifically 350M and 1B, low-rank Muon is reported to outperform Muon, AdamW, and SGD in validation perplexity, while saving up to $r$0–$r$1 in observed GPU time; for smaller 60M models, it is slightly worse than Muon but still better than adaptive and vanilla optimizers. The same work states that ranks 100 or 200 are used in practice for LLMs (He et al., 15 Sep 2025).

3. Spectral steepest descent on the low-rank manifold

LoRA-Muon derives Muon’s spectral steepest-descent rule directly for low-rank adaptation. In LoRA, the trainable matrix is parameterized as $r$2 with rank $r$3, and the relevant geometry is the tangent space

$r$4

Rather than optimizing factors independently, LoRA-Muon solves a decoupled trust-region problem in this tangent space and specializes the linear minimization oracle to the spectral norm. The resulting update uses Gram whitening and the matrix sign: $r$5 with $r$6, $r$7, $r$8, and $r$9. The method is paired with a split weight-decay rule

$$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$0

which is described as maintaining gauge invariance and matching the regularization effect of weight decay on $$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$1.

The empirical claim of LoRA-Muon is not merely that it works at low rank, but that it preserves the learning-rate geometry of dense spectral optimization. Its optimal learning rates are reported to transfer across rank, width, depth, and factor-rescaling. In a compute-matched TinyShakespeare study, a rank-$$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$2 proxy recovers the dense best tested learning rate, and a rank-$$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$3 LoRA-Muon run attains lower mean validation loss than the dense baseline in the seed-averaged sweep. The rank-$$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$4 proxy uses about $$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$5 as many trainable parameters and allows $$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$6 as many optimizer steps under a fixed compute budget. The paper further states that LoRA-Muon is distinct from Spectron because it is gauge-invariant, and that it computes the same spectral update as the simplified QR-coordinate core of LoRA-RITE without QR decomposition and without storing second moments, making it more accelerator-friendly and memory-efficient (Cesista et al., 11 Jun 2026).

4. Low-rank state compression, unbiased projection, and compressibility-oriented updates

A separate line of work applies low-rank structure to Muon’s optimizer state rather than to the parameterization. LoRA-Pre interprets the exponential moving average as online linear regression: $$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$7 which is a gradient step on $$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$8. It then factorizes the momentum as $$M_t=\beta M_{t-1}+(1-\beta)\frac{\partial E}{\partial W^\ell}, \qquad W_{t+1}=W_t^\ell-\eta\,\operatorname{Orth}(M_t^\ell),$$9, derives closed-form low-rank updates for the factors, and extends the same framework to Muon’s first-order state. The paper reports that LoRA-Pre achieves the highest performance across Llama-family models from 60M to 1B, and that it achieves comparable or superior results using only $\operatorname{Orth}(X)=UV^\top$0 the rank of baseline methods. In fine-tuning, it reports gains of 3.14 points on Llama-3.1-8B and 6.17 points on Llama-2-7B relative to standard LoRA (Wang et al., 27 Feb 2026).

GUM addresses a different low-rank problem: the bias introduced by gradient low-rank projection. It combines GaLore’s projection mechanism with Muon and a layerwise sampling scheme. With probability $\operatorname{Orth}(X)=UV^\top$1, a block receives the compensated full-rank component

$\operatorname{Orth}(X)=UV^\top$2

and with probability $\operatorname{Orth}(X)=UV^\top$3, it receives the rescaled low-rank component

$\operatorname{Orth}(X)=UV^\top$4

This gives an unbiased estimator $\operatorname{Orth}(X)=UV^\top$5 with $\operatorname{Orth}(X)=UV^\top$6. The paper proves that GUM matches the convergence guarantees of the base Muon algorithm while preserving the memory efficiency of low-rank techniques, and reports non-trivial improvements over GaLore as well as better performance than full-parameter training in LLM fine-tuning and pretraining (Pan et al., 20 Oct 2025).

NuMuon instead imposes low rank directly on the Muon update through a nuclear-norm constraint. Starting from Muon’s linear minimization oracle, it restricts

$\operatorname{Orth}(X)=UV^\top$7

so that the optimal update becomes

$\operatorname{Orth}(X)=UV^\top$8

The motivation is empirical: despite full-rank updates, Muon-trained models exhibit pronounced low-rank structure in their weight matrices and are readily compressible under standard pipelines. Across billion-parameter-scale models, NuMuon is reported to increase weight compressibility and improve post-compression model quality while retaining Muon’s favorable convergence behavior; the same paper reports average improvements in post-compression accuracy of up to 55.8% (Dolatabadi et al., 4 Mar 2026).

5. Rank collapse, effective dimensionality, and Muon as a geometric remedy

One of the most explicit uses of Muon against a low-rank pathology appears in work on Feedback Alignment (FA). There, rank collapse refers to error signals and gradients becoming constrained to a low-dimensional subspace. The paper defines the effective rank of a matrix $\operatorname{Orth}(X)=UV^\top$9 as

$X=U\Sigma V^\top$0

Using CIFAR10 and deeper CNNs, the authors report that FA gradients have considerably lower effective rank than BP gradients, and that the gradient trajectory covariance also collapses. For a 4-layer CNN, the reported effective rank is 12 for FA versus 100 for BP. The proposed interpretation is that FA explores only a small subspace of parameter space, while Muon mitigates this by orthogonalizing the momentum update so that all nonzero singular directions are weighted equally. On CIFAR100 with ResNet-18, the reported accuracies are 1.4% for the FA baseline, 25.3% for FA + Muon, 37.1% for FA + BatchNorm, and 46.1% for FA + Muon + BatchNorm (Boeshertz et al., 9 Jun 2026).

The same study uses several controls to isolate geometry from noise. Artificially restricting updates to the top-2 or top-5 singular directions devastates FA but not BP, which the paper interprets as evidence that FA is unusually sensitive to reduced update dimensionality. Injecting high-dimensional isotropic noise does not help, indicating that merely adding random directions is insufficient. By contrast, interpolating between SGD and Muon through spectral flattening with the Freon optimizer improves FA as the singular-value spectrum becomes flatter. In this setting, low rank is the failure mode, and Muon is introduced as a mechanism for restoring effective dimensionality.

6. Misconceptions, regime dependence, and adjacent applications

A recurring misconception is that Muon should be interpreted as an implicit low-rank regularizer. The spectral-analysis literature rejects this interpretation explicitly. Muon replaces $X=U\Sigma V^\top$1 by its polar factor $X=U\Sigma V^\top$2, which yields a flat update spectrum. Under explicit alignment assumptions, the polar update is proved to be the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, the paper derives exact singular-value dynamics and identifies a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. At fixed Frobenius norm, the distinguished state therefore has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments report that Muon’s converged nuclear norm is higher than the actual minimum-nuclear-norm interpolant, with a mean $X=U\Sigma V^\top$3 gap for $X=U\Sigma V^\top$4; in small NanoGPT pretraining Muon improves validation loss relative to AdamW, while in a matched small-ViT control the ranking reverses (Beneventano et al., 7 Jun 2026).

A second limitation is that uniform spectral whitening can fail outside the pretraining regimes in which Muon was initially successful. In cross-modality vision-language-action training, the action module is reported to have inherently low-rank gradients, and in reinforcement learning with verifiable rewards the gradients have low signal-to-noise ratio. In both cases, the criticism is that Muon drives all singular values toward 1 and therefore amplifies noisy tail directions. Pion is proposed as a drop-in replacement that uses a two-stage Promotion+Suppression mechanism, described as a high-pass Newton–Schulz iteration: dominant singular values are anchored at 1 and noisy tail components are suppressed toward 0. On LIBERO Object after 1,500 training steps with VLA-Adapter, the reported success rates are 100% for Pion, 97.0% for Muon, and 32.2% for AdamW; in RLVR post-training on Qwen3-1.7B/4B with GRPO and GMPO, Muon is reported to collapse to zero while Pion outperforms AdamW on MATH and GSM8K (Fan et al., 19 May 2026).

A third issue concerns fine-tuning models pretrained with a different optimizer. Controlled experiments on Adam-pretrained models report that naively switching to Muon degrades performance because of optimizer mismatch, and that the disruption scales with update strength. LoRA mitigates this by constraining the update: the reported reduction in the performance gap between Adam and Muon under full fine-tuning is 39–78%, sometimes vanishing entirely, and for small to moderate ranks LoRA-Muon matches or beats LoRA-Adam; as rank grows, the mismatch gap returns (Qu et al., 11 May 2026). Beyond mainstream deep-network training, Muon steps have also been inserted into low separation rank tensor generalized linear models. LSRTR-M preserves the original block coordinate scheme while replacing projection-based factor updates with Muon steps, and is reported to converge faster in synthetic linear, logistic, and Poisson LSR-TGLMs while reducing training time on Vessel MNIST 3D by about 50% and maintaining competitive classification performance (Liang et al., 6 Apr 2026). Taken together, these results suggest that low-rank Muon methods are most coherent when low-rank structure is used as a computational or geometric design constraint, not when Muon is assumed to behave like a nuclear-norm or rank-minimizing regularizer.