Low-Rank Matrix-Signed Gradient Descent

• The paper introduces low-rank matrix-signed gradient descent, which approximates gradient updates by applying the matrix sign to a low-rank approximation, ensuring efficient and stable updates.
• It employs low-rank orthogonalization through Gaussian sketching and QR factorization to reduce computational cost and filter out noise by suppressing small singular values.
• The method offers theoretical convergence guarantees and shows empirical gains in large-scale transformer training, outperforming traditional optimizers in various settings.

Low-rank matrix-signed gradient descent is a matrix-aware first-order optimization scheme in which a matrix update is first approximated in low rank and then replaced by its matrix sign, typically the orthogonal factor $UV^\top$ from a reduced SVD $M=U\Sigma V^\top$. In current usage, the term is associated primarily with low-rank orthogonalization and the resulting deterministic low-rank matrix-signed gradient descent and stochastic low-rank Muon methods for large-scale neural-network training (He et al., 15 Sep 2025). The defining idea is not elementwise sign quantization, but spectral normalization of a low-rank approximation to a gradient or momentum matrix. This places the topic at the intersection of matrix optimization, low-rank approximation, and Muon-style orthogonalized updates.

1. Conceptual basis and terminology

The underlying viewpoint is that many neural-network parameters are naturally matrices, and that training can therefore be formulated as matrix optimization: $\min_{X\in\mathbb{R}^{m\times n}} f(X).$ Within this viewpoint, standard optimizers such as SGD, Adam, and AdamW are treated as fundamentally vector or elementwise methods, whereas matrix-aware methods exploit singular directions, row and column structure, and two-sided geometry (He et al., 15 Sep 2025).

A central distinction is between elementwise sign and matrix sign. For a nonzero matrix $M\in\mathbb{R}^{m\times n}$ with reduced SVD

$M=U\Sigma V^\top,$

the matrix sign is defined as

$\mathrm{msgn}(M)=UV^\top.$

This operator discards singular values and preserves only singular directions. In the same source, matrix orthogonalization is identified with this operation because $UV^\top$ is the closest semi-orthogonal matrix to $M$ in Frobenius norm. The paper also states the steepest-descent interpretation

$$-\mathrm{msgn}(M)=\arg\min_{\|\Delta\|\le 1}\langle M,\Delta\rangle,$$

where $\|\cdot\|$ is the spectral norm (He et al., 15 Sep 2025).

This already separates low-rank matrix-signed methods from two nearby but distinct families. First, they are not signSGD-style coordinatewise sign methods. Second, they are not ordinary low-rank factor or projection methods whose low-rank structure is imposed on the parameter matrix itself. Here the low-rank object is the gradient or momentum matrix used to define the step.

2. Low-rank orthogonalization

The key mechanism is low-rank orthogonalization, which explicitly leverages the low-rank nature of gradients during neural-network training. Let $M=U\Sigma V^\top$0, let $M=U\Sigma V^\top$1, and choose a rank parameter $M=U\Sigma V^\top$2, typically $M=U\Sigma V^\top$3. The construction uses Gaussian sketching:

1. Draw $M=U\Sigma V^\top$4 Gaussian.
2. Form $M=U\Sigma V^\top$5.
3. Compute a QR factorization of $M=U\Sigma V^\top$6, obtaining a column-orthogonal matrix $M=U\Sigma V^\top$7.
4. Return

$M=U\Sigma V^\top$8

A central identity states that

$M=U\Sigma V^\top$9

so the returned matrix is exactly the matrix sign of the projected low-rank approximation $\min_{X\in\mathbb{R}^{m\times n}} f(X).$0 (He et al., 15 Sep 2025).

The approximation guarantee is stated in Frobenius norm. For any $\min_{X\in\mathbb{R}^{m\times n}} f(X).$1 with $\min_{X\in\mathbb{R}^{m\times n}} f(X).$2,

$\min_{X\in\mathbb{R}^{m\times n}} f(X).$3

Thus, if $\min_{X\in\mathbb{R}^{m\times n}} f(X).$4 is well approximated by rank $\min_{X\in\mathbb{R}^{m\times n}} f(X).$5, then the sketched projector $\min_{X\in\mathbb{R}^{m\times n}} f(X).$6 is also accurate (He et al., 15 Sep 2025).

The paper attributes two distinct benefits to this construction. Computationally, it replaces full orthogonalization of an $\min_{X\in\mathbb{R}^{m\times n}} f(X).$7 matrix by QR on $\min_{X\in\mathbb{R}^{m\times n}} f(X).$8 and matrix-sign computation on the smaller matrix $\min_{X\in\mathbb{R}^{m\times n}} f(X).$9. Statistically, it suppresses directions associated with very small singular values, which are described as unstable under noise. This suggests a spectral denoising effect in addition to runtime reduction.

3. Algorithmic forms

The deterministic low-rank matrix-signed gradient descent method applies low-rank orthogonalization directly to the full gradient. At iteration $M\in\mathbb{R}^{m\times n}$0, one computes $M\in\mathbb{R}^{m\times n}$1, obtains a low-rank approximation $M\in\mathbb{R}^{m\times n}$2, defines

$M\in\mathbb{R}^{m\times n}$3

and updates

$M\in\mathbb{R}^{m\times n}$4

This is the fixed-rank form of low-rank matrix-signed gradient descent (He et al., 15 Sep 2025).

A safeguarded variant imposes an explicit low-rank approximation condition,

$M\in\mathbb{R}^{m\times n}$5

and then uses the same update

$M\in\mathbb{R}^{m\times n}$6

In the theorem stated for this version, the schedules are

$M\in\mathbb{R}^{m\times n}$7

The stochastic extension is low-rank Muon. The original Muon update is

$M\in\mathbb{R}^{m\times n}$8

Low-rank Muon preserves the momentum recursion but applies low-rank orthogonalization to the momentum matrix rather than to the full matrix: $M\in\mathbb{R}^{m\times n}$9

$M=U\Sigma V^\top,$0

$M=U\Sigma V^\top,$1

The optimizer therefore differs from deterministic low-rank matrix-signed GD along two axes: it uses a stochastic oracle $M=U\Sigma V^\top,$2, and it orthogonalizes a momentum-like matrix rather than the raw gradient (He et al., 15 Sep 2025).

4. Theoretical guarantees

The theory is formulated for matrix optimization with a lower-bounded objective,

$M=U\Sigma V^\top,$3

and a nuclear/spectral dual-form Lipschitz condition,

$M=U\Sigma V^\top,$4

The stationarity notion is nuclear norm stationarity,

$M=U\Sigma V^\top,$5

A key descent lemma states that if

$M=U\Sigma V^\top,$6

then

$M=U\Sigma V^\top,$7

This makes the role of low-rank approximation explicit: progress is controlled by the nuclear norm of the true gradient and penalized by the approximation error $M=U\Sigma V^\top,$8 (He et al., 15 Sep 2025).

For fixed-rank low-rank matrix-signed GD with

$M=U\Sigma V^\top,$9

the paper proves that for all $\mathrm{msgn}(M)=UV^\top.$0,

$\mathrm{msgn}(M)=UV^\top.$1

When the weighted approximation-error sum remains $\mathrm{msgn}(M)=UV^\top.$2, the paper states that the method achieves

$\mathrm{msgn}(M)=UV^\top.$3

For the safeguarded variant, with

$\mathrm{msgn}(M)=UV^\top.$4

the paper defines

$\mathrm{msgn}(M)=UV^\top.$5

and proves that an $\mathrm{msgn}(M)=UV^\top.$6-approximate stationary point is reached when

$\mathrm{msgn}(M)=UV^\top.$7

that is, with complexity

$\mathrm{msgn}(M)=UV^\top.$8

For low-rank Muon, the stochastic oracle is assumed to satisfy a heavy-tailed moment condition: $\mathrm{msgn}(M)=UV^\top.$9 The theorem uses

$UV^\top$0

and yields approximate stochastic stationarity with complexity

$UV^\top$1

The source describes this as the first such result for a Muon-type algorithm under heavy-tailed noise (He et al., 15 Sep 2025).

5. Relation to earlier low-rank matrix optimization

Low-rank matrix-signed gradient descent emerged against a background of low-rank optimization methods that are structurally related but algorithmically different. Earlier work on low-rank policy-gradient reinforcement learning organized actor and critic parameters as low-rank matrices and optimized factor matrices by stochastic gradient ascent and descent; that work explicitly contains no signSGD-style method, no elementwise sign operator, and no matrix-sign update (Rozada et al., 2024). Studies of deep matrix factorization analyzed how vanilla gradient descent exhibits an implicit low-rank bias through mode-dependent spectral learning speeds and effective-rank plateaus, but again without an explicit signed-gradient rule (Chou et al., 2020). Scaled gradient descent for low-rank estimation used adaptive Gram-matrix preconditioners in factor space,

$UV^\top$2

to remove dependence on the matrix condition number, but this was explicitly presented as a conditioning-aware gradient method rather than a sign-based optimizer (Tong et al., 2020). Projected and Riemannian approaches likewise relied on truncated SVD projection, tangent-space projection, and retractions on the fixed-rank manifold, not on matrix-sign normalization (Zhang et al., 2024, Li et al., 2022).

This suggests that low-rank matrix-signed gradient descent should be read as an overview of two previously separate tendencies: low-rank exploitation and matrix-sign orthogonalization. The low-rank component comes from approximation of the update matrix; the signed component comes from replacing that approximation by its polar or orthogonal factor.

6. Empirical behavior, applications, and limitations

The main application reported so far is foundation-model training. The experiments treat transformer weight matrices blockwise and examine GPT-2 and LLaMA pretraining on FineWeb10B, FineWeb100B, and FineWebEdu10B. For Muon-type methods, embeddings and head layers are trained with AdamW, following the stated practice (He et al., 15 Sep 2025).

The paper presents singular-value plots of momentum updates $UV^\top$3 for Q/K/V matrices across layers and training iterations, and states that this partly supports the low-rank nature of the momentum updates. Synthetic GPU experiments also report that low-rank orthogonalization is substantially faster than full Newton–Schulz orthogonalization and truncated SVD, and that on noisy nearly-low-rank matrices it has much lower variance in estimated matrix signs than full Newton–Schulz orthogonalization (He et al., 15 Sep 2025).

In language-model pretraining, the results are mixed for small models and stronger for larger ones. For GPT-2 60M, low-rank Muon is reported to be worse than Muon, though still better than AdamW and SGDM. For larger GPT-2 models it generally improves on vanilla Muon; for example, on FineWeb10B at 350M, the reported validation perplexities are Muon 28.48, LR-Muon100 27.32, and LR-Muon200 25.64. On FineWebEdu10B at 1B, the corresponding values are Muon 19.54 and LR-Muon200 18.85. For LLaMA the gains are described as stronger; on FineWebEdu10B at 1B, the reported values are Muon 22.67 and LR-Muon200 19.54 (He et al., 15 Sep 2025).

Several caveats are explicit. The method can be less effective for smaller models; full-training speedups are smaller than orthogonalization-benchmark speedups because forward and backward passes dominate runtime; and rank selection remains a practical design parameter, with the main reported comparisons using ranks $UV^\top$4 and $UV^\top$5 (He et al., 15 Sep 2025). A plausible implication is that the method is most advantageous when optimizer cost is nontrivial and the gradient or momentum matrices exhibit strong spectral concentration.

Low-rank matrix-signed gradient descent therefore occupies a specific place in the optimizer landscape. It is neither generic low-rank factor optimization nor elementwise sign descent. Its defining operation is the matrix sign of a low-rank approximation, and its current significance lies in showing that low-rank spectral filtering and Muon-style orthogonalization can be combined without losing first-order convergence guarantees, while producing promising empirical behavior in large-model training (He et al., 15 Sep 2025).