---
title: Muon+ Optimizer for Deep Networks
url: https://www.emergentmind.com/topics/muon-eae2e0db-098b-4009-a245-35e29a268ed8
type: topic
---

# Muon+ Optimizer for Deep Networks

Muon+ is a first-order optimizer for deep networks, especially large language models, that extends the Muon optimizer by inserting one additional normalization step after orthogonalization. In Muon, a layer-wise momentum matrix is orthogonalized and used directly as the update direction; in Muon+, the orthogonalized matrix is further normalized along a chosen direction before the parameter update. The method is evaluated on GPT-style models ranging from 130M to 774M parameters and LLaMA-style models ranging from 60M to 1B parameters, in both the compute-optimal regime and an overtraining regime with a token-to-parameter ratio of approximately 200, where it yields consistent improvements in training and validation perplexity relative to Muon [2602.21545].

## 1. Conceptual position within matrix-structured optimization

Muon operates layer-wise in matrix form rather than on flattened parameter vectors. For each weight matrix \(W_t \in \mathbb{R}^{m \times n}\), it maintains a momentum matrix \(M_t\), orthogonalizes that matrix, and applies the resulting semi-orthogonal update with a global learning rate and a dimensional prefactor. In the formulation reported for Muon, the update is
\[
\begin{aligned}
\mathbf{M}_t &= \mu \mathbf{M}_{t-1} + (1 - \mu)\,\mathbf{G}_t \\
\mathbf{O}_t &= \mathrm{Ortho}(\mathbf{M}_t) \\
\mathbf{W}_{t} &= \mathbf{W}_{t-1} - \eta \cdot \sqrt{m/n} \cdot \mathbf{O}_t .
\end{aligned}
\]
Here \(\mathrm{Ortho}(\mathbf{M})\) is the semi-orthogonal matrix closest to \(\mathbf{M}\) in Frobenius norm, replacing the singular values of \(\mathbf{M}\) by 1 [2602.21545].

Muon+ preserves this structure and changes only the post-orthogonalization stage. Its update is
\[
\begin{aligned}
\mathbf{M}_t &= \mu \mathbf{M}_{t-1} + (1 - \mu)\,\mathbf{G}_t, \\
\mathbf{O}_t &= \mathrm{Norm}_{(d)}\!\left(\mathrm{Ortho}(\mathbf{M}_t)\right), \\
\mathbf{W}_t &= \mathbf{W}_{t-1} - \eta \cdot \sqrt{m/n} \cdot \mathbf{O}_t,
\end{aligned}
\]
where \(\mathrm{Norm}_{(d)}\) is an \(\ell_2\)-normalization operator applied along a chosen direction \(d \in \{\mathrm{col}, \mathrm{row}, \mathrm{col\_row}, \mathrm{row\_col}\}\) [2602.21545].

This design places Muon+ among optimizers that impose structural constraints on updates at the matrix level rather than relying on coordinate-wise second moments. The paper explicitly contrasts this with AdamW-style element-wise scaling and argues that the added normalization regularizes the structure of the orthogonalized update [2602.21545].

## 2. Mathematical definition of the additional normalization

For a matrix \(\mathbf{X} = [x_{ij}] \in \mathbb{R}^{m \times n}\), Muon+ defines several directional normalization operators. Column-wise normalization is
\[
\mathrm{Norm}_{(\mathrm{col})}(\mathbf{X})
:= \mathbf{X}\,\mathbf{D}_{\mathrm{col}}^{-1},
\]
with
\[
\mathbf{D}_{\mathrm{col}}
:= \mathrm{diag}\!\left(
\sqrt{\sum_{i=1}^{m} x_{i1}^{2}}, \ldots, \sqrt{\sum_{i=1}^{m} x_{in}^{2}}
\right).
\]
Row-wise normalization is
\[
\mathrm{Norm}_{(\mathrm{row})}(\mathbf{X})
:= \mathbf{D}_{\mathrm{row}}^{-1}\mathbf{X},
\]
with
\[
\mathbf{D}_{\mathrm{row}}
:= \mathrm{diag}\!\left(
\sqrt{\sum_{j=1}^{n} x_{1j}^{2}}, \ldots, \sqrt{\sum_{j=1}^{n} x_{mj}^{2}}
\right).
\]
Two composed variants are also defined:
\[
\mathrm{Norm}_{(\mathrm{col\_row})}(\mathbf{X})
:= \mathrm{Norm}_{(\mathrm{row})}\!\bigl(\mathrm{Norm}_{(\mathrm{col})}(\mathbf{X})\bigr),
\]
\[
\mathrm{Norm}_{(\mathrm{row\_col})}(\mathbf{X})
:= \mathrm{Norm}_{(\mathrm{col})}\!\bigl(\mathrm{Norm}_{(\mathrm{row})}(\mathbf{X})\bigr).
\]
An \(\epsilon\) term, for example \(10^{-8}\), is used for numerical stability in the implementation [2602.21545].

The orthogonalization itself is approximated in practice with Newton–Schulz polar iterations rather than a full SVD. Across the reported experiments, the paper uses 5 iterations for the polar step. This means that the only new hyperparameter introduced by Muon+ relative to Muon is the normalization direction \(d\), while the momentum coefficient, learning-rate schedule, and weight decay are inherited from the Muon baselines [2602.21545].

The interpretation advanced in the paper is that orthogonalization constrains angular structure but does not equalize row- or column-wise update magnitudes, whereas the extra normalization enforces a more uniform step scale along rows, columns, or both. The paper presents this as the principal reason for the empirical advantage of Muon+ over vanilla Muon [2602.21545].

## 3. Implementation regime and optimizer partitioning

Muon+ follows standard Muon practice in its parameter grouping. It is applied only to the “hidden” layers, specifically the main linear weights in attention and MLP blocks. Embeddings, unembeddings, layer norms, and positional encodings remain on AdamW. This split is used throughout the GPT and LLaMA experiments and is presented as the practical training configuration rather than as an auxiliary ablation [2602.21545].

The reported training stack uses the FineWeb corpus. GPT models use the GPT tokenizer with vocabulary size 50,257; LLaMA models use the LLaMA-2 tokenizer with vocabulary size 32,000. All training is performed in mixed precision with bfloat16 on H100 or A100 GPUs. The compute-optimal experiments follow the Hoffmann et al. scaling-law regime with token-to-parameter ratio approximately 20, and the overtraining experiments extend this to approximately 200 [2602.21545].

For GPT, the reported architectures are GPT-Small at 124M parameters, GPT-Base at 362M, and GPT-Large at 774M. Their token budgets are 3.0B, 7.2B, and 15.5B respectively. For LLaMA-style models, the paper reports 58M, 134M, 368M, and 1339M parameter models with token budgets 1.1B, 2.2B, 6.4B, and 13.1B respectively. The overtraining runs use 72B tokens for GPT-Base and 72B tokens for LLaMA-350M [2602.21545].

Relative to Muon, the additional computational overhead of Muon+ is described as negligible: it adds only tensor norms and divisions after orthogonalization. Relative to AdamW, however, Muon+ retains Muon’s more expensive polar-decomposition stage. The paper emphasizes that this extra cost is still dominated by the forward/backward pass in large-LLM settings [2602.21545].

## 4. Empirical results on GPT and LLaMA pre-training

In compute-optimal GPT training, Muon+ improves validation perplexity across all reported scales. For GPT-Small trained on 3.0B tokens, Muon records validation perplexity 29.66 and Muon+ records 27.64. For GPT-Base on 7.2B tokens, the values are 21.70 and 19.98. For GPT-Large on 15.5B tokens, they are 17.82 and 16.91. The paper notes that the absolute gain decreases with model size, but remains consistent [2602.21545].

In compute-optimal LLaMA-style training, Muon+ also improves on both AdamW and Muon. For the 60M model trained on 1.1B tokens, AdamW gives 33.10, Muon 25.75, and Muon+ 25.25. For 130M on 2.2B tokens, the values are 23.64, 19.06, and 18.65. For 350M on 6.4B tokens, they are 16.18, 14.02, and 13.41. For 1B on 13.1B tokens, they are 14.38, 10.68, and 10.31. In every listed configuration, Muon+ is the best-performing optimizer among those three [2602.21545].

The overtraining results preserve the same ordering. On GPT-Base trained with 72B tokens, Muon records validation perplexity 16.97 and Muon+ 15.84. On LLaMA-350M trained with 72B tokens, Muon records 11.48 and Muon+ 11.03. The training-loss curves are described as remaining consistently below Muon throughout training rather than converging back to the same trajectory late in optimization [2602.21545].

A plausible implication is that the benefit of the extra normalization is not limited to an early-training transient or a narrow scale regime. The reported evidence instead places it in both compute-optimal and long-horizon training settings, including token-to-parameter ratio near 200 [2602.21545].

## 5. Ablations: normalization direction, learning-rate sensitivity, and orthogonalization backend

The direction of normalization is a central ablation. In the paper’s LLaMA sweeps, every normalization variant outperforms pure Muon (“none”). For LLaMA-60M, the best validation perplexities are 25.75 for none, 25.34 for col, 25.29 for row, 25.27 for col_row, and 25.25 for row_col. For LLaMA-130M, the values are 19.34, 19.16, 18.98, 18.65, and 18.68. For LLaMA-350M, they are 14.11, 13.73, 13.46, 13.41, and 13.44. The paper summarizes this as “Row > Col” and “col_row and row_col are best and nearly identical” [2602.21545].

The learning-rate sweeps show that Muon+ is less sensitive to learning-rate choice, especially at larger model sizes. For LLaMA-350M at learning rate 0.02, Muon yields 14.11 and the best Muon+ variant, col_row, yields 13.43. At learning rate 0.04, Muon yields 14.02 and col_row yields 13.41. At still larger learning rates such as 0.06 and 0.08, the paper reports that Muon degrades more sharply than Muon+ [2602.21545].

The improvement is also reported to be independent of the particular orthogonalization backend. For LLaMA-350M, using the You et al. polar iteration with 5 iterations, Muon gives 14.01 and Muon+ 13.38. Using the Jordan implementation with 5 iterations, Muon gives 14.02 and Muon+ 13.41. Using PolarExpress with 5 iterations, Muon gives 13.90 and Muon+ 13.27. This suggests that the gain is attributable to the normalization stage rather than to a particular polar approximation [2602.21545].

The paper’s interpretation is that the missing ingredient in vanilla Muon is not a more elaborate manifold construction or second-moment adaptation, but the structural normalization applied after orthogonalization. That thesis is the organizing principle of the ablation section [2602.21545].

## 6. Relation to NorMuon, Mano, and adjacent optimizer proposals

The paper frames Muon+ as a minimal intervention that isolates the effect that, in its reading, accounts for much of the improvement previously associated with more elaborate Muon variants. In the reported GPT ablation against NorMuon, GPT-Small at learning rate 0.005 yields validation perplexity 29.66 for Muon, 27.91 for Muon+, 28.29 for NorMuon with \(\beta_1 = 0.95, \beta_2 = 0\), and 28.42 for NorMuon with \(\beta_1 = 0.95, \beta_2 = 0.95\). For GPT-Base, the corresponding values are 21.70, 19.98, 20.67, and 20.72 [2602.21545].

On this basis, the paper argues that NorMuon’s improvement over Muon is largely attributable to its normalization component, not to its second-moment machinery, because the \(\beta_2 = 0\) version already improves on Muon while the \(\beta_2 = 0.95\) version does not improve further. Muon+, which keeps only orthogonalization plus normalization, outperforms both NorMuon variants in those reported runs [2602.21545].

A comparable reading is offered for Mano. The Muon+ paper describes Mano as a manifold-based optimizer that projects updates onto an oblique manifold and then normalizes them, and it states that Mano’s own ablations indicate that update normalization is the main driver of gains, while the manifold-specific projection contributes much less. This suggests a broader optimizer pattern in which post-orthogonalization normalization, rather than manifold formalism itself, is the dominant intervention [2602.21545].

This does not establish equivalence between Muon+ and those methods, but it does motivate Muon+ as a stripped-down alternative: first-moment only, no extra running statistics beyond momentum, and no manifold bookkeeping beyond the existing orthogonalization step [2602.21545].

## 7. Practical usage, reproducibility, and limits of the reported evidence

For practical deployment, the paper recommends retaining Muon hyperparameters and introducing only the normalization direction. The reported best directions are usually col_row or row_col, with row also strong and typically better than col. Weight decay is fixed at 0.1 in all runs. GPT experiments use a NanoGPT-style schedule with constant learning rate for the first 40% of steps and then linear decay to zero; LLaMA experiments use cosine decay with 10% warmup. Learning-rate sweeps for Muon and Muon+ are reported rather than replaced by a different tuning protocol [2602.21545].

The study’s stated scope is causal language-model pre-training on FineWeb using GPT- and LLaMA-style architectures up to about 1B parameters. It does not evaluate other modalities, very small models, reinforcement-learning fine-tuning, or post-training alignment settings. It also reports that the absolute gains shrink at larger scales, even though they remain consistent in sign [2602.21545].

The implementation is released at the repository linked in the paper, and the reported training setup includes architecture tables, training configurations, and full learning-rate-by-normalization sweeps. This supports direct substitution of Muon+ for Muon in existing codebases that already use orthogonalized layer-wise updates [2602.21545].

Taken together, the available evidence positions Muon+ as a narrowly targeted modification of Muon rather than a wholesale replacement of optimizer design. Its defining claim is that the update
\[
\mathbf{O}_t = \mathrm{Norm}_{(d)}(\mathrm{Ortho}(\mathbf{M}_t))
\]
captures most of the practical benefit associated with more elaborate extensions, while preserving the original Muon structure and incurring negligible extra cost over Muon itself [2602.21545].

Source: https://www.emergentmind.com/topics/muon-eae2e0db-098b-4009-a245-35e29a268ed8