---
title: 'Muon-Signum: Hybrid Optimizer'
url: https://www.emergentmind.com/topics/muon-signum
type: topic
---

# Muon-Signum: Hybrid Optimizer

Searching arXiv for the cited Muon-related papers to ground the article in current literature.
Muon-Signum is a momentum-based optimization flow for smooth homogeneous neural networks that applies Muon to matrix-valued hidden-layer parameters and Signum to the final vector parameter under a common decaying learning rate. In the formulation studied in "The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks" [2602.16340], its significance is not merely algorithmic: Muon-Signum is analyzed as an approximate steepest-descent trajectory for a hybrid norm, and under stated assumptions its parameter direction converges to the direction of a Karush–Kuhn–Tucker (KKT) point of a corresponding max-margin problem. The optimizer therefore occupies a precise intermediate position between Muon’s spectral-norm bias and Signum’s $\ell_\infty$ bias, with experiments on two-layer homogeneous networks showing that the margin identity depends on the optimizer choice [2602.16340].

## 1. Definition and update rule

Muon-Signum is defined on network parameters $\theta=(W_1,\dots,W_K,u)$, where $W_1,\dots,W_K$ are the weight matrices of the hidden layers and $u$ is the final weight vector [2602.16340]. The method runs Muon on each $W_k$ and Signum on $u$, using a common decaying learning rate $\eta(t)$.

In continuous-time form, with momentum estimates $m_t^{(W_k)}$ for the matrix blocks and $m_t^{(u)}$ for the vector block, the dynamics are
$$
\frac{dW_k}{dt}
\;=\;
-\,\eta(t)\;
\arg\min_{X:\,\|X\|_{\mathrm{sp}}=1}
\bigl\langle X,m_t^{(W_k)}\bigr\rangle
\;=\;
-\,\eta(t)\,U_k(t)V_k(t)^T,
$$
where $m_t^{(W_k)}$ has singular value decomposition $U_k\Sigma_k V_k^T$, and
$$
\frac{du}{dt}
\;=\;
-\,\eta(t)\;\mathrm{sign}\bigl(m_t^{(u)}\bigr),
$$
with $\mathrm{sign}(\cdot)$ applied coordinate-wise [2602.16340]. Here $\|\cdot\|_{\mathrm{sp}}$ is the spectral norm.

This blockwise construction aligns directly with standard formulations of the two constituent optimizers. Muon updates a matrix parameter by the polar factor of a momentum matrix, $W_{t+1}=W_t-\alpha\,\mathrm{Polar}(M_t)$, where $\mathrm{Polar}(M_t)=U_tV_t^\top$ for $M_t=U_t\Sigma_tV_t^\top$ [2602.09314]. Signum updates by the coordinate-wise sign of an exponential moving average, $\theta_{t+1}=\theta_t-\alpha\,\mathrm{sign}(m_t)$ [2602.09314]. Muon-Signum combines these two geometries in a single optimizer state partition.

A useful contextual observation from later work is that Muon and Signum can both be written as linear minimization oracle steps for different norms: Muon for the spectral norm and Signum for the $\ell_\infty$ norm [2605.19811]. This suggests a common geometric interpretation rather than an ad hoc combination of unrelated update rules.

## 2. Hybrid norm and steepest-descent interpretation

The analysis in [2602.16340] views Muon-Signum as a special case of normalized momentum-steepest-descent on the hybrid norm
$$
\|\,(W,u)\|_{\max}
\;=\;
\max\{\|W\|_{\mathrm{sp}},\|u\|_\infty\}.
$$
Its dual norm is
$$
\|\,(G,g)\|_*
\;=\;
\|G\|_{1,\mathrm{nucl}}+\|g\|_1,
$$
where $\|G\|_{1,\mathrm{nucl}}$ is the sum of singular values of the matrix gradient block and $\|g\|_1$ is the $\ell_1$ norm of the vector gradient block [2602.16340].

A central fact used in the paper is that simultaneous normalized momentum-steepest-descent steps on each block with respect to the block’s own norm are equivalent to a single normalized momentum-steepest-descent step for the max norm. If $\nu(t)=\eta(t)$ and $M_t=(m_t^{(W)},m_t^{(u)})$ denotes the full momentum estimate, then
$$
\frac{d\theta}{dt}
=
\eta(t)\;
\arg\min_{\|\Delta\|_{\max}=1}
\langle \Delta,M_t\rangle
=
-\,\eta(t)\,
\arg\max_{\|\Delta\|_{\max}=1}
\langle M_t,\Delta\rangle.
$$
Under a decaying schedule satisfying $\eta(t)=o(t^{1/L-1})$ with $\int_0^\infty \eta=\infty$, together with mild trajectory-boundedness and directional-convergence assumptions, the approximate-steepest-descent framework yields directional convergence to a KKT solution of the max-margin problem in the max norm [2602.16340].

This places Muon-Signum in the same broad family as other geometry-driven optimizers whose implicit bias is characterized via steepest descent in a non-Euclidean norm. Related work on non-Euclidean gradient noise scales makes the same geometric distinction explicit: Signum is associated with $\ell_\infty/\ell_1$ geometry, and Muon with spectral/nuclear geometry [2602.03001]. That work addresses stochastic batch-size adaptation rather than implicit bias, but it reinforces the interpretation that Muon-Signum combines two specific primal-dual geometries rather than simply mixing update heuristics.

## 3. Margin-maximization problem and KKT structure

For an $L$-homogeneous separable model $f(x;\theta)$, the relevant margin-maximization problem is stated in [2602.16340] as
$$
\min_\theta \;\tfrac12\,\|\theta\|_{\max}^2
\quad \text{s.t.}\quad
\forall i,\; y_i f(x_i;\theta)\ge 1,
$$
where
$$
\|\theta\|_{\max}
=
\max\{\|W\|_{\mathrm{sp}},\|u\|_\infty\}.
$$

A feasible point $\theta^*$ is a KKT point if there exist multipliers $\lambda_i\ge 0$, subgradients
$$
H_i\in \partial_\theta f(x_i;\theta^*),
$$
and
$$
k\in \partial\!\left(\tfrac12\|\theta^*\|_{\max}^2\right)
$$
such that

1. $\sum_n \lambda_i y_i H_i - k = 0$,
2. $\sum_n \lambda_i (y_i f(x_i;\theta^*)-1)=0$ [2602.16340].

By the dual-of-max-norm fact cited in the same work, the subgradient term $k$ decomposes into a nuclear-norm subgradient on the matrix block $W$ and a sign-vector contribution on the vector block $u$ [2602.16340]. This decomposition is the variational counterpart of the optimizer itself: matrix blocks are controlled by spectral geometry, while the final vector is controlled by $\ell_\infty$ geometry.

The KKT characterization matters because the implicit-bias statement is not merely that Muon-Signum tends toward large-margin solutions in an informal sense. The convergence target is the direction of a KKT point of a specific constrained optimization problem, with the hybrid norm determining which parameters are margin-limiting. The paper summarizes this as maximizing the “bottleneck” margin in whichever block is largest [2602.16340]. A plausible implication is that the selected classifier can differ materially from both pure spectral-margin and pure $\ell_\infty$-margin solutions when the hidden-layer matrix norm and final-layer vector norm compete.

## 4. Implicit-bias theorem

The main implicit-bias theorem for Muon-Signum in [2602.16340] assumes:

- $f$ is $C^1$-smooth and $L$-homogeneous,
- $\eta(t)>0$ is non-increasing with $\int_0^\infty \eta=\infty$ and $\eta(t)=o(t^{1/L-1})$,
- the trajectory remains bounded away from the origin and converges in direction to some $\bar\theta$ with positive margin.

Under these assumptions, Muon-Signum satisfies
$$
\theta_t/\|\theta_t\|\to \bar\theta,\qquad \|\bar\theta\|>0,
$$
and $\bar\theta$ is the direction of a KKT point of the max-margin program defined by the hybrid norm [2602.16340]. The paper states this as Corollary “Muon-Signum.”

The theorem extends earlier lines of work in two directions identified in the paper’s abstract. First, it extends steepest-descent implicit-bias results from homogeneous models to normalized steepest descent with an optional learning-rate schedule. Second, it shows that momentum steepest-descent algorithms such as Muon, MomentumGD, and Signum are approximate steepest-descent trajectories under a decaying learning-rate schedule, then extends this analysis to Adam, Muon-Signum, and Muon-Adam [2602.16340].

The specific placement of Muon-Signum relative to Adam is technically important. The same paper states that Adam, without the stability constant, maximizes the $\ell_\infty$ margin, whereas Muon-Signum and Muon-Adam maximize a hybrid norm [2602.16340]. This indicates that the hybrid-norm phenomenon is not unique to one optimizer construction; it arises whenever different parameter blocks are assigned distinct geometries.

## 5. Relation to Muon, Signum, Adam, and related hybrids

The comparison in [2602.16340] is explicit. Muon alone is normalized momentum-steepest descent in the spectral norm of $W$ and $\ell_2$ for vector blocks, so it converges to the spectral-norm margin-maximizer. Signum alone is normalized momentum-steepest descent in the $\ell_\infty$ norm of $\theta$, hence it maximizes the $\ell_\infty$ hard margin. Muon-Signum instead uses
$$
\|\,(W,u)\|=\max\{\|W\|_{\mathrm{sp}},\|u\|_\infty\},
$$
so its implicit solution is the one maximizing the hybrid bottleneck margin [2602.16340].

The same source reports the practical comparison as follows: Signum $\approx$ Adam for the $\ell_\infty$ margin, Muon $\approx$ NGD for spectral margin, and Muon-Signum lies in between, matching whichever block dominates [2602.16340]. This is not a statement of interpolation by averaging; rather, it is interpolation induced by a max norm across heterogeneous parameter blocks.

Additional literature sharpens the algorithmic relationship between the components. "Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory" [2602.09314] presents a unified ladder
Element-wise sign-descent $\rightarrow$ Signum $\rightarrow$ Adam and Matrix spectral-descent $\rightarrow$ Muon $\rightarrow$ Shampoo,
with the latter optimizers in each chain adding adaptation beyond pure normalized descent. That paper also states that Muon consistently outperforms Signum on language-model perplexity under its reported settings, while Shampoo improves further via left/right adaptation on weight matrices [2602.09314]. These results concern data efficiency rather than implicit bias, but they contextualize why a hybrid such as Muon-Signum is of interest: it combines a strong matrix direction with a cheap sign-based vector update.

A separate line of work, "LionMuon: Alternating Spectral and Sign Descent for Efficient Training" [2605.19811], studies alternation between Muon and sign-based steps rather than the blockwise hybrid construction of Muon-Signum. There, Muon and Signum are again treated as norm-specific linear minimization oracle updates, and the analysis proves interpolation between spectral and $\ell_\infty$ regimes under heavy-tailed noise [2605.19811]. This suggests a broader pattern: several recent optimizers mix spectral and sign geometries, but Muon-Signum is distinctive in [2602.16340] because its hybrid norm arises from simultaneous blockwise normalized momentum-steepest-descent and yields a direct max-margin interpretation.

## 6. Experimental evidence and observed behavior

The experiments reported for Muon-Signum in [2602.16340] use two-layer homogeneous networks with a hidden layer and scalar output, trained on MNIST even/odd with exponential loss and a decaying schedule $\eta_0 t^{-0.8}$. The study considers both ReLU and smooth squared-ReLU activations.

The paper reports the following pattern in Figure 2a:

| Optimizer | Reported largest margin |
|---|---|
| NGD ($\ell_2$) | largest $\ell_2$ margin |
| Signum/Adam | largest $\ell_\infty$ margin |
| Muon | largest spectral margin |
| Muon-Signum | largest hybrid margin $\max\{\text{spectral},\infty\}$ |

The experiments also plot the cosine alignment of $\theta_t$ with $\theta_{\text{final}}$, which confirms directional convergence in Figure 2b [2602.16340]. The reported empirical outcome is therefore closely aligned with the theoretical picture: Muon-Signum converges to the max-margin solution associated with the hybrid norm and realizes the corresponding margin values in practice.

The validation strategy is notable because it isolates the optimizer-dependent geometry in homogeneous networks, where implicit-bias results are most transparent. This differs from large-scale language-model studies of Muon and Signum, which focus on token efficiency, perplexity, or compute tradeoffs. For example, [2602.09314] reports final validation perplexities for Muon and Signum on C4 with LLaMA 3 architectures, and [2605.19811] studies validation loss versus FLOPs for alternating Muon-sign methods. Those works demonstrate practical performance differences between spectral and sign updates, but the Muon-Signum experiments in [2602.16340] are designed specifically to validate margin predictions rather than downstream scaling behavior.

A plausible implication is that Muon-Signum’s most direct theoretical relevance lies in settings where parameter homogeneity and separability make margin-based implicit bias observable, while its practical relevance in broader stochastic training may depend on how strongly the matrix and vector blocks differ in their effective geometry.

## 7. Interpretation, scope, and limitations

Within the framework of [2602.16340], Muon-Signum should be understood as the composite normalized-momentum-steepest-descent algorithm associated with the max of spectral and $\ell_\infty$ norms. Its defining property is not simply that it mixes Muon and Signum updates, but that this mixture corresponds to a single hybrid norm and therefore to a single hybrid max-margin problem.

Several misconceptions are thereby excluded. Muon-Signum is not presented as optimizing a weighted sum of spectral and $\ell_\infty$ margins; the relevant norm is the maximum of the two block norms [2602.16340]. Nor is its limit characterized as an arbitrary compromise between Muon and Signum. The theory states that the direction converges to that of a KKT point of a precisely defined constrained problem, and the optimizer “sits in between” only because the max norm selects whichever block is limiting [2602.16340].

The formal guarantees are conditional. The theorem requires $C^1$ smoothness, $L$-homogeneity, a non-increasing learning rate with $\int_0^\infty\eta=\infty$ and $\eta(t)=o(t^{1/L-1})$, and assumptions that the trajectory stays bounded away from the origin and converges in direction to a positive-margin limit [2602.16340]. The results therefore do not by themselves establish identical behavior for arbitrary architectures, losses, or finite-step stochastic training procedures.

Recent adjacent work indicates broader relevance of the underlying geometric split. Non-Euclidean batch-size adaptation for Signum and Muon uses optimizer-aligned noise scales derived from the corresponding dual norms, with reported reductions in training steps of up to 66% on a 160 million parameter Llama model [2602.03001]. Alternating spectral and sign descent has also been shown to interpolate between the two regimes in both theory and empirical compute efficiency [2605.19811]. These results do not analyze Muon-Signum itself, but they support the view that spectral and sign geometries form a coherent design space for modern optimizers.

In summary, Muon-Signum is a hybrid optimizer whose matrix blocks follow Muon’s spectral descent and whose final vector block follows Signum’s coordinate-wise sign descent, all under a common momentum-steepest-descent interpretation. For smooth homogeneous neural networks, its implicit bias is toward KKT points of the max-margin problem induced by the norm $\max\{\|W\|_{\mathrm{sp}},\|u\|_\infty\}$, and experiments reported in [2602.16340] show that it realizes the corresponding hybrid margin behavior in practice.

Source: https://www.emergentmind.com/topics/muon-signum