Muon-Signum: Hybrid Optimizer

• Muon-Signum is a hybrid optimizer that combines Muon’s spectral descent on matrix parameters with Signum’s coordinate-wise updates on vector parameters under a decaying learning rate.
• It interprets updates as approximate steepest-descent steps for a hybrid max norm, guiding parameter directions toward KKT points of a constrained max-margin problem.
• Empirical results on two-layer homogeneous networks validate that Muon-Signum interpolates between spectral and ℓ∞ margin biases, confirming its theoretical implicit bias.

Searching arXiv for the cited Muon-related papers to ground the article in current literature. Muon-Signum is a momentum-based optimization flow for smooth homogeneous neural networks that applies Muon to matrix-valued hidden-layer parameters and Signum to the final vector parameter under a common decaying learning rate. In the formulation studied in "The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks" (Gronich et al., 18 Feb 2026), its significance is not merely algorithmic: Muon-Signum is analyzed as an approximate steepest-descent trajectory for a hybrid norm, and under stated assumptions its parameter direction converges to the direction of a Karush–Kuhn–Tucker (KKT) point of a corresponding max-margin problem. The optimizer therefore occupies a precise intermediate position between Muon’s spectral-norm bias and Signum’s $\ell_\infty$ bias, with experiments on two-layer homogeneous networks showing that the margin identity depends on the optimizer choice (Gronich et al., 18 Feb 2026).

1. Definition and update rule

Muon-Signum is defined on network parameters $\theta=(W_1,\dots,W_K,u)$, where $W_1,\dots,W_K$ are the weight matrices of the hidden layers and $u$ is the final weight vector (Gronich et al., 18 Feb 2026). The method runs Muon on each $W_k$ and Signum on $u$, using a common decaying learning rate $\eta(t)$.

In continuous-time form, with momentum estimates $m_t^{(W_k)}$ for the matrix blocks and $m_t^{(u)}$ for the vector block, the dynamics are

$$\frac{dW_k}{dt} \;=\; -\,\eta(t)\; \arg\min_{X:\,\|X\|_{\mathrm{sp}}=1} \bigl\langle X,m_t^{(W_k)}\bigr\rangle \;=\; -\,\eta(t)\,U_k(t)V_k(t)^T,$$

where $\theta=(W_1,\dots,W_K,u)$0 has singular value decomposition $\theta=(W_1,\dots,W_K,u)$1, and

$\theta=(W_1,\dots,W_K,u)$2

with $\theta=(W_1,\dots,W_K,u)$3 applied coordinate-wise (Gronich et al., 18 Feb 2026). Here $\theta=(W_1,\dots,W_K,u)$4 is the spectral norm.

This blockwise construction aligns directly with standard formulations of the two constituent optimizers. Muon updates a matrix parameter by the polar factor of a momentum matrix, $\theta=(W_1,\dots,W_K,u)$5, where $\theta=(W_1,\dots,W_K,u)$6 for $\theta=(W_1,\dots,W_K,u)$7 (Eschenhagen et al., 10 Feb 2026). Signum updates by the coordinate-wise sign of an exponential moving average, $\theta=(W_1,\dots,W_K,u)$8 (Eschenhagen et al., 10 Feb 2026). Muon-Signum combines these two geometries in a single optimizer state partition.

A useful contextual observation from later work is that Muon and Signum can both be written as linear minimization oracle steps for different norms: Muon for the spectral norm and Signum for the $\theta=(W_1,\dots,W_K,u)$9 norm (Bolatov et al., 19 May 2026). This suggests a common geometric interpretation rather than an ad hoc combination of unrelated update rules.

2. Hybrid norm and steepest-descent interpretation

The analysis in (Gronich et al., 18 Feb 2026) views Muon-Signum as a special case of normalized momentum-steepest-descent on the hybrid norm

$W_1,\dots,W_K$0

Its dual norm is

$W_1,\dots,W_K$1

where $W_1,\dots,W_K$2 is the sum of singular values of the matrix gradient block and $W_1,\dots,W_K$3 is the $W_1,\dots,W_K$4 norm of the vector gradient block (Gronich et al., 18 Feb 2026).

A central fact used in the paper is that simultaneous normalized momentum-steepest-descent steps on each block with respect to the block’s own norm are equivalent to a single normalized momentum-steepest-descent step for the max norm. If $W_1,\dots,W_K$5 and $W_1,\dots,W_K$6 denotes the full momentum estimate, then

$W_1,\dots,W_K$7

Under a decaying schedule satisfying $W_1,\dots,W_K$8 with $W_1,\dots,W_K$9, together with mild trajectory-boundedness and directional-convergence assumptions, the approximate-steepest-descent framework yields directional convergence to a KKT solution of the max-margin problem in the max norm (Gronich et al., 18 Feb 2026).

This places Muon-Signum in the same broad family as other geometry-driven optimizers whose implicit bias is characterized via steepest descent in a non-Euclidean norm. Related work on non-Euclidean gradient noise scales makes the same geometric distinction explicit: Signum is associated with $u$0 geometry, and Muon with spectral/nuclear geometry (Naganuma et al., 3 Feb 2026). That work addresses stochastic batch-size adaptation rather than implicit bias, but it reinforces the interpretation that Muon-Signum combines two specific primal-dual geometries rather than simply mixing update heuristics.

3. Margin-maximization problem and KKT structure

For an $u$1-homogeneous separable model $u$2, the relevant margin-maximization problem is stated in (Gronich et al., 18 Feb 2026) as

$u$3

where

$u$4

A feasible point $u$5 is a KKT point if there exist multipliers $u$6, subgradients

$u$7

and

$u$8

such that

1. $u$9,
2. $W_k$0 (Gronich et al., 18 Feb 2026).

By the dual-of-max-norm fact cited in the same work, the subgradient term $W_k$1 decomposes into a nuclear-norm subgradient on the matrix block $W_k$2 and a sign-vector contribution on the vector block $W_k$3 (Gronich et al., 18 Feb 2026). This decomposition is the variational counterpart of the optimizer itself: matrix blocks are controlled by spectral geometry, while the final vector is controlled by $W_k$4 geometry.

The KKT characterization matters because the implicit-bias statement is not merely that Muon-Signum tends toward large-margin solutions in an informal sense. The convergence target is the direction of a KKT point of a specific constrained optimization problem, with the hybrid norm determining which parameters are margin-limiting. The paper summarizes this as maximizing the “bottleneck” margin in whichever block is largest (Gronich et al., 18 Feb 2026). A plausible implication is that the selected classifier can differ materially from both pure spectral-margin and pure $W_k$5-margin solutions when the hidden-layer matrix norm and final-layer vector norm compete.

4. Implicit-bias theorem

The main implicit-bias theorem for Muon-Signum in (Gronich et al., 18 Feb 2026) assumes:

• $W_k$6 is $W_k$7-smooth and $W_k$8-homogeneous,
• $W_k$9 is non-increasing with $u$0 and $u$1,
• the trajectory remains bounded away from the origin and converges in direction to some $u$2 with positive margin.

Under these assumptions, Muon-Signum satisfies

$u$3

and $u$4 is the direction of a KKT point of the max-margin program defined by the hybrid norm (Gronich et al., 18 Feb 2026). The paper states this as Corollary “Muon-Signum.”

The theorem extends earlier lines of work in two directions identified in the paper’s abstract. First, it extends steepest-descent implicit-bias results from homogeneous models to normalized steepest descent with an optional learning-rate schedule. Second, it shows that momentum steepest-descent algorithms such as Muon, MomentumGD, and Signum are approximate steepest-descent trajectories under a decaying learning-rate schedule, then extends this analysis to Adam, Muon-Signum, and Muon-Adam (Gronich et al., 18 Feb 2026).

The specific placement of Muon-Signum relative to Adam is technically important. The same paper states that Adam, without the stability constant, maximizes the $u$5 margin, whereas Muon-Signum and Muon-Adam maximize a hybrid norm (Gronich et al., 18 Feb 2026). This indicates that the hybrid-norm phenomenon is not unique to one optimizer construction; it arises whenever different parameter blocks are assigned distinct geometries.

5. Relation to Muon, Signum, Adam, and related hybrids

The comparison in (Gronich et al., 18 Feb 2026) is explicit. Muon alone is normalized momentum-steepest descent in the spectral norm of $u$6 and $u$7 for vector blocks, so it converges to the spectral-norm margin-maximizer. Signum alone is normalized momentum-steepest descent in the $u$8 norm of $u$9, hence it maximizes the $\eta(t)$0 hard margin. Muon-Signum instead uses

$\eta(t)$1

so its implicit solution is the one maximizing the hybrid bottleneck margin (Gronich et al., 18 Feb 2026).

The same source reports the practical comparison as follows: Signum $\eta(t)$2 Adam for the $\eta(t)$3 margin, Muon $\eta(t)$4 NGD for spectral margin, and Muon-Signum lies in between, matching whichever block dominates (Gronich et al., 18 Feb 2026). This is not a statement of interpolation by averaging; rather, it is interpolation induced by a max norm across heterogeneous parameter blocks.

Additional literature sharpens the algorithmic relationship between the components. "Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory" (Eschenhagen et al., 10 Feb 2026) presents a unified ladder Element-wise sign-descent $\eta(t)$5 Signum $\eta(t)$6 Adam and Matrix spectral-descent $\eta(t)$7 Muon $\eta(t)$8 Shampoo, with the latter optimizers in each chain adding adaptation beyond pure normalized descent. That paper also states that Muon consistently outperforms Signum on language-model perplexity under its reported settings, while Shampoo improves further via left/right adaptation on weight matrices (Eschenhagen et al., 10 Feb 2026). These results concern data efficiency rather than implicit bias, but they contextualize why a hybrid such as Muon-Signum is of interest: it combines a strong matrix direction with a cheap sign-based vector update.

A separate line of work, "LionMuon: Alternating Spectral and Sign Descent for Efficient Training" (Bolatov et al., 19 May 2026), studies alternation between Muon and sign-based steps rather than the blockwise hybrid construction of Muon-Signum. There, Muon and Signum are again treated as norm-specific linear minimization oracle updates, and the analysis proves interpolation between spectral and $\eta(t)$9 regimes under heavy-tailed noise (Bolatov et al., 19 May 2026). This suggests a broader pattern: several recent optimizers mix spectral and sign geometries, but Muon-Signum is distinctive in (Gronich et al., 18 Feb 2026) because its hybrid norm arises from simultaneous blockwise normalized momentum-steepest-descent and yields a direct max-margin interpretation.

6. Experimental evidence and observed behavior

The experiments reported for Muon-Signum in (Gronich et al., 18 Feb 2026) use two-layer homogeneous networks with a hidden layer and scalar output, trained on MNIST even/odd with exponential loss and a decaying schedule $m_t^{(W_k)}$0. The study considers both ReLU and smooth squared-ReLU activations.

The paper reports the following pattern in Figure 2a:

Optimizer	Reported largest margin
NGD ($m_t^{(W_k)}$1)	largest $m_t^{(W_k)}$2 margin
Signum/Adam	largest $m_t^{(W_k)}$3 margin
Muon	largest spectral margin
Muon-Signum	largest hybrid margin $m_t^{(W_k)}$4

The experiments also plot the cosine alignment of $m_t^{(W_k)}$5 with $m_t^{(W_k)}$6, which confirms directional convergence in Figure 2b (Gronich et al., 18 Feb 2026). The reported empirical outcome is therefore closely aligned with the theoretical picture: Muon-Signum converges to the max-margin solution associated with the hybrid norm and realizes the corresponding margin values in practice.

The validation strategy is notable because it isolates the optimizer-dependent geometry in homogeneous networks, where implicit-bias results are most transparent. This differs from large-scale language-model studies of Muon and Signum, which focus on token efficiency, perplexity, or compute tradeoffs. For example, (Eschenhagen et al., 10 Feb 2026) reports final validation perplexities for Muon and Signum on C4 with LLaMA 3 architectures, and (Bolatov et al., 19 May 2026) studies validation loss versus FLOPs for alternating Muon-sign methods. Those works demonstrate practical performance differences between spectral and sign updates, but the Muon-Signum experiments in (Gronich et al., 18 Feb 2026) are designed specifically to validate margin predictions rather than downstream scaling behavior.

A plausible implication is that Muon-Signum’s most direct theoretical relevance lies in settings where parameter homogeneity and separability make margin-based implicit bias observable, while its practical relevance in broader stochastic training may depend on how strongly the matrix and vector blocks differ in their effective geometry.

7. Interpretation, scope, and limitations

Within the framework of (Gronich et al., 18 Feb 2026), Muon-Signum should be understood as the composite normalized-momentum-steepest-descent algorithm associated with the max of spectral and $m_t^{(W_k)}$7 norms. Its defining property is not simply that it mixes Muon and Signum updates, but that this mixture corresponds to a single hybrid norm and therefore to a single hybrid max-margin problem.

Several misconceptions are thereby excluded. Muon-Signum is not presented as optimizing a weighted sum of spectral and $m_t^{(W_k)}$8 margins; the relevant norm is the maximum of the two block norms (Gronich et al., 18 Feb 2026). Nor is its limit characterized as an arbitrary compromise between Muon and Signum. The theory states that the direction converges to that of a KKT point of a precisely defined constrained problem, and the optimizer “sits in between” only because the max norm selects whichever block is limiting (Gronich et al., 18 Feb 2026).

The formal guarantees are conditional. The theorem requires $m_t^{(W_k)}$9 smoothness, $m_t^{(u)}$0-homogeneity, a non-increasing learning rate with $m_t^{(u)}$1 and $m_t^{(u)}$2, and assumptions that the trajectory stays bounded away from the origin and converges in direction to a positive-margin limit (Gronich et al., 18 Feb 2026). The results therefore do not by themselves establish identical behavior for arbitrary architectures, losses, or finite-step stochastic training procedures.

Recent adjacent work indicates broader relevance of the underlying geometric split. Non-Euclidean batch-size adaptation for Signum and Muon uses optimizer-aligned noise scales derived from the corresponding dual norms, with reported reductions in training steps of up to 66% on a 160 million parameter Llama model (Naganuma et al., 3 Feb 2026). Alternating spectral and sign descent has also been shown to interpolate between the two regimes in both theory and empirical compute efficiency (Bolatov et al., 19 May 2026). These results do not analyze Muon-Signum itself, but they support the view that spectral and sign geometries form a coherent design space for modern optimizers.

In summary, Muon-Signum is a hybrid optimizer whose matrix blocks follow Muon’s spectral descent and whose final vector block follows Signum’s coordinate-wise sign descent, all under a common momentum-steepest-descent interpretation. For smooth homogeneous neural networks, its implicit bias is toward KKT points of the max-margin problem induced by the norm $m_t^{(u)}$3, and experiments reported in (Gronich et al., 18 Feb 2026) show that it realizes the corresponding hybrid margin behavior in practice.