Muon-Adam: Hybrid Spectral-Adaptive Optimizer

• Muon-Adam is a hybrid optimization method that fuses spectral updates for matrices with Adam-style adaptive updates for vectors.
• It combines exact matrix orthogonalization with adaptive moment estimation to induce an implicit bias toward max-margin solutions.
• Practical performance relies on careful tuning of matrix and vector learning rates to balance spectral descent and adaptive coordinate updates.

Muon-Adam is a hybrid optimization scheme that combines Muon-style matrix orthogonalization with Adam-style adaptive coordinate updates. In the formulation analyzed for smooth homogeneous neural networks, parameters are partitioned as $\theta=(W_1,\dots,W_K,u)$, with Muon applied to matrix blocks $W_k$ and Adam applied to the remaining vector block $u$; the resulting dynamics are shown to induce an implicit bias toward Karush–Kuhn–Tucker (KKT) points of a hybrid max-margin problem (Gronich et al., 18 Feb 2026). A complementary non-Euclidean optimization perspective formulates the practical algorithm as constrained steepest descent in a mixed product norm, placing Muon-Adam within a broader family of spectral and adaptive methods (Crawshaw et al., 10 Oct 2025).

1. Definition and update structure

In the homogeneous-network analysis, Muon-Adam is studied on binary classification data $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$ with a smooth $L$-homogeneous model $f(x;\theta)$, meaning $f(x;\alpha\theta)=\alpha^L f(x;\theta)$. The full-batch loss is

$$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$

with $\ell(u)=e^{-u}$ or $\ell(u)=\log(1+e^{-u})$ (Gronich et al., 18 Feb 2026).

The parameter vector is partitioned as

$W_k$0

where each $W_k$1 is a matrix and $W_k$2 collects the remaining vector parameters. Muon-Adam then applies distinct update mechanisms to these two components. For the matrix blocks, it uses Muon with exact SVD orthogonalization; for the vector block, it uses Adam-style first- and second-moment adaptation. In continuous time, the matrix and vector dynamics are

$W_k$3

and

$W_k$4

where $W_k$5 is the subgradient block for $W_k$6, $W_k$7 is the subgradient for $W_k$8, and $W_k$9 denotes exact orthogonalization (Gronich et al., 18 Feb 2026).

A discrete practical formulation uses the same division of labor. Matrix layers $u$0 are updated by

$u$1

while non-matrix parameters $u$2 use Adam buffers and the standard adaptive step

$u$3

This formulation identifies Muon-Adam as a side-by-side composition of spectral descent for matrix blocks and Adam for the remaining parameters (Crawshaw et al., 10 Oct 2025).

2. Model assumptions and admissible trajectories

The implicit-bias theory for Muon-Adam is developed under a specific structural regime. The network must satisfy the smooth homogeneous assumptions $u$4 and $u$5 for some $u$6. The loss $u$7 must be strictly decreasing, twice-differentiable, and exponentially tailed, with logistic and exponential losses given as examples (Gronich et al., 18 Feb 2026).

The learning-rate schedule is also restricted. The base schedule $u$8 must satisfy

$u$9

and it must be non-increasing. These conditions ensure persistent motion while enforcing sufficient decay for the asymptotic steepest-descent approximation (Gronich et al., 18 Feb 2026).

Three trajectory assumptions are then imposed. First, the trajectory must be nontrivial, in the sense that eventually $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$0. Second, the direction must converge: $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$1 with positive margin $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$2. Third, Adam must be well-defined: on a short interval $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$3, all gradient squares exceed $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$4, ensuring $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$5 always (Gronich et al., 18 Feb 2026).

These are not generic optimizer assumptions; they define the scope in which the max-margin result is proved. A plausible implication is that Muon-Adam’s asymptotic characterization is most transparent in homogeneous models where directional convergence can be justified or observed empirically.

3. Hybrid norm geometry and steepest-descent interpretation

The central geometric object is a hybrid norm under which Muon-Adam behaves as an Approximate Steepest Descent (ASD) trajectory. In the homogeneous-model analysis, the relevant norm is

$\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$6

Its dual norm is

$\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$7

Thus the matrix part is governed by spectral/nuclear duality, while the vector part is governed by $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$8 duality (Gronich et al., 18 Feb 2026).

The ASD proof checks three conditions: the integrated step length diverges, the normalized step aligns asymptotically with the negative subgradient, and the size ratio satisfies $\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}$9. On the $L$0-blocks, exact orthogonalization yields normalized momentum steepest descent for the spectral norm. On the $L$1-block, the Adam ratio $L$2 approximates the sign of $L$3. The mixed dual norm selects the dominant nuclear-versus-$L$4 contribution, and asymptotic alignment is recovered by partitioning time according to which component dominates (Gronich et al., 18 Feb 2026).

A related but distinct formulation casts practical Muon-Adam as constrained steepest descent in the infinity-product norm

$L$5

In that view, the linear minimization oracle on each matrix block is $L$6, and the $L$7-block uses the Adam-style adaptive direction (Crawshaw et al., 10 Oct 2025). Taken together, these formulations place Muon-Adam at the intersection of spectral steepest descent and adaptive coordinate descent.

4. Max-margin bias and KKT characterization

The hard margin of a parameter vector is defined by

$L$8

The associated max-margin problem under a norm $L$9 is

$f(x;\theta)$0

A point $f(x;\theta)$1 with multipliers $f(x;\theta)$2, subgradients $f(x;\theta)$3, and $f(x;\theta)$4 satisfies the KKT conditions if

$f(x;\theta)$5

These conditions are the stationarity object to which Muon-Adam converges directionally (Gronich et al., 18 Feb 2026).

Once the ASD conditions are established and $f(x;\theta)$6, the general KKT blueprint implies that any directional limit $f(x;\theta)$7 is a KKT point of the hybrid-norm max-margin problem. The resulting margin is

$f(x;\theta)$8

with the hybrid norm combining matrix spectral norms and the vector $f(x;\theta)$9 norm (Gronich et al., 18 Feb 2026).

The main convergence statements make this explicit. Theorem 4.2 states that any ASD trajectory with asymptotic alignment and diverging norm converges directionally to a KKT point of the max-margin problem under the corresponding norm. Theorem 4.4 identifies Adam’s limit with a KKT point for $f(x;\alpha\theta)=\alpha^L f(x;\theta)$0. Theorem 4.6 states that Muon-Adam biases the hybrid max-spectral/$f(x;\alpha\theta)=\alpha^L f(x;\theta)$1 margin

$f(x;\alpha\theta)=\alpha^L f(x;\theta)$2

under the same homogeneity, decay, and directional-convergence assumptions (Gronich et al., 18 Feb 2026).

5. Relation to Adam, Muon-Signum, MomentumGD, and spectral families

Muon-Adam is one member of a family of momentum-based non-Euclidean optimizers whose implicit biases are determined by the norm geometry of their effective steepest-descent trajectories. Standard Adam, without the stability constant $f(x;\alpha\theta)=\alpha^L f(x;\theta)$3, is analyzed as maximizing the $f(x;\alpha\theta)=\alpha^L f(x;\theta)$4-margin. MomentumGD approximates normalized $f(x;\alpha\theta)=\alpha^L f(x;\theta)$5-steepest descent and biases the $f(x;\alpha\theta)=\alpha^L f(x;\theta)$6-margin. Muon-Signum, which applies Muon to matrices and Signum to vector parameters, biases the hybrid max-margin based on spectral norm for matrices and $f(x;\alpha\theta)=\alpha^L f(x;\theta)$7 for vectors. Muon-Adam “sits strictly between Muon-Signum (pure sign) and pure Adam,” because tuning $f(x;\alpha\theta)=\alpha^L f(x;\theta)$8 and $f(x;\alpha\theta)=\alpha^L f(x;\theta)$9 trades off spectral versus $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$0 influence in the hybrid norm (Gronich et al., 18 Feb 2026).

The non-Euclidean steepest-descent perspective sharpens this comparison. Muon-Adam is identified with constrained steepest descent, whereas MuonMax is derived from a regularized variant using an $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$1-style product norm. In that framework, Muon-Adam normalizes each matrix block individually by its polar factor and retains Adam on the remaining parameters, while MuonMax rescales matrix steps through a dual norm involving sums of nuclear norms (Crawshaw et al., 10 Oct 2025).

A broader unified spectral perspective places Muon and Adam at the endpoints of the spectral transform family

$$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$2

with $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$3 corresponding to full orthogonalization and $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$4 corresponding to the identity transform. In the RMS-normalized case, $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$5 is exactly an Adam-style update, while $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$6 yields an AdamZ update. This suggests that Muon-Adam can be viewed as one concrete hybridization of two geometries that also admit interpolation at the level of singular-value compression (Qi et al., 4 Feb 2026).

6. Empirical behavior and practical profile

In the homogeneous-network experiments accompanying the implicit-bias analysis, Muon-Adam was evaluated on two-layer homogeneous nets trained on MNIST. The reported behavior follows the theoretical geometry: NGD and MomentumGD cause the $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$7-margin to rise fastest; Signum and Adam maximize the $$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$8-margin; Muon maximizes the spectral-norm margin; and Muon-Adam yields the largest value of the hybrid margin

$$L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),$$9

The same experiments report cosine alignment of $\ell(u)=e^{-u}$0 to $\ell(u)=e^{-u}$1 above .99 in late training, supporting the directional-convergence assumption used in the theory (Gronich et al., 18 Feb 2026).

In language-model experiments from the non-Euclidean gradient study, Muon-Adam achieved mean validation loss $\ell(u)=e^{-u}$2 on FineWeb1B, while Muon-Adam + Momo achieved $\ell(u)=e^{-u}$3. The same study reports that plain Muon-Adam “requires precise tuning,” with factor-of-3 shifts in $\ell(u)=e^{-u}$4 or $\ell(u)=e^{-u}$5 capable of increasing validation loss by more than $\ell(u)=e^{-u}$6; adding Momo broadens the viable learning-rate region, and MuonMax-Momo is reported as the most robust variant on new tasks (Crawshaw et al., 10 Oct 2025).

These observations delimit Muon-Adam’s empirical profile. On the one hand, the hybrid optimizer realizes the norm-dependent margin behavior predicted by the ASD/KKT theory. On the other hand, its practical performance depends materially on the coupling between the matrix learning rate and the base Adam learning rate. This suggests that Muon-Adam is best understood not as a generic replacement for either parent optimizer, but as a geometrically structured hybrid whose behavior is controlled by the relative weight assigned to spectral and adaptive components.