---
title: 'Muon-Adam: Hybrid Spectral-Adaptive Optimizer'
url: https://www.emergentmind.com/topics/muon-adam
type: topic
---

# Muon-Adam: Hybrid Spectral-Adaptive Optimizer

Muon-Adam is a hybrid optimization scheme that combines Muon-style matrix orthogonalization with Adam-style adaptive coordinate updates. In the formulation analyzed for smooth homogeneous neural networks, parameters are partitioned as \(\theta=(W_1,\dots,W_K,u)\), with Muon applied to matrix blocks \(W_k\) and Adam applied to the remaining vector block \(u\); the resulting dynamics are shown to induce an implicit bias toward Karush–Kuhn–Tucker (KKT) points of a hybrid max-margin problem [2602.16340]. A complementary non-Euclidean optimization perspective formulates the practical algorithm as constrained steepest descent in a mixed product norm, placing Muon-Adam within a broader family of spectral and adaptive methods [2510.09827].

## 1. Definition and update structure

In the homogeneous-network analysis, Muon-Adam is studied on binary classification data \(\{(x_i,y_i)\}\subset\mathbb R^d\times\{\pm1\}\) with a smooth \(L\)-homogeneous model \(f(x;\theta)\), meaning \(f(x;\alpha\theta)=\alpha^L f(x;\theta)\). The full-batch loss is
\[
L(\theta)=\sum_{i=1}^m \ell\bigl(y_i f(x_i;\theta)\bigr),
\]
with \(\ell(u)=e^{-u}\) or \(\ell(u)=\log(1+e^{-u})\) [2602.16340].

The parameter vector is partitioned as
\[
\theta=(W_1,\dots,W_K,u),
\]
where each \(W_k\) is a matrix and \(u\) collects the remaining vector parameters. Muon-Adam then applies distinct update mechanisms to these two components. For the matrix blocks, it uses Muon with exact SVD orthogonalization; for the vector block, it uses Adam-style first- and second-moment adaptation. In continuous time, the matrix and vector dynamics are
\[
\dot W_k=-\,\eta_0^M\,\eta(t)\,U_k(t)V_k(t)^\top,
\qquad
M_k(t)=\mathrm{EMA}(G_k(t),c_1),
\]
and
\[
\dot u=-\,\eta_0^A\,\eta(t)\,\frac{\hat\mu_t}{\sqrt{\hat\omega_t}},
\qquad
\mu_t=\mathrm{EMA}(g_t^{(u)},c_1),
\qquad
\omega_t=\mathrm{EMA}\bigl((g_t^{(u)})^2,c_2\bigr),
\]
where \(G_k(t)\) is the subgradient block for \(W_k\), \(g_t^{(u)}\) is the subgradient for \(u\), and \(M_k(t)=U_k\Sigma_kV_k^\top\mapsto U_kV_k^\top\) denotes exact orthogonalization [2602.16340].

A discrete practical formulation uses the same division of labor. Matrix layers \(W^\ell\) are updated by
\[
W_{t+1}^\ell = W_t^\ell-\eta_m\,\mathrm{polar}(M_t^\ell),
\]
while non-matrix parameters \(\vartheta\) use Adam buffers and the standard adaptive step
\[
\vartheta_{t+1}=\vartheta_t-\eta_b\,\frac{m_t}{\sqrt{v_t}+\epsilon}.
\]
This formulation identifies Muon-Adam as a side-by-side composition of spectral descent for matrix blocks and Adam for the remaining parameters [2510.09827].

## 2. Model assumptions and admissible trajectories

The implicit-bias theory for Muon-Adam is developed under a specific structural regime. The network must satisfy the smooth homogeneous assumptions \(f(\cdot;\theta)\in C^1\) and \(f(x;\alpha\theta)=\alpha^L f(x;\theta)\) for some \(L\ge 1\). The loss \(\ell\) must be strictly decreasing, twice-differentiable, and exponentially tailed, with logistic and exponential losses given as examples [2602.16340].

The learning-rate schedule is also restricted. The base schedule \(\eta(t)\) must satisfy
\[
\eta(t)\ge 0,\qquad \int_0^\infty \eta(t)\,dt=\infty,\qquad \eta(t)=o\bigl(t^{\frac1L-1}\bigr),
\]
and it must be non-increasing. These conditions ensure persistent motion while enforcing sufficient decay for the asymptotic steepest-descent approximation [2602.16340].

Three trajectory assumptions are then imposed. First, the trajectory must be nontrivial, in the sense that eventually \(\|\theta_t\|\ge N_{\min}>0\). Second, the direction must converge:
\[
\frac{\theta_t}{\|\theta_t\|}\to \bar\theta
\]
with positive margin \(\gamma(\bar\theta)>0\). Third, Adam must be well-defined: on a short interval \([0,\tau]\), all gradient squares exceed \(\rho>0\), ensuring \(v_t>0\) always [2602.16340].

These are not generic optimizer assumptions; they define the scope in which the max-margin result is proved. A plausible implication is that Muon-Adam’s asymptotic characterization is most transparent in homogeneous models where directional convergence can be justified or observed empirically.

## 3. Hybrid norm geometry and steepest-descent interpretation

The central geometric object is a hybrid norm under which Muon-Adam behaves as an Approximate Steepest Descent (ASD) trajectory. In the homogeneous-model analysis, the relevant norm is
\[
\|\theta\|=\max\Bigl\{\alpha\,\max_k \|W_k\|_{\mathit{sp}},\,\|u\|_\infty\Bigr\},
\qquad
\alpha=\frac{\eta_0^A}{\eta_0^M}.
\]
Its dual norm is
\[
\|g\|_\star
=
\sum_k \frac1\alpha \|\nabla_{W_k}L\|_{\mathit{nuc}}
+\|\nabla_uL\|_1.
\]
Thus the matrix part is governed by spectral/nuclear duality, while the vector part is governed by \(\ell_\infty/\ell_1\) duality [2602.16340].

The ASD proof checks three conditions: the integrated step length diverges, the normalized step aligns asymptotically with the negative subgradient, and the size ratio satisfies \(\|\theta_t\|/(\int_0^t \nu)\le 1\). On the \(W\)-blocks, exact orthogonalization yields normalized momentum steepest descent for the spectral norm. On the \(u\)-block, the Adam ratio \(\hat m_t/\sqrt{\hat v_t}\) approximates the sign of \(g_t^{(u)}\). The mixed dual norm selects the dominant nuclear-versus-\(\ell_1\) contribution, and asymptotic alignment is recovered by partitioning time according to which component dominates [2602.16340].

A related but distinct formulation casts practical Muon-Adam as constrained steepest descent in the infinity-product norm
\[
\|(W^1,\dots,W^L,\vartheta)\|
=
\max\Bigl\{
\max_\ell \|W^\ell\|_{2\to 2},
\frac{\eta_m}{\eta_b}\|\vartheta\|_{\mathrm{ada}\infty}
\Bigr\}.
\]
In that view, the linear minimization oracle on each matrix block is \(-\mathrm{polar}(M_t^\ell)\), and the \(\vartheta\)-block uses the Adam-style adaptive direction [2510.09827]. Taken together, these formulations place Muon-Adam at the intersection of spectral steepest descent and adaptive coordinate descent.

## 4. Max-margin bias and KKT characterization

The hard margin of a parameter vector is defined by
\[
\gamma(\theta):=\min_{i\in[m]} y_i f(x_i;\theta).
\]
The associated max-margin problem under a norm \(\|\cdot\|\) is
\[
\min_\theta \frac12\|\theta\|^2
\qquad
\text{s.t. } y_i f(x_i;\theta)\ge 1 \quad (i=1,\dots,m).
\]
A point \(\theta^\star\) with multipliers \(\{\lambda_i\}\), subgradients \(h_i\in\partial f(x_i;\theta^\star)\), and \(k\in\partial \frac12\|\theta^\star\|^2\) satisfies the KKT conditions if
\[
\sum_i \lambda_i y_i h_i = k,\qquad
\sum_i \lambda_i\bigl(y_i f(x_i;\theta^\star)-1\bigr)=0,\qquad
\lambda_i\ge 0.
\]
These conditions are the stationarity object to which Muon-Adam converges directionally [2602.16340].

Once the ASD conditions are established and \(\|\theta_t\|\to\infty\), the general KKT blueprint implies that any directional limit \(\bar\theta\) is a KKT point of the hybrid-norm max-margin problem. The resulting margin is
\[
\gamma_{\max}
=
\max_{\|\theta\|\le 1}\min_i y_i f(x_i;\theta),
\]
with the hybrid norm combining matrix spectral norms and the vector \(\ell_\infty\) norm [2602.16340].

The main convergence statements make this explicit. Theorem 4.2 states that any ASD trajectory with asymptotic alignment and diverging norm converges directionally to a KKT point of the max-margin problem under the corresponding norm. Theorem 4.4 identifies Adam’s limit with a KKT point for \(\|\cdot\|_\infty\). Theorem 4.6 states that Muon-Adam biases the hybrid max-spectral/\(\infty\) margin
\[
\|\theta\|=\max\Bigl\{\alpha\,\max_k \|W_k\|_{\mathit{sp}},\,\|u\|_\infty\Bigr\}
\]
under the same homogeneity, decay, and directional-convergence assumptions [2602.16340].

## 5. Relation to Adam, Muon-Signum, MomentumGD, and spectral families

Muon-Adam is one member of a family of momentum-based non-Euclidean optimizers whose implicit biases are determined by the norm geometry of their effective steepest-descent trajectories. Standard Adam, without the stability constant \(\varepsilon\), is analyzed as maximizing the \(\ell_\infty\)-margin. MomentumGD approximates normalized \(\ell_2\)-steepest descent and biases the \(\ell_2\)-margin. Muon-Signum, which applies Muon to matrices and Signum to vector parameters, biases the hybrid max-margin based on spectral norm for matrices and \(\ell_\infty\) for vectors. Muon-Adam “sits strictly between Muon-Signum (pure sign) and pure Adam,” because tuning \(\eta_0^M\) and \(\eta_0^A\) trades off spectral versus \(\infty\) influence in the hybrid norm [2602.16340].

The non-Euclidean steepest-descent perspective sharpens this comparison. Muon-Adam is identified with constrained steepest descent, whereas MuonMax is derived from a regularized variant using an \(\ell_2\)-style product norm. In that framework, Muon-Adam normalizes each matrix block individually by its polar factor and retains Adam on the remaining parameters, while MuonMax rescales matrix steps through a dual norm involving sums of nuclear norms [2510.09827].

A broader unified spectral perspective places Muon and Adam at the endpoints of the spectral transform family
\[
\Psi_p(O_t)=U\Sigma^pV^\top,
\qquad p\in[0,1],
\]
with \(p=0\) corresponding to full orthogonalization and \(p=1\) corresponding to the identity transform. In the RMS-normalized case, \(p=1\) is exactly an Adam-style update, while \(p=0\) yields an AdamZ update. This suggests that Muon-Adam can be viewed as one concrete hybridization of two geometries that also admit interpolation at the level of singular-value compression [2602.04669].

## 6. Empirical behavior and practical profile

In the homogeneous-network experiments accompanying the implicit-bias analysis, Muon-Adam was evaluated on two-layer homogeneous nets trained on MNIST. The reported behavior follows the theoretical geometry: NGD and MomentumGD cause the \(\ell_2\)-margin to rise fastest; Signum and Adam maximize the \(\ell_\infty\)-margin; Muon maximizes the spectral-norm margin; and Muon-Adam yields the largest value of the hybrid margin
\[
\max\Bigl\{\alpha\,\|W\|_{\mathit{sp}},\,\|u\|_\infty\Bigr\}.
\]
The same experiments report cosine alignment of \(\theta_t\) to \(\theta_{\rm last}\) above .99 in late training, supporting the directional-convergence assumption used in the theory [2602.16340].

In language-model experiments from the non-Euclidean gradient study, Muon-Adam achieved mean validation loss \(3.559\pm 0.0014\) on FineWeb1B, while Muon-Adam + Momo achieved \(3.555\pm 0.0004\). The same study reports that plain Muon-Adam “requires precise tuning,” with factor-of-3 shifts in \(\eta_m\) or \(\eta_b\) capable of increasing validation loss by more than \(0.1\); adding Momo broadens the viable learning-rate region, and MuonMax-Momo is reported as the most robust variant on new tasks [2510.09827].

These observations delimit Muon-Adam’s empirical profile. On the one hand, the hybrid optimizer realizes the norm-dependent margin behavior predicted by the ASD/KKT theory. On the other hand, its practical performance depends materially on the coupling between the matrix learning rate and the base Adam learning rate. This suggests that Muon-Adam is best understood not as a generic replacement for either parent optimizer, but as a geometrically structured hybrid whose behavior is controlled by the relative weight assigned to spectral and adaptive components.

Source: https://www.emergentmind.com/topics/muon-adam