---
title: 'Newton-Muon: Newton-Based Optimization'
url: https://www.emergentmind.com/topics/newton-muon
type: topic
---

# Newton-Muon: Newton-Based Optimization

Newton-Muon is an optimization algorithm that extends the Muon optimizer family by incorporating an explicit Newton-type local quadratic surrogate model in matrix-structured parameter spaces, yielding a step that fundamentally right-preconditions the update direction using input second-moment information. It operates by applying the matrix-sign functional to the product of the gradient with a right inverse of the data covariance, implemented efficiently via a SVD-free Newton–Schulz polynomial iteration. Newton-Muon can be interpreted as a specialization of the general polar-step Muon method with an explicit, data-adaptive right-preconditioner, and is empirically demonstrated to improve both optimization efficiency and training convergence on modern large-scale neural network tasks [2604.01472].

## 1. Theoretical Foundations and Motivating Surrogate

Newton-Muon originates from a local quadratic modeling of the empirical risk in matrix coordinates. Specifically, for a layerwise parameter matrix $W\in\mathbb R^{m\times n}$ and a loss $f(W)$, the minimization of a Taylor-expansion-based surrogate

\[
J(Q) = -\operatorname{tr}(QG^\top) + \frac{1}{2N}\operatorname{tr}(H Q(Z Z^\top) Q^\top)
\]

is conducted, where $G = \nabla_W f(W)$ (the batch gradient), $H\in\mathbb R^{m\times m}$ is an output-space curvature approximation (often treated as identity or isotropic), and $Z\in\mathbb R^{n\times N}$ concatenates the layer inputs [2604.01472]. Under a Kronecker-factored curvature approximation, the unique minimizer emerges as

\[
Q^* = \Sigma_W^{1/2} \, \mathrm{msgn}(\Sigma_W^{1/2} G (Z Z^\top)^{-1}),
\]

where $\Sigma_W$ encodes the left-side covariance of the parameter update, and $\mathrm{msgn}(X)$ denotes the orthogonally normalized matrix sign operator (mapping $X = U S V^\top \mapsto U V^\top$). In practical regimes, the isotropic proxy $\Sigma_W \propto I$ is adopted, reducing the update to

\[
Q \propto \mathrm{msgn}(G (Z Z^\top)^{-1}).
\]

This derivation demonstrates that Newton-Muon is a "right-preconditioned" polar-step optimizer, in contrast to standard Muon which assumes isotropic (identity) input covariance and omits the right-preconditioning [2604.01472].

## 2. Efficient Implementation: Newton–Schulz-Orthonormalization

Newton-Muon requires computation of $\mathrm{msgn}(G (Z Z^\top)^{-1})$. Direct SVD factorization is prohibitively expensive for large-scale training. Instead, Newton–Schulz polynomial iterations are used to approximate the necessary matrix functions without explicit decompositions [2509.24406, 2604.01472]. In general, the Newton–Schulz map for a symmetric positive definite matrix $A$ is of the form

\[
X_{k+1} = \frac{1}{2} X_k (3I - A X_k^2),
\]

and for more stable/rapid convergence, practitioners often use quintic polynomials, e.g.,

\[
X_{k+1} = a X_k + b X_k (X_k^\top X_k) + c X_k (X_k^\top X_k)^2
\]

with coefficients optimized to minimize the uniform polynomial approximation error over the spectral interval of $A$ (e.g., $a=3.4445$, $b=-4.7750$, $c=2.0315$) [2509.24406]. For Newton-Muon, this iteration is applied to the preconditioned gradient matrix, leveraging the fact that a handful ($3$–$5$) of such steps suffices to flatten the spectrum in practice.

## 3. Comparison to Muon and Spectral Normalization Perspective

Standard Muon applies the normalized-matrix-sign step $W \gets W - \eta\, \mathrm{msgn}(G)$, implicitly assuming isotropic input activation statistics. Newton-Muon, by contrast, explicitly incorporates the right-preconditioning $K=Z Z^\top$ (typically estimated as a running or blockwise moment of activations) and computes $W_{t+1} \gets W_t - \eta\,\mathrm{msgn}(G_t K_t^{-1})$ [2604.01472]. This aligns the update directions to the input data geometry, providing orthogonal equivariance under joint right-rotations $(G,Z) \mapsto (G O_n, O_n^\top Z)$, which standard Muon lacks if $Z Z^\top$ is far from the identity [2604.01472].

More generally, Newton-Muon can be regarded as a member of a parametric spectral-normalization family $\Psi_p(O) = U\Sigma^pV^\top$ ($p \in [0,1]$), with $p=0$ (Muon), $p=1$ (SGD/Adam), and fractional $p$ providing intermediate degrees of spectral compression [2602.04669]. Theoretical and empirical analyses reveal that full flattening ($p=0$) is not always optimal, but for raw first-moment inputs, it provides strong stabilization, while for RMS-normalized (Adam-type) updates, mild compression often suffices [2602.04669].

## 4. Convergence, Practical Tuning, and Theoretical Guarantees

Newton-Muon inherits nonconvex convergence guarantees from the Muon family when implemented with a finite number of Newton–Schulz steps. The convergence rate to a stationary point matches the SVD-polar-idealization up to a constant $\chi_q$ that converges doubly-exponentially to $1$ in the number of Newton–Schulz steps $q$ and polynomial degree $\kappa$ [2601.19156, 2510.19933]. In practical terms, $q=3$–$5$ and $\kappa=2$ yield a negligible overhead compared to the ideal case, with a wall-clock speedup of $4$–$10\times$ versus SVD-based orthogonalization.

Recent analysis further demonstrates that as right-preconditioning (through $K_t^{-1}$) becomes more accurate, the effective descent direction is better conditioned with respect to the layer's input geometry. Empirically, Newton-Muon reaches the same target validation loss as standard Muon in $6\%$ fewer iteration steps and $4\%$ less wall-clock time on GPT-2 pretraining setups, with only a $1.8\%$ per-step computational overhead from Cholesky inversion and Newton–Schulz polynomial evaluation [2604.01472].

However, the preconditioning matrix $K_t$ estimation carries its own tradeoffs, requiring periodic batchwise updates, stability regularization (e.g., ridge penalties), and blockwise or low-rank approximations to avoid excessive computational cost in very wide layers [2604.01472].

## 5. Practical Variants, Extensions, and Empirical Findings

Block-diagonal and low-rank variants of Newton-Muon have been shown to suffice in practice, particularly for multi-branch layers or large MLP contractions [2604.01472]. Efficient polynomial inversion routines (e.g., via specialized batched GEMM kernels or operator-specific kernels) are commonly employed. Newton-Muon has been deployed in hybrid schemes, e.g., applying AdamW on small parameters and Newton-Muon on hidden-layer weights, yielding improved convergence and generalization on tasks such as CIFAR-10 and large language modeling.

Notably, the empirical results indicate that the inclusion of right-preconditioning systematically improves both iteration efficiency and stability, particularly in the presence of pronounced input anisotropy, a regime common to realistic deep architectures [2604.01472]. Fixed or blockwise $K_t$ updates, as well as regularized Cholesky/symmetrized inverses, are effective in practice.

## 6. Limitations and Open Problems

A crucial limitation of current Newton-Muon implementations is the reliance on the isotropic-weight approximation for the left covariance $\Sigma_W\propto I$, as unbiased and practical alternatives remain an open research challenge [2604.01472]. The Kronecker-factored surrogate for the Hessian omits off-token coupling in transformers; more accurate yet efficient surrogates may further improve second-order adaptation. For extremely large-scale distributed scenarios, the estimation and communication of $K_t$ (activation second moments) can be a bottleneck, motivating work on randomized or structured sketches for distributed preconditioning [2604.01472].

Newton-Muon thus represents a principled, theoretically sound, and empirically effective extension of polar-step geometry-aware optimization. Its key distinguishing feature is explicit adaptation to data geometry via input-moment right-preconditioning, efficiently realized through SVD-free Newton–Schulz polynomial iteration and carefully regulated batchwise updates [2604.01472].

Source: https://www.emergentmind.com/topics/newton-muon