---
title: 'NAMO: Norm-Based Adaptive Momentum Optimizer'
url: https://www.emergentmind.com/topics/namo-norm-based-adaptive-moment-estimation-with-orthogonalized-momentum
type: topic
---

# NAMO: Norm-Based Adaptive Momentum Optimizer

Norm-Based Adaptive Moment Estimation with Orthogonalized Momentum (NAMO) is a class of first-order stochastic optimization algorithms for training high-dimensional models, especially those with matrix-structured parameters such as deep neural networks. NAMO combines orthogonalized momentum updates, originally featured in the Muon optimizer, with norm-based adaptive scaling mechanisms akin to those in AdaGrad and Adam. This unification preserves the implicit regularization and stability benefits of orthogonalized steps while equipping the update magnitude with adaptive control suited for non-stationary, noisy optimization landscapes. Several variants exist, notably the scalar-scaled NAMO, its diagonal extension NAMO-D, and generalized forms for large-scale or memory-efficient training. NAMO and related algorithms provide rigorous convergence guarantees and have demonstrated empirical superiority over classic optimizers in training deep architectures including GPT-2 and LLaMA [2509.02981] [2411.07120] [2602.17080].

## 1. Algorithmic Foundations

NAMO’s core mechanism is the combination of two distinct components:  
**(i) Orthogonalized momentum:** The optimizer maintains an exponential moving average of gradients $M_t$ and orthogonalizes it via a matrix factorization (typically polar/SVD) to obtain an update direction $O_t$ with $O_t O_t^\top = I$. This direction realizes steepest descent under the spectral norm, providing robust updates in the presence of ill-conditioned objectives.  
**(ii) Norm-based adaptive scaling:** The update step size is adaptively modulated. Original formulations use either AdaGrad-style accumulators (scalar or subset-based) or second-moment estimates as in Adam, ensuring the step size contracts in flat or noisy regimes and increases in sharp, informative landscapes.

For parameter matrices $\Theta_t\in\mathbb{R}^{m\times n}$, the prototypical NAMO update sequence is:
\[
\begin{aligned}
  g_t &= \nabla\mathcal{L}_t(\Theta_{t-1}), \\
  M_t &= \mu_1 M_{t-1} + (1-\mu_1)g_t, \\
  v_t &= \mu_2 v_{t-1} + (1-\mu_2)\|g_t\|_F^2, \\
  O_t &= \text{Orth}(M_t), \\
  \alpha_t &= \|\hat M_t\|_F/(\sqrt{\hat v_t} + \epsilon), \\
  \Theta_t &= \Theta_{t-1} - \eta\alpha_t O_t.
\end{aligned}
\]
Here, bias correction applies to moving averages, and $\epsilon > 0$ prevents division by zero. The diagonal extension (NAMO-D) further enables neuron-wise adaptation via per-column scaling matrices, with clamping to maintain well-conditioned updates [2602.17080].

## 2. Architectural Variants and Memory-Efficient Extensions

Several NAMO variants address different scaling and memory constraints:

- **Scalar NAMO:** Uses a global norm-based step size; requires only a single additional scalar beyond the Muon baseline [2509.02981] [2602.17080].
- **NAMO-D:** Augments scalar scaling with a right multiplication by a diagonal matrix $D_t$, whose entries are adaptively clipped per neuron or unit. This improves alignment with block-diagonal Hessian structures and enhances stability in highly overparameterized or structured tasks [2602.17080].
- **Subset-Norm and Subspace-Momentum NAMO:** Parameters are partitioned into $c$ subsets, each with its dedicated accumulator, or momentum is restricted to a low-dimensional subspace. This reduces memory from $O(d)$ to $O(\sqrt{d} + k)$, essential for billion-parameter models [2411.07120].

The following table summarizes state complexity for different variants:

| Variant         | State Memory Complexity | Step Size Adaptivity         |
|-----------------|------------------------|-----------------------------|
| Muon            | $O(mn)$                | Fixed                       |
| Scalar NAMO     | $O(mn) + 1$            | Norm-based scalar           |
| NAMO-D          | $O(mn) + n$            | Per-column adaptive, clamped|
| SN/SM NAMO      | $O(\sqrt{d} + k)$      | Subset-wise/adaptive        |

## 3. Theoretical Guarantees

NAMO achieves optimal convergence rates under both deterministic and stochastic settings, contingent on Lipschitz-spectral smoothness and bounded-variance noise:

- **Deterministic (full-batch):**
  \[
  \frac{1}{T} \sum_{t=1}^T \|\nabla L(\Theta_{t-1})\|_F = O(T^{-1/2})
  \]
- **Stochastic (mini-batch):**
  \[
  \frac{1}{T} \sum_{t=1}^T \mathbb{E}\|\nabla L(\Theta_{t-1})\|_F = O(T^{-1/4} + \sigma b^{-1/4} T^{-1/8})
  \]
The rates interpolate to $O(T^{-1/2})$ with increasing batch size or vanishing noise. These results match established lower bounds for first-order methods in both convex and nonconvex regimes [2509.02981] [2411.07120] [2602.17080].

In the case of memory-efficient NAMO combining subset-norm with subspace-momentum, high-probability guarantees hold for nonconvex objectives under sub-Gaussian noise, with only slightly inflated constants relative to vanilla AdaGrad or SGD [2411.07120].

## 4. Empirical Performance and Practical Implementation

Experimental results highlight NAMO’s empirical advantages:

- **Small- to medium-scale tasks:** On function regression and CIFAR-10, NAMO consistently outperforms Adam and Muon on both convergence speed and final test error. In particular, it achieves 2–3% higher test accuracy on CIFAR-10 than Muon/Adam and shows faster test loss decay in synthetic regression [2509.02981].
- **Large-scale language models:** In pretraining GPT-2 (124M and 355M) on OpenWebText, both NAMO and its diagonal variant achieve lower training and validation loss than AdamW and Muon, with NAMO-D benefiting from per-neuron scaling (final 124M validation losses: AdamW 3.0643, NAMO 3.0351, NAMO-D 3.0246) [2602.17080]. On LLaMA 1B, memory-optimized SN/SM NAMO attains Adam-level perplexity in half as many tokens while requiring 80% less optimizer state memory (0.84 GB vs. 5 GB for Adam) [2411.07120].

NAMO has negligible memory or compute overhead compared to baseline Muon and significantly reduced cost compared to Adam in large models. Orthogonalization of the momentum matrix is typically handled via Newton–Schulz iterations or SVD; for SN/SM NAMO, subspace dimension $k$ and subset size $c$ are tunable for further trade-offs.

## 5. Orthogonality, Regularization, and Algorithmic Significance

NAMO’s preservation of orthogonality in update directions is founded on its interpretation as a spectral-norm steepest descent, with
\[
O_t = \arg\min_{Z} \langle Z, M_t \rangle + \tfrac{1}{2} \|Z\|_2^2,
\]
where $\langle\cdot,\cdot\rangle$ is the matrix inner product. This spectral regularization both stabilizes training under highly ill-conditioned or degenerate settings and mitigates pathological behaviors of classic adaptive optimizers. In particular, NAMO avoids the sign-bias and step-size explosion/collapse issues known in Adam and RMSProp, ensuring null-gradient consistency (vanishing update magnitude at stationary points) [2509.02981] [2602.17080].

The diagonal extension (NAMO-D) aligns with empirical Hessian block-diagonality, enabling fine-grained adaptation without excessive condition number inflation due to clamping parameter $c$.

## 6. Hyperparameter Selection and Practical Guidance

Typical hyperparameters mirror those of Adam and Muon, with momentum factors $\mu_1\approx0.95$, $\mu_2\approx0.99$, learning rate $\eta$ selected by short grid search, and stepsize floor $\epsilon=10^{-6}$. For NAMO-D, clamping $c\in[0.1,1.0]$ trades adaptivity for conditioning. Subset-norm NAMO benefits from grouping parameters by rows or columns to achieve $c\sim\sqrt{d}$ for optimal memory savings.

Implementation requires minimal change from Muon or Adam pipelines: a single line replaces fixed scalar with adaptive norm-based scaling, and for subset/subspace versions, standard linear algebra operations suffice [2509.02981] [2411.07120] [2602.17080].

## 7. Applications, Limitations, and Prospective Directions

NAMO and its variants are effective in training both small and large neural network models, with improved stability, faster convergence, and reduced memory usage relative to legacy optimizers. Key use cases include language model pretraining (e.g., GPT-2, LLaMA) and scenarios where matrix-structured updates and memory constraints are critical. The mechanistic correspondence with the spectral structure of neural network Hessians and block-wise adaptivity aligns NAMO-D with model architectures exhibiting pronounced modularity or locality.

*This suggests* continued relevance of NAMO-like methods as model scales and architectural heterogeneity increase. A plausible implication is that further diagonal or block-diagonal extensions, possibly integrating dynamic structure-aware clamping, may enhance stability and sample efficiency in forthcoming large-scale training regimes.

**References:**
- [2509.02981] AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates  
- [2602.17080] Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum  
- [2411.07120] Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees

Source: https://www.emergentmind.com/topics/namo-norm-based-adaptive-moment-estimation-with-orthogonalized-momentum