---
title: 'AuON: Alternative Unit-Norm Optimizer'
url: https://www.emergentmind.com/topics/auon
type: topic
---

# AuON: Alternative Unit-Norm Optimizer

AuON, short for **Alternative Unit-norm momentum updates by Normalized nonlinear scaling**, is a linear-time optimizer for deep learning that transforms a gradient or momentum update by first normalizing it, then measuring its tail-heaviness with a hyperbolic-cosine RMS statistic, and finally shrinking the entire update by a single global factor so that its spectral norm is strictly bounded. In the formulation presented by Ryuzan, AuON is intended as a linear-time alternative to semi-orthogonal momentum updates such as Muon: it preserves the direction and internal structure of the update, suppresses spiky updates through tail-aware scaling, and avoids explicit semi-orthogonalization and the $\mathcal{O}(n^2)$ overhead of Newton–Schulz iterations [2509.24320].

## 1. Optimization context and motivating problem

AuON is positioned within the broader line of **orthogonality-inspired optimizers**. The motivating observation is that gradients and momentum updates in deep networks are often **ill-conditioned**: a small number of directions can dominate, while others remain weak. In this setting, strict orthogonalization replaces an update matrix $G \in \mathbb{R}^{m \times n}$ with its polar factor $Q := U V^\top$ from the SVD $G = U \Sigma V^\top$, thereby preserving singular vectors while flattening singular values to $1$. This yields an update with spectral norm $\|Q\|_2 = 1$ on the active subspace and promotes more isotropic exploration [2509.24320].

The paper identifies two limitations of strict orthogonalization. First, SVD- or QR-based methods incur $\mathcal{O}(n^3)$ cost for square-like matrices, which is prohibitive in modern deep networks. Second, orthogonal-SGDM-style approaches can underperform well-tuned SGD with momentum because they orthogonalize gradients first and apply momentum afterward, thereby destroying the variance-reducing property of momentum, removing magnitude information in $\Sigma$, and over-constraining updates to isometries [2509.24320].

Muon improves on this by applying momentum before semi-orthogonalization and approximating the polar factor by Newton–Schulz iterations, reducing the dominant cost to $\mathcal{O}(n^2)$ per step. AuON is introduced to remove that remaining quadratic bottleneck. Its stated objective is to retain several benefits associated with semi-orthogonal updates—spectral-norm control, better conditioning, and stability with larger learning rates—while using only elementwise operations and simple reductions, hence linear time in the number of parameters [2509.24320].

| Method | Defining operation | Reported cost |
|---|---|---|
| Strict orthogonalization | SVD/QR-based orthogonalization | $\mathcal{O}(n^3)$ |
| Muon | Newton–Schulz iterations on momentum | $\mathcal{O}(n^2)$ per step |
| AuON | Frobenius normalization plus cosh-RMS scaling | $\mathcal{O}(N)$ |
| Hybrid-AuON | One Newton–Schulz iteration plus AuON scaling | Between AuON and Muon |

This comparison suggests that AuON is less an orthogonalizer than a **trust-region reconditioner** for momentum updates: it does not construct a semi-orthogonal matrix, but instead seeks to preserve correlation structure while shrinking updates into a bounded spectral region.

## 2. Core transformation and update semantics

For an update tensor or matrix $G$ with $N$ elements, AuON defines a three-stage transformation. In the notation used in the paper,

$$
\widetilde{G} = \frac{G}{\|G\|_F + \varepsilon_0}, \qquad
r = \frac{1}{N}\sum_{i,j} \cosh^2(\widetilde{G}_{ij}), \qquad
U = \frac{\widetilde{G}}{r + \varepsilon}.
$$

The resulting $U$ is then used in the parameter update
$$
\theta_{t+1} = \theta_t - \eta\, U.
$$

An equivalent presentation computes
$$
x = \cosh(\mathrm{update}), \qquad
\mathrm{rms} = \sqrt{\frac{1}{N}\sum_{i=1}^{N} x_i^2}
= \frac{\|\cosh(\mathrm{update})\|_F}{\sqrt{N}},
$$
and then scales by $\mathrm{rms} + \varepsilon$. The nonlinear part is therefore confined to the statistic used to determine a global scaling factor; the final update itself is a uniformly scaled version of the normalized input [2509.24320].

The paper emphasizes four consequences of this construction. First, AuON is **scale-invariant** after the initial Frobenius normalization: multiplying the raw update by a positive scalar does not materially change the transformed direction. Second, it is **tail-aware** because $\cosh(z)$ is quadratic near zero and exponential in the tails; large coordinates contribute disproportionately to the scaling statistic and therefore induce stronger shrinkage. Third, it **preserves direction** in the precise sense that all coordinates are scaled by the same factor, leaving their relative ratios unchanged. Fourth, it **preserves structural alignment** because it does not decorrelate rows or columns the way full orthogonalization does [2509.24320].

In practical use, AuON is generally applied not to the raw gradient but to a momentum-smoothed update:
$$
m_t = (1-\beta)m_{t-1} + \beta\, g_t.
$$
The transformed update $U_t$ is then computed from $m_t$. This ordering mirrors the motivation taken from Muon and from the critique of methods that orthogonalize before momentum [2509.24320].

A common misconception is that AuON applies $\cosh$ directly to the update sent to parameters. The paper states the opposite: $\cosh$ is used only to compute the scalar shrinkage factor. Another misconception is that “Alternative Unit-norm” means exact unit $\ell_2$ norm or exact unit RMS. The paper explicitly notes that AuON does **not** force the step to exactly unit L2 or unit RMS; rather, it guarantees a bound on the spectral norm and makes the step smaller when the cosh-RMS statistic is larger [2509.24320].

## 3. Spectral-norm trust region and semi-orthogonal interpretation

The central formal claim is that AuON places each transformed update inside a **strict spectral-norm trust region**. For
$$
U = \frac{\widetilde{G}}{r+\varepsilon},
$$
the paper derives
$$
\|U\|_2 \;\leq\; \frac{1}{r+\varepsilon} \;\leq\; \frac{1}{1+\tfrac{1}{N}+\varepsilon} \;<\; 1.
$$
Using $\cosh(x) \ge 1 + x^2/2$ and $\|\widetilde{G}\|_F \approx 1$, it follows that $r+\varepsilon > 1$, so every AuON step lies in a strictly contracted spectral ball $\|U\|_2 \le \rho < 1$ [2509.24320].

This trust-region interpretation is strengthened by the paper’s analysis of **tail sensitivity**. If one coordinate of the normalized update satisfies $\widetilde{G}_{ij}=a$, then
$$
r \ge \frac{\cosh(a)}{N}, \qquad
\|U\|_2 \le \frac{N}{\cosh(a)} \sim \frac{2}{N} e^{-|a|}.
$$
A single large spike therefore induces exponential contraction of the allowable spectral norm. This is the formal mechanism behind the claim that AuON damps heavy-tailed or harmful updates [2509.24320].

The paper also analyzes **correlation energy** and a form of “near semi-orthogonality.” Defining
$$
M := U^\top U,
$$
one has
$$
M = \frac{\widetilde{G}^\top \widetilde{G}}{(r+\varepsilon)^2}, \qquad
\operatorname{tr}(M) = \frac{\|\widetilde{G}\|_F^2}{(r+\varepsilon)^2}.
$$
The isotropy residual obeys
$$
\left\|M - \frac{1}{n}\operatorname{tr}(M) I\right\|_F
=
\frac{\left\|\widetilde{G}^\top \widetilde{G} - \frac{1}{n}\operatorname{tr}(\widetilde{G}^\top \widetilde{G}) I \right\|_F}{(r+\varepsilon)^2}.
$$
Thus, both total spectral energy and deviation from isotropy are shrunk by $(r+\varepsilon)^{-2}$. The paper is explicit that this is **not** true orthogonalization: AuON does not drive off-diagonal correlations to zero. Instead, it uniformly contracts correlation energy and anisotropy, which is presented as an approximation to some semi-orthogonal properties in a single cheap pass [2509.24320].

Under standard assumptions—$L$-smoothness, alignment of the AuON update with the true gradient, and bounded variance—the appendix proves an average-gradient-norm convergence rate of $\mathcal{O}(1/\sqrt{T})$ when $\eta = \Theta(1/\sqrt{T})$. The deterministic trust-region bound yields $\sigma^2 \le \rho^2 < 1$, which the paper states tightens the variance term relative to vanilla SGD [2509.24320].

## 4. Hybrid-AuON, implementation, and computational profile

Hybrid-AuON augments the base method with a single Newton–Schulz-like step before the AuON scaling. For a normalized update matrix $X$, the paper gives
$$
A = X X^\top, \qquad
B = bA + cA^2, \qquad
X \leftarrow aX + BX,
$$
with example constants $a=1.0$, $b=-0.5$, and $c=0.375$ in the code. This step is described as reducing correlations between columns or rows and moving $X$ somewhat closer to semi-orthogonal, after which the standard AuON normalization is applied [2509.24320].

The computational distinction is straightforward. AuON uses norms, $\cosh$, RMS computation, and scalar divisions, all of which are reported as $\mathcal{O}(N)$ per tensor. Hybrid-AuON adds one matrix multiplication $X X^\top$ and one $A^2$, giving cost on the order of $\mathcal{O}(n^2)$ for square matrices, but still less than Muon because it performs only one Newton–Schulz iteration rather than several [2509.24320].

Implementation details in the reference PyTorch code are concrete. The AuON scaling function `zeropower_via_cosh_rms(G, steps, newton_s1)` casts to `bfloat16`, normalizes by L2 norm, optionally performs the Newton–Schulz step for Hybrid-AuON, and then applies the cosh-RMS scaling. The optimizer is integrated as a Muon-like class with a standard momentum buffer. Numerical stability is handled by small constants such as $10^{-7}$ and $10^{-8}$, and the paper reports that $\cosh$ is usually safe because it is applied to normalized inputs [2509.24320].

The reported hyperparameter behavior is also distinctive. AuON typically uses larger learning rates than AdamW in the provided experiments—for example, $0.24$ versus $0.003$ in the nanoGPT experiment—and momentum values around $\beta \approx 0.9$. This suggests that the trust-region effect is intended not merely as a stabilizer but as an enabler of larger nominal step sizes [2509.24320].

## 5. Empirical behavior across language and vision benchmarks

The paper reports experiments on language modeling with nanoGPT on SmolLM-Corpus, an MNIST MLP, and CIFAR-10 CNNs. In the nanoGPT-style transformer setting—hidden size $512$, $6$ layers, $8$ heads, FFN dimension $1536$, RoPE, RMSNorm, SwiGLU, FlashAttention-2, trained for $6000$ steps with global batch size $128$ on a $500$k-token SmolLM-Corpus subset—the reported results are:

- **AdamW**: loss $0.0686$, accuracy $0.9846$, perplexity $1.07$
- **AuON**: loss $0.0476$, accuracy $0.9897$, perplexity $1.05$
- **Hybrid-AuON**: loss $0.0422$, accuracy $0.9908$, perplexity $1.04$
- **Muon**: loss $0.0375$, accuracy $0.9919$, perplexity $1.04$

The paper’s interpretation is that all AuON variants and Muon outperform AdamW under the given tuning, with Muon best and Hybrid-AuON close behind [2509.24320].

On MNIST with an MLP, the reported final losses are $0.2410$ for SGD, $0.0178$ for AdamW, and $0.0365$ for AuON, with runtimes of $65.36$ s, $69.28$ s, and $74.10$ s respectively. AuON is therefore described as much better than SGD but worse than tuned AdamW in that simple setting, with modest overhead of about $13\%$ versus SGD [2509.24320].

On a small CIFAR-10 CNN experiment, the reported results are:

- **SGD**: final loss $1.0821$, test accuracy $65.99\%$, time $88.10$ s
- **AdamW**: final loss $0.8768$, test accuracy $72.48\%$, time $88.94$ s
- **AuON**: final loss $0.8150$, test accuracy $74.42\%$, time $89.39$ s

In this case, AuON achieves the best reported accuracy and lowest loss while having runtime essentially identical to SGD and AdamW. In a larger reduced-scale CIFAR-10 setup with approximately $19.9$M parameters, $100$ epochs, batch size $128$, AdamW base learning rate $10^{-3}$, AuON learning rate $0.04$, weight decay $10^{-4}$, and momentum $(\beta_1,\beta_2)=(0.9,0.9)$, the paper reports $79.90\%$ test accuracy for AdamW and $80.38\%$ for AuON, a gain of $0.48\%$ [2509.24320].

The overall empirical picture given by the paper is therefore specific rather than universal: AuON consistently improves over SGD, often matches or slightly surpasses AdamW, Hybrid-AuON is somewhat better than AuON and closer to Muon, and Muon remains best on some metrics but at higher computational cost.

## 6. Practical interpretation, limitations, and research position

The paper recommends AuON when better conditioning and stability than SGD are desired, when a drop-in alternative to AdamW with similar runtime is attractive, and when larger learning rates are useful but Muon’s Newton–Schulz overhead is undesirable. Hybrid-AuON is recommended when one can afford a modest increase in compute to obtain partial decorrelation and performance closer to Muon. Muon remains preferable when strong semi-orthogonality is required and $\mathcal{O}(n^2)$ cost is acceptable [2509.24320].

Several limitations are stated explicitly. AuON does **not** explicitly decorrelate directions, so applications requiring stronger semi-orthogonal behavior may favor Muon or Hybrid-AuON. On very large transformer models, the paper reports potential instabilities such as exploding attention logits, similar to issues observed for other orthogonalization-based optimizers. Large-scale evaluations are left to future work, and techniques such as QK-clipping are mentioned as possible mitigations [2509.24320].

In the broader literature position given by the paper, AuON belongs to a family that includes SVD-based orthogonal-SGDM, Muon and its variants, and trust-region interpretations of orthogonalized gradients. It is also connected to non-Euclidean trust-region methods, because orthogonalized momentum can be viewed as solving a trust-region problem in the spectral norm, whereas AuON attains a related effect by directly enforcing $\|U\|_2 < 1$ through cosh-RMS scaling. The cosh-based scaling is further related to cosh gradient systems and tail-sensitive metrics, which the paper invokes as a way of reconditioning updates to be more robust to heavy tails [2509.24320].

Future directions proposed in the paper include scaling AuON and Hybrid-AuON to larger architectures and longer runs, combining them with QK-clipping in transformers, designing more sophisticated correlation-reducing steps compatible with linear-time cost, and integrating AuON with adaptive stepsize schemes in the style of AdaGrad-meets-Muon [2509.24320].

Taken together, these elements define AuON as a spectral-trust-region optimizer rather than a literal orthogonalizer: it preserves update direction and correlation structure, contracts spectral energy and anisotropy through a tail-aware global normalization, and aims to capture part of the optimization behavior associated with semi-orthogonal momentum updates at $\mathcal{O}(N)$ cost.

Source: https://www.emergentmind.com/topics/auon