---
title: 'Kaon Optimizer: Randomized Spectral Descent'
url: https://www.emergentmind.com/topics/kaon-optimizer
type: topic
---

# Kaon Optimizer: Randomized Spectral Descent

Kaon is a spectral optimizer developed as a deliberately "absurd" variant within the family of non-Euclidean matrix descent schemes. Introduced to challenge the consensus that precise geometric structure is central for deep network optimization, Kaon instead replaces the singular values of each layerwise gradient with normalized random noise. Despite discarding all coherent geometry, Kaon matches the performance of recent spectral optimizers such as Muon and Freon, demonstrating that optimization performance is dominated not by spectrum shaping but rather by local alignment and descent-potential, combined with appropriate step-size hyperparameter selection [2605.11181].

## 1. Spectral Descent Optimizers: From Muon and Freon to Kaon

Spectral descent algorithms operate on gradients $G\in\mathbb{R}^{m\times n}$ in deep learning by decomposing $G$ into singular values and modifying the descent direction in the spectral domain. Muon achieves full "spectrum-whitening" through a linear-minimization-oracle (LMO) step:
\[
D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.
\]
Freon generalizes Muon to steepest descent in Schatten–$p$ quasi-norms:
\[
D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],
\]
with $c=0$ recovering SGD and $c=1/2$ yielding Muon. Empirically, the best Freon exponents for language models such as GPT-2 fall in the quasi-norm regime $c\in(1/2,1)$, outside of any proper unitarily invariant LMO.

Kaon takes this abstraction further by discarding any spectrum structure, simply replacing each singular value $\sigma_i$ of $G$ with a random positive number (sampled i.i.d. and normalized to unit $\ell_2$ norm), thereby proving that specific spectral geometry is not required for effective deep learning descent [2605.11181].

## 2. Formal Algorithmic Description

Given a gradient $G_k = \nabla f(X_k)$ at iteration $k$, the singular value decomposition is
\[
G_k = U_k\,\mathrm{diag}(\sigma_k)\,V_k^\top, \qquad \sigma_k \in \mathbb{R}^r_{>0}, \; r = \min(m,n).
\]

The general preconditioned spectral descent update is:
\[
X_{k+1} = X_k - \alpha_k \langle G_k, D_k \rangle D_k, \qquad D_k = U_k\,\mathrm{diag}(p_k(\sigma_k))\,V_k^\top.
\]

The Kaon optimizer selects $p_k(\cdot)$ as follows:

- Draw $E_k \in \mathbb{R}^r$ with entries $E_{k,i} \overset{\text{iid}}{\sim}\mathrm{Uniform}(0,1)$ (or any positive law)
- Normalize: $p_k(\sigma_k)_i = E_{k,i}/\|E_k\|_2$
- Construct $D_k = U_k\,\mathrm{diag}(p_k(\sigma_k))\,V_k^\top$
- Update: $X_{k+1} = X_k - \eta_k \langle G_k, D_k \rangle D_k$

**Pseudocode**:
```plaintext
Input:  G∈ℝ^{m×n}, step-size η>0, mini-batch noise generator
Output: updated parameter matrix X

Initialize X ← X_prev
Compute G = ∇f(X_prev)
Compute SVD: [U, σ, V] ← SVD(G)
Draw E ∈ ℝ^r with iid positive entries (e.g., Uniform(0,1))
Set p = E / ‖E‖₂
Form direction D = U · diag(p) · Vᵀ
Compute inner-product μ = ⟨G, D⟩
Update X ← X_prev − η · μ · D
Return X
```

Hyperparameters include the learning-rate $\eta$, the distribution for $E$, batch size, and standard optimizer settings such as momentum and weight decay.

## 3. Theoretical Guarantees

Kaon, as a member of the preconditioned spectral descent family, inherits convergence guarantees under standard assumptions for smooth objective functions. The convergence theorem states:

Let $f:\mathbb{R}^{m\times n}\rightarrow\mathbb{R}$ be differentiable, bounded below, with $L$-Lipschitz continuous gradient under a unitarily invariant norm $\|\cdot\|$. For each iteration $k$, choose
\[
D_k = U_k\,\mathrm{diag}(p_k(\sigma_k))\,V_k^\top, \quad p_k(\sigma_k) = \frac{E_k}{\|E_k\|_2}
\]
with $E_k$ drawn i.i.d. from a positive distribution. Then, for step-size $\eta\in(0,2)$, almost surely:
\[
\min_{0\leq k < K} \|G_k\|_*^2 = \mathcal{O}(1/K), \qquad \liminf_{k\to\infty}\|G_k\|_* = 0,
\]
where $\|\cdot\|_*$ is the dual norm (e.g., Frobenius).

Key ingredients in the proof are that, almost surely, constants $m_k>0$ and $M_k<\infty$ exist such that $\langle \sigma_k,p_k(\sigma_k)\rangle\ge m_k\|\sigma_k\|_*$ and $\|p_k(\sigma_k)\|\le M_k$. Since $\mathbb{E}[m_k^2/M_k^2]>0$, the standard argument for preconditioned spectral descent yields global sublinear convergence [2605.11181].

## 4. Alignment and Descent Potential

All spectral methods, including Kaon, operate by balancing two core local quantities:

- **Alignment** ($\gamma_k$): Ratio of global batch-gradient descent along $D_k$ to local mini-batch descent.
  \[
  \gamma_k := \frac{\langle G_k, D_k\rangle}{\langle \widetilde G_k,D_k\rangle}, \quad (G_k = \mathbb{E}[\widetilde G_k])
  \]
- **Descent Potential** ($\Phi_k$): Directional gain per step, normalized by curvature.
  \[
  \Phi_k := \frac{\langle \widetilde G_k,D_k\rangle^2}{\langle D_k, \nabla^2 f(Z_k)[D_k]\rangle}
  \]
  where $Z_k$ is an intermediate point between $X_k$ and $X_{k+1}$.

The one-step Taylor expansion yields:
\[
f(X_{k+1}) - f(X_k) =
- \Phi_k (\gamma_k\alpha_k - \tfrac12 \alpha_k^2 \lambda_k)
\]
with optimal step-size $\alpha_k^* = \gamma_k/\lambda_k$ and
$\lambda_k = \langle D_k, \nabla^2 f(Z_k)[D_k]\rangle$.

All spectral methods trade lower alignment $(\gamma_k<1)$ for larger descent potential $\Phi_k$. Kaon, while introducing randomization in the spectrum, maintains this tradeoff dynamically, provided step-size and other hyperparameters are tuned accordingly [2605.11181].

## 5. Empirical Results and Comparisons

On a language modeling pretraining task using WikiText-2 with a GPT-2–style architecture (124M parameters), Kaon performances were benchmarked against SGD, TruncatedSGD, Muon, and Freon. Results include:

| Optimizer          | Best Final Loss    | Convergence Speed | LR Sensitivity      |
|--------------------|-------------------|-------------------|---------------------|
| Muon               | 3.53              | ~50k tokens/halve | Flat, similar to Kaon|
| Kaon               | 3.55              | ~50k tokens/halve | Flat                |
| Freon (c=2/3)      | 3.54              | ~50k tokens/halve | Flat                |
| Freon (c=3/4)      | 3.52              | ~50k tokens/halve | Flat                |
| TruncSGD           | ~5.5              | Much slower       | High                |
| SGD                | ~6.7              | Stalled           | High                |

- All three spectral methods (Muon, Freon, Kaon) achieve nearly identical validation loss ($\sim$3.3–3.5) after similar numbers of training steps ($\sim$200k tokens).
- Kaon’s learning-rate sensitivity mirrors Muon’s and is notably less acute than that of SGD.
- Varying the noise generation (number of chaotic map iterations $T\in\{1,\ldots,10\}$ in Kaon) has negligible effect on performance; the estimator is robust to chaotic parameters within small ranges [2605.11181].

## 6. Implementation Guidelines and Practical Caveats

- The same learning-rate schedule as for Muon or Freon is recommended (matrix-update LR $\eta=2\times10^{-2}$, base LR in $[8\times10^{-3},2\times10^{-2}]$).
- Five chaotic map iterations per step suffice for noise generation; further iterations do not improve results.
- Momentum ($\sim 0.95$) and standard weight-decay/mask-norm clipping settings should be maintained.
- For large models, the SVD in Kaon can be replaced with a single Newton–Schulz iteration plus a single random vector sample to reduce computational overhead, with empirically similar behavior.
- Monitor alignment $\gamma_k$ (should remain above 0.6) and descent potential $\Phi_k$ on representative layers to ensure optimizer health.
- Since Kaon’s preconditioner is randomized, there is a nonzero risk of poor mini-batch draws; running two random seeds is advised when sweeping learning rates to ensure reliability.

## 7. Broader Significance and Implications

Kaon’s empirical equivalence with Muon and Freon, despite its disregard of geometric structure, falsifies the presumption that global spectrum geometry is essential for SGD performance improvement. The key determinants are shown to be local alignment and descent potential, not spectrum “whitening” per se. This suggests new directions in optimizer design, where spectral manipulation can be replaced with any mechanism that ensures suppression of large gradient modes and maintains sufficient expected descent potential, provided step size is tuned in accordance with local batch dynamics [2605.11181].

Source: https://www.emergentmind.com/topics/kaon-optimizer