Kaon Optimizer: Randomized Spectral Descent

• Kaon optimizer is a spectral descent method that replaces singular values with normalized random noise to disrupt the conventional geometric structure in deep learning.
• It achieves comparable performance to optimizers like Muon and Freon by emphasizing local alignment and descent potential over precise spectral shaping.
• Empirical results and theoretical guarantees demonstrate sublinear convergence and robust learning-rate sensitivity, making it effective for complex tasks.

Kaon is a spectral optimizer developed as a deliberately "absurd" variant within the family of non-Euclidean matrix descent schemes. Introduced to challenge the consensus that precise geometric structure is central for deep network optimization, Kaon instead replaces the singular values of each layerwise gradient with normalized random noise. Despite discarding all coherent geometry, Kaon matches the performance of recent spectral optimizers such as Muon and Freon, demonstrating that optimization performance is dominated not by spectrum shaping but rather by local alignment and descent-potential, combined with appropriate step-size hyperparameter selection (Shumaylov et al., 11 May 2026).

1. Spectral Descent Optimizers: From Muon and Freon to Kaon

Spectral descent algorithms operate on gradients $G\in\mathbb{R}^{m\times n}$ in deep learning by decomposing $G$ into singular values and modifying the descent direction in the spectral domain. Muon achieves full "spectrum-whitening" through a linear-minimization-oracle (LMO) step: $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$ Freon generalizes Muon to steepest descent in Schatten–$p$ quasi-norms: $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$ with $c=0$ recovering SGD and $c=1/2$ yielding Muon. Empirically, the best Freon exponents for LLMs such as GPT-2 fall in the quasi-norm regime $c\in(1/2,1)$, outside of any proper unitarily invariant LMO.

Kaon takes this abstraction further by discarding any spectrum structure, simply replacing each singular value $\sigma_i$ of $G$ with a random positive number (sampled i.i.d. and normalized to unit $G$0 norm), thereby proving that specific spectral geometry is not required for effective deep learning descent (Shumaylov et al., 11 May 2026).

2. Formal Algorithmic Description

Given a gradient $G$1 at iteration $G$2, the singular value decomposition is

$G$3

The general preconditioned spectral descent update is: $G$4

The Kaon optimizer selects $G$5 as follows:

• Draw $G$6 with entries $G$7 (or any positive law)
• Normalize: $G$8
• Construct $G$9
• Update: $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$0

Pseudocode:

$c=0$8

Hyperparameters include the learning-rate $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$1, the distribution for $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$2, batch size, and standard optimizer settings such as momentum and weight decay.

3. Theoretical Guarantees

Kaon, as a member of the preconditioned spectral descent family, inherits convergence guarantees under standard assumptions for smooth objective functions. The convergence theorem states:

Let $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$3 be differentiable, bounded below, with $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$4-Lipschitz continuous gradient under a unitarily invariant norm $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$5. For each iteration $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$6, choose

$$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$7

with $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$8 drawn i.i.d. from a positive distribution. Then, for step-size $$D_{\rm Muon} = (GG^\top)^{-1/2}G, \quad X_{k+1} = X_k - \eta \langle G_k, D_{\rm Muon}\rangle D_{\rm Muon}.$$9, almost surely: $p$0 where $p$1 is the dual norm (e.g., Frobenius).

Key ingredients in the proof are that, almost surely, constants $p$2 and $p$3 exist such that $p$4 and $p$5. Since $p$6, the standard argument for preconditioned spectral descent yields global sublinear convergence (Shumaylov et al., 11 May 2026).

4. Alignment and Descent Potential

All spectral methods, including Kaon, operate by balancing two core local quantities:

• Alignment ($p$7): Ratio of global batch-gradient descent along $p$8 to local mini-batch descent.

$p$9

• Descent Potential ($$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$0): Directional gain per step, normalized by curvature.

$$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$1

where $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$2 is an intermediate point between $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$3 and $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$4.

The one-step Taylor expansion yields: $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$5 with optimal step-size $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$6 and $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$7.

All spectral methods trade lower alignment $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$8 for larger descent potential $$D_{\rm Freon}(c) = (GG^\top)^{-c}G, \quad c\in[0,1],$$9. Kaon, while introducing randomization in the spectrum, maintains this tradeoff dynamically, provided step-size and other hyperparameters are tuned accordingly (Shumaylov et al., 11 May 2026).

5. Empirical Results and Comparisons

On a language modeling pretraining task using WikiText-2 with a GPT-2–style architecture (124M parameters), Kaon performances were benchmarked against SGD, TruncatedSGD, Muon, and Freon. Results include:

Optimizer	Best Final Loss	Convergence Speed	LR Sensitivity
Muon	3.53	~50k tokens/halve	Flat, similar to Kaon
Kaon	3.55	~50k tokens/halve	Flat
Freon (c=2/3)	3.54	~50k tokens/halve	Flat
Freon (c=3/4)	3.52	~50k tokens/halve	Flat
TruncSGD	~5.5	Much slower	High
SGD	~6.7	Stalled	High

• All three spectral methods (Muon, Freon, Kaon) achieve nearly identical validation loss ($c=0$03.3–3.5) after similar numbers of training steps ($c=0$1200k tokens).
• Kaon’s learning-rate sensitivity mirrors Muon’s and is notably less acute than that of SGD.
• Varying the noise generation (number of chaotic map iterations $c=0$2 in Kaon) has negligible effect on performance; the estimator is robust to chaotic parameters within small ranges (Shumaylov et al., 11 May 2026).

6. Implementation Guidelines and Practical Caveats

• The same learning-rate schedule as for Muon or Freon is recommended (matrix-update LR $c=0$3, base LR in $c=0$4).
• Five chaotic map iterations per step suffice for noise generation; further iterations do not improve results.
• Momentum ($c=0$5) and standard weight-decay/mask-norm clipping settings should be maintained.
• For large models, the SVD in Kaon can be replaced with a single Newton–Schulz iteration plus a single random vector sample to reduce computational overhead, with empirically similar behavior.
• Monitor alignment $c=0$6 (should remain above 0.6) and descent potential $c=0$7 on representative layers to ensure optimizer health.
• Since Kaon’s preconditioner is randomized, there is a nonzero risk of poor mini-batch draws; running two random seeds is advised when sweeping learning rates to ensure reliability.

7. Broader Significance and Implications

Kaon’s empirical equivalence with Muon and Freon, despite its disregard of geometric structure, falsifies the presumption that global spectrum geometry is essential for SGD performance improvement. The key determinants are shown to be local alignment and descent potential, not spectrum “whitening” per se. This suggests new directions in optimizer design, where spectral manipulation can be replaced with any mechanism that ensures suppression of large gradient modes and maintains sufficient expected descent potential, provided step size is tuned in accordance with local batch dynamics (Shumaylov et al., 11 May 2026).