DynMuon: Dynamic Spectral Shaping

• DynMuon is a dynamic spectral-shaping method for Muon that generalizes matrix updates by varying the exponent p during training.
• It employs a logistic schedule that transitions p from positive, enhancing high-curvature modes early, to mildly negative, accentuating flatter modes later.
• Experimental results demonstrate up to 26.5% fewer training steps and consistently lower validation loss with minimal computational overhead in transformer models.

DynMuon is a dynamic spectral-shaping method for Muon, formulated by replacing a matrix update $M = U \Sigma V^\top$ with $U \Sigma^p V^\top$ and varying the exponent $p$ over training rather than fixing it at the Muon choice $p=0$. In this formulation, $p=1$ recovers standard gradient descent, $p=0$ recovers Muon’s polar update $UV^\top$, and $p<0$ yields inverse-spectrum emphasis. DynMuon’s central claim is that the optimal spectral shaping is stage dependent: positive $p$ is advantageous early, when high-curvature directions dominate the residual signal, whereas mildly negative $p$ is advantageous later, when low-curvature directions still contain useful training signal. The reported result is consistently lower validation loss than Muon, together with $U \Sigma^p V^\top$0 fewer steps to reach the same target loss, at only $U \Sigma^p V^\top$1 per-step overhead across the evaluated settings (Wu et al., 16 May 2026).

1. Spectral-shaping formulation

DynMuon is defined within the broader class of Muon-like matrix updates. Standard matrix-valued descent applies the raw or momentum-averaged gradient $U \Sigma^p V^\top$2 directly, so if $U \Sigma^p V^\top$3, an SGD-style step is $U \Sigma^p V^\top$4. Muon instead discards the singular-value magnitudes and applies the polar factor, $U \Sigma^p V^\top$5. DynMuon generalizes both by introducing the one-parameter family

$U \Sigma^p V^\top$6

described as a spectral-shaping operation (Wu et al., 16 May 2026).

$U \Sigma^p V^\top$7	Update	Interpretation
$U \Sigma^p V^\top$8	$U \Sigma^p V^\top$9	raw gradient descent
$p$0	$p$1	Muon
$p$2	$p$3	inverse-spectrum update

This parameterization makes explicit that Muon is not an isolated optimizer but a specific point in a continuous family. Large $p$4 emphasizes strong modes, while small or negative $p$5 emphasizes flat modes. The significance of DynMuon is therefore not merely a new schedule, but a reinterpretation of Muon as one static choice inside a broader spectral-control framework. The paper’s stated conclusion is that fixed Muon $p$6 is suboptimal in both early and late phases of training (Wu et al., 16 May 2026).

2. Noise-aware local model

The theoretical core of DynMuon is a local quadratic approximation to the population loss $p$7 around a nearby minimizer $p$8, augmented by an approximation of gradient noise. Writing the residual as $p$9, the model assumes

$p=0$0

where $p=0$1 is a normalized effective Hessian with eigenvalues $p=0$2, and $p=0$3 is a scalar curvature scale. The stochastic gradient is then modeled as

$p=0$4

Under spectral shaping with exponent $p=0$5, and after absorbing scalar factors into $p=0$6, the residual in mode $p=0$7 evolves as

$p=0$8

and the second moment satisfies

$p=0$9

These expressions expose the signal–noise trade-off that motivates DynMuon: contraction in mode $p=1$0 is governed by $p=1$1, so increasing $p=1$2 accelerates contraction in high-curvature directions, while noise amplification scales as $p=1$3, so decreasing $p=1$4 amplifies noise more strongly in flat modes (Wu et al., 16 May 2026).

Two stage-dependent regimes follow from this model. Early in training, residual signal is concentrated in high-curvature, or “strong,” modes; positive $p=1$5 accelerates their contraction while incurring only mild noise amplification. Later in training, strong modes have largely decayed, while flatter modes still carry signal relative to their smaller noise; mildly negative $p=1$6 then reallocates update strength toward those flatter modes. The paper also emphasizes the boundary of this effect: if $p=1$7 becomes too negative, noise amplification overwhelms the benefit. Empirical validation is reported on a frozen final-layer matrix in a GPT-style block, using measurements of mode-wise curvature, residual-signal energy, and noise levels (Wu et al., 16 May 2026).

3. Dynamic schedule and efficient realization

DynMuon operationalizes the stage-dependent theory by scheduling $p=1$8 to decrease smoothly from a positive early value $p=1$9 to a mildly negative late value $p=0$0. For step $p=0$1, the schedule is

$p=0$2

Typical choices are $p=0$3, $p=0$4, $p=0$5, and $p=0$6. This logistic schedule yields $p=0$7 in the first few percent of training, transitions around $p=0$8, and approaches $p=0$9 in the late stage. The intended effect is explicit: positive early $UV^\top$0 boosts contraction of strong modes, and mild negative late $UV^\top$1 focuses on flat modes (Wu et al., 16 May 2026).

Exact SVD at every optimization step would be prohibitive, so DynMuon preserves the Newton–Schulz iterations already used by Muon to approximate the polar factor and augments them with a low-order polynomial correction for arbitrary negative exponents. The implementation logic is given as follows: if $UV^\top$2, return the raw matrix $UV^\top$3; if $UV^\top$4, return $UV^\top$5; otherwise return $UV^\top$6. The Fast-Spectral routine normalizes the input by its Frobenius norm, computes a Newton–Schulz approximation $UV^\top$7, forms $UV^\top$8, $UV^\top$9, and $p<0$0, then applies the second-order Taylor approximation

$p<0$1

to approximate $p<0$2, returning $p<0$3. The stated additional cost is only $p<0$4, matching Muon’s asymptotic runtime (Wu et al., 16 May 2026).

This design is significant because it separates the conceptual change—dynamic spectral shaping—from the dominant systems bottleneck of repeated full SVD. The paper’s argument is that dynamic shaping can be made practical without abandoning the efficient Newton–Schulz infrastructure already associated with Muon.

4. Experimental characterization

The reported experiments cover GPT-style transformers at $p<0$5M, $p<0$6M, and $p<0$7B parameters in a modded-NanoGPT style, together with a Qwen-style $p<0$8M decoder-only model. Training uses FineWeb and FineWeb-Edu, both described as public web-scraped corpora, with sequence length $p<0$9 and global batch $p$0. Training budgets span $p$1B to $p$2B tokens, corresponding to $p$3K to $p$4K steps. Baselines include Muon $p$5 with NS-based orthogonalization, AdamW for non-matrix parameters, and NorMuon in ablation. The main metrics are validation cross-entropy loss curves and “steps-to-target,” where the target is Muon’s $p$6-training-time loss (Wu et al., 16 May 2026).

Across all GPT scales on $p$7B tokens, DynMuon achieves $p$8 fewer steps to the same loss, while validation-loss curves remain consistently below Muon. Similar gains are reported on FineWeb-Edu, across learning-rate sweeps, across token budgets, and on the Qwen model. The paper further reports that the polynomial correction closely matches exact SVD shaping. These results are presented as evidence that the stage-dependent schedule is not restricted to a single architecture, dataset, or token budget (Wu et al., 16 May 2026).

A central empirical implication is that the benefit is not described solely as faster wall-clock execution, but as improved optimization efficiency under a fixed training objective. The measured per-step overhead is $p$9, so the claimed improvement is a reduction in optimization steps for a given loss threshold rather than a purely systems-level acceleration.

5. Position within the Muon ecosystem

DynMuon is part of a rapidly expanding Muon-centered optimization literature. The underlying premise is that matrix-orthogonalization-based optimizers offer an alternative to conventional element-wise methods because the update acts at the level of full weight matrices rather than individual coordinates. DynMuon extends this family by making the singular-value transformation explicit and time dependent, whereas Muon fixes the transformation to the polar factor (Wu et al., 16 May 2026).

A separate but related development is DMuon, which addresses the distributed-training cost of Muon rather than the choice of spectral exponent. DMuon preserves exact Muon semantics while reformulating the Newton–Schulz iteration in Gram space and executing it owner-centrically on sharded training systems. For a weight matrix $p$0 with momentum-aggregated gradient $p$1, Muon applies

$p$2

and DMuon replaces the direct $p$3 Newton–Schulz step by a Gram-space recurrence on $p$4 using $p$5, yielding $p$6 and $p$7. The reported outcomes are $p$8–$p$9 end-to-end per-step speedup versus vanilla Muon, $U \Sigma^p V^\top$00–$U \Sigma^p V^\top$01 optimizer-step speedup, and step-time overhead versus AdamW of $U \Sigma^p V^\top$02 to $U \Sigma^p V^\top$03 across workloads including Wall-OSS-0.5, Pi$U \Sigma^p V^\top$04, Wall-WM, and Qwen2.5-7B on $U \Sigma^p V^\top$05–$U \Sigma^p V^\top$06 GPUs (Chen et al., 25 Jun 2026).

The relationship between these works is methodological rather than identical. DynMuon studies which spectrum to apply over training; DMuon studies how to execute Muon efficiently in distributed environments. A plausible implication is that progress on dynamic spectral shaping increases the importance of scalable matrix-aware execution, because the practical adoption of Muon-family optimizers depends on both optimization behavior and distributed systems efficiency.

6. Limitations, open problems, and nomenclature

DynMuon’s stated limitations are specific. The schedule is hand-tuned and does not re-estimate $U \Sigma^p V^\top$07 online from data, leaving open the possibility of better adaptive schemes. The local quadratic/noise model is presented as a simplified guide rather than a full convergence theory. A fuller theory of convergence under dynamic shaping remains open. The method also focuses on matrix-valued updates, and future work is explicitly suggested in two directions: dynamic shaping in other optimizer families and combinations with second-order curvature information (Wu et al., 16 May 2026).

These caveats delimit the current scope of the method. The theoretical analysis is local and mode based, not a complete nonconvex training theory; the schedule is logistic and stage dependent, not data-adaptive; and the experiments are confined to transformer and language-model settings in which Muon is already positioned as a strong baseline. This suggests that DynMuon is best understood as a principled refinement of Muon’s spectral policy rather than a universal replacement for all optimizers.

The nomenclature can also cause confusion. DynMuon is distinct from DMuon, whose subject is distributed Muon execution (Chen et al., 25 Jun 2026). It is also unrelated to the gauge-theoretic object “dyonium” studied in “Dyonium Induced Fermion Number Violation” (Sugamoto, 2023), which concerns an $U \Sigma^p V^\top$08 theory with a Higgs doublet, $U \Sigma^p V^\top$09-string confinement, fermionic zero modes, and anomaly-driven fermion-number violation rather than optimization. This distinction matters because the similar strings “DynMuon,” “DMuon,” and “dyonium” refer to unrelated technical domains.

7. Conceptual significance

DynMuon’s principal contribution is the claim that spectral shaping of matrix-valued updates should vary over training rather than remain fixed. In the paper’s formulation, optimization behavior depends jointly on local curvature, stochastic-gradient and label-noise effects, and training stage. Positive $U \Sigma^p V^\top$10 accelerates contraction in high-curvature directions when those directions still dominate the residual; mildly negative $U \Sigma^p V^\top$11 later reallocates update strength toward low-curvature directions that continue to carry signal. The resulting picture reframes Muon from a single orthonormalization rule into one point on a dynamic spectrum-management continuum (Wu et al., 16 May 2026).

In that sense, DynMuon functions both as an optimizer and as an interpretive framework for Muon-like methods. Its empirical results support the claim that a single fixed spectral policy is not optimal throughout training, and its algorithmic construction shows that this policy can be adjusted with negligible extra cost. For work on large-scale transformer training, the broader significance lies in the explicit coupling of curvature-sensitive signal contraction, mode-wise noise amplification, and training-stage scheduling within a practical matrix-aware optimizer family.