--- title: 'HTMuon: Heavy-Tailed Matrix Optimization & Physics' url: https://www.emergentmind.com/topics/htmuon type: topic --- # HTMuon: Heavy-Tailed Matrix Optimization & Physics The term **HTMuon** appears in multiple research contexts. In machine learning, it denotes a matrix-valued optimizer introduced as “HTMuon: Improving Muon via Heavy-Tailed Spectral Correction,” where it is presented as a drop-in replacement and improvement over Muon for LLM pretraining and other large deep networks [2603.10067]. In CMS Run 3 trigger language, the same term corresponds to the **High Multiplicity Trigger (HMT)** based on cathode strip chamber hit multiplicities in the muon system [2606.27165]. A separate high-energy physics usage applies “HTMuon” to the Higgs–muon coupling at a high-energy muon collider, especially in the context of multi-boson production and precision tests of \(y_\mu\) [2305.12814]. Across these usages, the term consistently refers to muon-related high-throughput or high-sensitivity structures, but the underlying objects, methods, and scientific goals are distinct. ## 1. HTMuon as a matrix-valued optimizer In the optimization literature, HTMuon is a **matrix-based optimizer** that operates on each parameter matrix \(W \in \mathbb{R}^{m\times n}\) rather than on flattened vectors. Its design goal is to preserve Muon’s geometry-aware handling of parameter interdependencies while replacing Muon’s orthogonalized, spectrally flat update rule with one whose singular values are explicitly heavy-tailed [2603.10067]. The motivation is tied to **Heavy-Tailed Self-Regularization (HT-SR)** theory. In that framework, well-trained deep networks exhibit heavy-tailed empirical spectral densities of weight correlation matrices, and smaller power-law exponents \(\alpha\) correlate with better model quality and generalization. The HTMuon paper argues that Muon’s practice of setting all singular values of the update to \(1\) suppresses the emergence of heavy-tailed weight spectra and over-emphasizes training along noise-dominated directions. The paper further reports that exact-SVD Muon, whose update spectrum is exactly flat, is consistently outperformed by Newton–Schulz Muon on LLaMA-60M and 135M pretraining on C4, with the latter exhibiting nontrivial spectral deviations that effectively down-weight noise-dominated directions [2603.10067]. Within this formulation, HTMuon interpolates between two limiting cases. When the singular-value transform exponent is \(p=0\), the method reduces to Muon, whose update is purely orthogonal. When \(p=1\), it recovers SGDM on matrices. For \(p\in(0,1)\), it remains matrix-based and geometry-aware, but produces an update spectrum that is heavier-tailed than Muon’s and empirically induces heavier-tailed weight spectra. The default choice reported in the paper is \(p=0.125\), selected through ablations on LLaMA, OpenWebText, and vision models [2603.10067]. ## 2. Algorithmic structure and heavy-tailed spectral correction HTMuon retains Muon’s momentum-and-SVD structure, but replaces spectral flattening with a power transform. If the momentum matrix is \[ M_t=\beta M_{t-1}+(1-\beta)G_t,\qquad G_t=\nabla_W L(W_t), \] and its singular value decomposition is \[ M_t=U_t\Sigma_tV_t^\top, \] then HTMuon defines the update matrix as \[ \Delta_t=U_t\Sigma_t^pV_t^\top,\qquad p\in(0,1). \] The weight update is \[ W_{t+1}=W_t-\eta\lambda W_t-\eta\, s\,\Delta_t, \] with shape scaling \[ s=\sqrt{\max(1,m/n)}. \] This is the central algorithmic difference from Muon, which replaces \(\Sigma_t\) by the identity and therefore erases all singular-value variation [2603.10067]. The power transform \(\sigma_i\mapsto \sigma_i^p\) preserves the singular vectors \(U_t\) and \(V_t\) while compressing singular values nonlinearly. In the formulation given in the paper, large singular values remain relatively large, small singular values are shrunk more strongly, and the relative ordering is preserved. The intended consequence is that the optimizer still exploits matrix geometry but no longer assigns equal magnitude to all spectral directions, including directions dominated by noise [2603.10067]. The paper also studies an ablation called **HTMuon\_HT**, in which the singular values are replaced by a fixed heavy-tailed sequence \((\Sigma_t)_{ii}=i^{-\alpha}\) rather than by the data-dependent spectrum of \(M_t\). That variant still improves over Muon, which the authors interpret as evidence that heavy-tail injection itself is beneficial. However, the full HTMuon method performs better, indicating that preserving and reshaping the data-dependent singular spectrum is stronger than imposing a fixed heavy-tailed template [2603.10067]. A related spectral argument is given through a toy model. If the singular values of \(M_t\) follow a power law \(s_k=s_1k^{-s}\), then the empirical spectral density of the correlation matrix of \(\Delta_t\) obeys a power law with exponent \[ \alpha=1+\frac{1}{2sp}. \] This formalizes the optimizer’s role as a **spectral correction mechanism** rather than a mere orthogonalizer [2603.10067]. ## 3. Geometric interpretation and theoretical analysis HTMuon is theoretically characterized as a steepest-descent method in a non-Euclidean matrix geometry. The paper considers the constrained first-order problem \[ \max_{\|\Delta\|_q\le \delta}\operatorname{Tr}(\widetilde{G}_t^\top\Delta),\qquad \widetilde{G}_t=-\nabla f(W_t), \] with \(\|\cdot\|_q\) a Schatten-\(q\) norm. If \(\widetilde{G}_t=U\Sigma V^\top\), the maximizer is \[ \Delta^*=\delta \frac{1}{\|\widetilde{G}_t\|_{p'}^{p'/q}}\,U\Sigma^{p'-1}V^\top, \] where \(p'\) is Hölder-conjugate to \(q\). Identifying the momentum with the negative gradient gives HTMuon’s update form \[ \Delta_t\propto U\Sigma^pV^\top \] when \(p=p'-1\). In this sense, HTMuon is steepest descent under a Schatten-\(q\) norm constraint, while Muon is the special case associated with a Schatten-\(\infty\) constraint and a spectrally flat update [2603.10067]. The same paper gives convergence analysis in smooth non-convex settings under three assumptions: Frobenius-smoothness, unbiased stochastic gradients with bounded variance, and bounded singular values of the momentum. With an adaptive step size \[ \eta_t=\frac{\langle G_t,\rho(M_t)\rangle}{L\|\rho(M_t)\|_F^2},\qquad \rho(M_t)=U_t\Sigma_t^pV_t^\top, \] the average stationarity measure in a suitable norm obeys an \(O(1/T)\)-type bound, and with \(\beta=1/\sqrt{T}\) and \(B=T\) the sample complexity is \[ T=O(\epsilon^{-4}) \] for reaching an \(\epsilon\)-stationary point. The paper emphasizes that this matches the asymptotic sample-complexity upper bound of Muon and SGDM while changing the update geometry [2603.10067]. A later analysis of Schatten-\(p\) geometries places HTMuon-like methods within the broader **SODA** framework and argues that optimizer choice is regime dependent. Even when the objective is smooth in the Schatten-\(\infty\) geometry, smaller Schatten-\(p\) geometries can be optimal in the **low-dimensional regime**, which the paper states includes Chinchilla scaling. The same analysis explains why Muon-like methods do not require warmup, why they naturally favor large batches, and provides a batch-size scaling rule for arbitrary \(p\) [2606.15268]. This suggests that HTMuon’s finite-\(p\)-style spectral shaping is not merely an empirical modification of Muon, but part of a broader family of non-Euclidean training geometries. ## 4. Empirical performance and implementation The empirical evaluation in the HTMuon paper spans LLM pretraining, image classification, and downstream commonsense reasoning. On LLaMA pretraining on C4, the reported reduction in perplexity relative to Muon reaches **up to \(0.98\)**, which is the headline quantitative example given in the abstract [2603.10067]. | Benchmark | Baseline | HTMuon | |---|---:|---:| | LLaMA-60M on C4 (PPL) | Muon 28.80 | 27.88 | | LLaMA-135M on C4 (PPL) | Muon 22.23 | 21.25 | | GPT-2 small on OpenWebText (PPL) | Muon 22.46 | 22.20 | | ResNet-50 on CIFAR-100 (top-1) | Muon 79.85% | 80.16% | | ViT-tiny on ImageNet-1K (top-1) | Muon 71.02% | 71.16% | | LLaMA-1B, 7-task average accuracy | Muon 40.03 | 41.08 | These results are accompanied by comparisons to Adam, AdamW, COSMOS, NorMuon, AdaMuon, Cautious, MARS, SOAP, GaLore, Sophia, and other baselines. HTMuon is also presented as a plug-in on top of Muon variants; for example, **HTMuon+NorMuon** improves over NorMuon on LLaMA-60M and LLaMA-135M, and the paper reports that HTMuon consistently improves over each Muon variant studied [2603.10067]. The main practical cost is computational overhead. Exact HTMuon requires an SVD per parameter matrix per update, which is significantly more expensive than Muon’s usual Newton–Schulz approximation. The paper therefore introduces two acceleration strategies: **interval updates**, where HTMuon is applied every \(k\) steps and Muon is used otherwise, and **HTMuon\_NS**, which approximates the spectral power transform without an exact SVD. On \(4\times\)A6000, the reported per-step wall times for LLaMA-60M are \(0.51\) s for Muon, \(1.26\) s for HTMuon with interval \(=1\), \(0.66\) s for HTMuon\_NS with interval \(=1\), and \(0.59\) s for HTMuon\_NS with interval \(=5\) [2603.10067]. The paper recommends \(p=0.125\) as a robust default, retains the same learning-rate, weight-decay, and momentum interfaces as Muon, and reports released code at `https://github.com/TDCSZ327/HTmuon` [2603.10067]. ## 5. Other scientific uses of “HTMuon” Outside optimization, **HTMuon** is also used in detector and collider physics. In CMS Run 3 trigger language, it refers to the **High Multiplicity Trigger (HMT)** for long-lived particle searches in the endcap muon system. At Level-1, the trigger counts anode and cathode hits in CSC chambers per bunch crossing, requires hits in at least \(5\) of \(6\) layers, and classifies chambers into **No Shower**, **Loose**, **Nominal**, and **Tight** categories using a 2-bit encoding. At the HLT level, CSC rechits are clustered in \(\eta\)-\(\phi\) with the Cambridge–Aachen algorithm and radius \[ R=0.4, \] with a cluster defined by at least \(50\) CSC rechits and a timing selection \[ -5.0~\text{ns}