Turbo-Muon Preconditioning Efficiency

• Turbo-Muon Preconditioning is a technique that accelerates orthogonality-based optimization by applying an Almost-Orthogonal (AOL) preconditioner to improve spectral conditioning.
• It refines the Newton–Schulz iterations by reducing iteration count and matrix multiplies while maintaining condition-number-independent convergence.
• Empirical results show a 2.8× speedup in orthogonalization and a 5–10% reduction in overall training time for large-scale deep learning models.

Turbo-Muon preconditioning is a methodology for accelerating orthogonality-based optimization in large-scale machine learning, specifically designed to address the computational bottleneck found in state-of-the-art Muon optimizers. At its core, Turbo-Muon introduces a diagonal preconditioning step—termed the Almost-Orthogonal (AOL) preconditioner—that dramatically improves the convergence of matrix polynomial iterations, especially Newton-Schulz-type, for orthogonal projection tasks. This technique makes the orthogonalization subroutine in Muon nearly twice as fast and enables substantial speedups in end-to-end training without loss in solution quality, with empirical and theoretical justification for its preconditioning benefits (Boissin et al., 4 Dec 2025, Ma et al., 20 Jan 2026, Amsel et al., 22 May 2025).

1. Orthogonality-Based Optimization and the Muon Framework

Muon is an optimization algorithm that leverages the polar decomposition of matrices for preconditioning weight updates in deep networks. It employs an update of the form

$W_{t+1} = W_t - \lambda\,\textrm{polar}(M_t)$

where $M_t$ is a momentum-averaged gradient matrix and $\textrm{polar}(M_t)=U V^\top$, given $M_t = U\Sigma V^\top$ via SVD. The polar factor corresponds to the matrix sign function, yielding the steepest descent direction in the operator norm. This approach transforms the gradient step into a spectrally balanced update, eliminating condition number dependence and aligning updates with the natural geometry of the objective (Amsel et al., 22 May 2025).

The high computational cost of this orthogonal projection, especially via iterative methods like Newton–Schulz, has historically limited the efficiency of Muon in large-scale settings. Newton–Schulz iterations for computing the polar factor require repetitive matrix–matrix multiplications, with each iteration incurring three matrix multiplies and practical deployments using 5 or more iterations per step (Boissin et al., 4 Dec 2025).

2. Turbo-Muon: Algorithm and Preconditioning Mechanism

Turbo-Muon modifies the Muon orthogonalization step by replacing the global Frobenius-norm scaling of the target matrix $X$ with the AOL preconditioner. Given $X_0\in\mathbb R^{n\times n}$, the procedure is:

1. AOL Preconditioning: Define the diagonal preconditioner

$$P = \operatorname{diag}(s_1,\dots,s_n),\qquad s_i=\left(\sum_{j=1}^n |(X_0^\top X_0)_{ij}|\right)^{-1/2}$$

yielding $X_1 = X_0P$. This ensures $\|X_1\|_2\leq 1$ (via Gershgorin’s circle) and regularizes the spread of singular values, improving conditioning for subsequent iterations.

2. Accelerated Newton–Schulz Iteration: Initialize $A_1 = X_1^\top X_1 = P(X_0^\top X_0)P$. Then perform four iterations:

$$\begin{aligned} B_k &= b_k A_k + c_k A_k^2 \ X_{k+1} &= a_k X_k + X_k B_k \ A_{k+1} &= X_{k+1}^\top X_{k+1} \end{aligned}$$

with $(a_k, b_k, c_k)$ chosen as fixed or adaptive kernel coefficients. The iteration count is reduced to four versus Muon+’s five, owing to improved spectral initialization (Boissin et al., 4 Dec 2025).

This process incurs only one additional matrix multiply beyond the Frobenius-norm scaling baseline due to efficient reuse of the Gram matrix computation. The AOL preconditioner thus reduces the overall computational cost from 15 to 13 multiplies per orthogonalization in the canonical deployment.

3. Convergence Properties and Spectral Decoupling

Turbo-Muon’s preconditioning effect can be analyzed via its spectral action. In matrix optimization problems such as symmetric matrix factorization or quadratic in-context learning objectives, Turbo-Muon’s update decouples into independent scalar recurrences along the eigenbasis of the optimal solution: $$\sigma_{i,t+1} = \sigma_{i,t} - \eta_t\,\mathrm{sign}(\sigma_{i,t}^3 - \lambda_i^*\sigma_{i,t})$$ where $U_t = U^*\Sigma_t O^\top$ and $\lambda_i^*$ are the canonical eigenvalues. Each mode converges linearly at a rate independent of the condition number, a phenomenon not available to gradient descent (GD) or sign-based first-order methods, whose convergence slows proportionally with ill-conditioning (Ma et al., 20 Jan 2026).

In the context of inverting structured Gram matrices (e.g., transformers), Turbo-Muon can be interpreted as preconditioning by an inverse square root of the gradient’s local Gram matrix, enforcing uniform step sizes across singular modes and neutralizing spectral disparities.

4. Efficiency Gains and Empirical Impact

Empirical evaluation demonstrates that Turbo-Muon achieves:

• 2.8× speedup in the Newton-Schulz orthogonalization step compared to baseline Muon.
• 5–10% reduction in full training time in large-scale settings, such as a 1.3B-parameter LLM on a single A100 GPU, with the orthogonalization subroutine’s share of total step time dropping from 10–20% to approximately 10% due to accelerated NS convergence.
• On benchmark tasks, such as NanoGPT-124M speedruns (FineWeb, 8×H100) and CIFAR-10 CNNs (single A100), the wall-clock runtime decreases with no statistical difference in validation loss or accuracy between Muon+ and Turbo-Muon (Boissin et al., 4 Dec 2025).

A summary of key workflow and timing differences is given below:

Optimizer	NS Iterations	Total Multiplies	Orthogonalization Speedup	Net Step-Time Change
Muon (Frobenius)	5	15	1×	Baseline
Muon+ (Triton opt.)	5	15	2.2×	~7% faster
Turbo-Muon (AOL+NS4)	4 (+1 AOL)	13	2.8×	~10% faster

5. Integration and Implementation

Turbo-Muon is designed as a drop-in replacement for existing Muon orthogonalization steps. No hyperparameter modification is required; practitioners replace the canonical Muon kernel: $$\textrm{muon\_ns}(X, \mathrm{iters}=5) \rightarrow \textrm{turbo\_muon\_ns}(X, \mathrm{iters}=4)$$ Triton-accelerated implementations are provided open-source. The cost of the AOL preconditioner is effectively amortized since the Gram matrix multiplication it requires overlaps with existing computation in the Newton-Schulz pipeline (Boissin et al., 4 Dec 2025).

6. Theoretical Role of Preconditioning and Spectral Orthogonalization

The effectiveness of the AOL preconditioner and spectral orthogonalization in Turbo-Muon is underpinned by the following phenomena:

• Condition-number-independent convergence: The spectral decomposition of the problem decouples the optimization into scalar updates, each unaffected by the global condition number. Both in matrix factorization and quadratic transformer inversion, the iteration complexity of Turbo-Muon shows no $\kappa$ dependence at leading order, contrasting sharply with GD and Adam-based methods (Ma et al., 20 Jan 2026).
• Optimal spectral alignment: The preconditioning acts as a square-root inverse of the local Gram matrix of the gradient, distributing curvature evenly and ensuring balanced progress along all spectral directions.
• Geometric balancing: The update direction becomes the unique closest orthogonal matrix to the gradient in Frobenius norm, formalized as a matrix sign projection:

$$\textrm{msign}(G) = P Q^\top = \arg\min_{X^\top X=I} \|X-G\|_F$$

where $G = P \Sigma Q^\top$ is the SVD.

7. Extensions, Relationships, and Future Directions

Turbo-Muon’s mechanism is complementary to other advances in GPU-friendly orthogonalization such as the Polar Express algorithm, which provides optimal spectral polynomial iterations for the polar step and additional stability under reduced precision (e.g., bfloat16). Polar Express can be further “turbo-charged” via adaptive polynomial degree, spectral-gap initialization, and dynamic spectrum estimation to achieve still higher efficiency for future hardware (Amsel et al., 22 May 2025).

The AOL preconditioning can be interpreted as an instance of general spectral preconditioning strategies, offering a template for improved convergence in any setting where orthogonality-based updates are bottlenecked by the cost of polar computations.

Finally, Turbo-Muon’s empirical demonstration of orthogonalization as a practical, scalable preconditioner closes the gap between theoretical proposals for curvature-insensitive optimization and real-world training dynamics in deep models, establishing a new baseline for orthogonality-based large-scale optimization (Boissin et al., 4 Dec 2025, Ma et al., 20 Jan 2026).