DeMuon: Decentralized Matrix Optimization

• DeMuon is a decentralized matrix optimization algorithm that extends the Muon optimizer to peer-to-peer networks while preserving matrix structure.
• It employs gradient tracking, the matrix sign map, and Newton–Schulz orthogonalization to improve consensus and convergence in noisy, heterogeneous environments.
• The algorithm achieves iteration-complexity guarantees comparable to centralized methods, making it a practical choice for large-scale, decentralized optimization.

Searching arXiv for DeMuon and closely related decentralized Muon work. DeMuon is a decentralized matrix optimization algorithm that extends the centralized Muon optimizer to peer-to-peer optimization over a communication graph. It is formulated for finite-sum matrix problems of the form

$$\min_{X\in\mathbb{R}^{m\times n}} f(X):=\frac1N\sum_{i=1}^N f_i(X),$$

where node $i$ holds a local objective $f_i$, communication is performed without a central coordinator, and each node observes noisy stochastic gradients. DeMuon preserves Muon’s matrix-structured update through the matrix sign map and Newton–Schulz orthogonalization, while adding gradient tracking to mitigate heterogeneity among local functions. Under heavy-tailed noise with finite $\alpha$-th moments, it provides an iteration-complexity guarantee for reaching an approximate stochastic stationary point in nuclear norm, with dependence on the target tolerance matching the best-known centralized dependence. It is presented as the first direct extension of Muon to decentralized optimization over graphs with provable complexity guarantees (He et al., 1 Oct 2025).

1. Problem class and motivation

DeMuon is motivated by three observations. First, Muon is matrix-native: rather than vectorizing parameters, it operates directly on matrices and uses orthogonalized updates. Second, most decentralized methods, including DSGD, EXTRA, and gradient-tracking variants, are typically applied after vectorization and therefore do not preserve matrix structure. Third, DeMuon explicitly targets stochastic gradients with only finite $\alpha$-th moments, $\alpha\in(1,2]$, rather than sub-Gaussian noise, placing heavy-tailed stochasticity at the center of the analysis (He et al., 1 Oct 2025).

The communication model is encoded by a mixing matrix $W\in\mathbb{R}^{N\times N}$ satisfying $W\ge 0$, primitiveness, and doubly stochasticity: $$W\mathbf 1_N=\mathbf 1_N,\qquad \mathbf 1_N^\top W=\mathbf 1_N^\top.$$ Its consensus rate is controlled by

$$\lambda := \left\|W-\frac{1}{N}\mathbf 1_N\mathbf 1_N^\top\right\| < 1.$$

This places DeMuon in the standard decentralized optimization setting over graphs, but its update geometry is explicitly matrix-based rather than Euclidean-vector-based (He et al., 1 Oct 2025).

A central design objective is therefore to combine three ingredients that are usually studied separately: Muon-style matrix orthogonalization, decentralized communication over a graph, and gradient tracking for heterogeneous local objectives. This suggests that DeMuon is best understood not as a minor variant of DSGD, but as a graph-based control mechanism wrapped around a matrix-aware update rule.

2. Update structure and local state

Each node $i$0 maintains three matrix-valued states: the parameter matrix $i$1, a momentum or EMA state $i$2, and a gradient-tracking variable $i$3. The algorithm uses the recursions

$i$4

$i$5

$i$6

with $i$7, $i$8, stepsize $i$9, and smoothing parameter $f_i$0 (He et al., 1 Oct 2025).

The matrix sign map is defined through the reduced SVD $f_i$1 by

$f_i$2

In the algorithmic interpretation, this is the Muon-style orthogonalized direction. The paper states that DeMuon inherits Muon’s Newton–Schulz orthogonalization idea, but its analysis works abstractly with $f_i$3 rather than unfolding the internal Newton–Schulz iterations in detail (He et al., 1 Oct 2025).

In stacked notation,

$f_i$4

the updates become

$f_i$5

$f_i$6

$f_i$7

where

$f_i$8

Because $f_i$9 is doubly stochastic,

$\alpha$0

with

$\alpha$1

This decomposition separates local tracking dynamics from the network-average dynamics and makes clear that DeMuon’s descent direction is an average of orthogonalized tracked signals rather than raw gradients (He et al., 1 Oct 2025).

3. Assumptions, geometry, and stationarity

The analysis is carried out under four assumptions. The global objective is lower bounded: $\alpha$2 Each local gradient is $\alpha$3-Lipschitz in matrix form: $\alpha$4 which yields the descent inequality

$\alpha$5

The stochastic gradient oracle is unbiased and heavy-tailed in nuclear norm: $\alpha$6

$\alpha$7

Finally, the network mixing matrix $\alpha$8 is primitive and doubly stochastic (He et al., 1 Oct 2025).

The stationarity notion is adapted to matrix optimization. An $\alpha$9-stationary point is defined by

$\alpha$0

for a uniformly random output index $\alpha$1, where

$\alpha$2

The use of the nuclear norm is consistent with Muon’s matrix geometry and with the identity

$\alpha$3

together with

$\alpha$4

which are the key operator-level properties used in the analysis (He et al., 1 Oct 2025).

A noteworthy network-control result is the uniform consensus bound

$\alpha$5

The paper highlights this as a spectral-norm disagreement bound, rather than the more common Frobenius-norm control. This suggests that DeMuon’s consensus analysis is aligned with the same matrix-operator viewpoint that defines its update rule.

4. Non-asymptotic analysis under heavy-tailed noise

The proof uses a Lyapunov-like potential

$\alpha$6

for suitable $\alpha$7. This combines objective decrease, momentum-estimation error, and tracking error into a single quantity (He et al., 1 Oct 2025).

A central descent inequality is

$\alpha$8

This makes the analytical structure explicit: the desired stationarity term appears with a negative sign, while the perturbations arise from EMA error, tracking error, and smoothness (He et al., 1 Oct 2025).

The gradient-tracking recursion satisfies

$\alpha$9

and the momentum error obeys both first-moment and $\alpha\in(1,2]$0-moment bounds: $\alpha\in(1,2]$1

$\alpha\in(1,2]$2

These recursions quantify how network disagreement, smoothing, and heavy-tailed stochasticity interact (He et al., 1 Oct 2025).

The main complexity theorem chooses

$\alpha\in(1,2]$3

and shows that for $\alpha\in(1,2]$4,

$\alpha\in(1,2]$5

whenever

$\alpha\in(1,2]$6

Equivalently,

$\alpha\in(1,2]$7

The paper emphasizes that this matches the best-known centralized dependence on $\alpha\in(1,2]$8 under heavy-tailed noise. In that sense, DeMuon’s graph-based decentralization does not worsen the tolerance scaling of the leading complexity order (He et al., 1 Oct 2025).

5. Position within decentralized Muon design space

Relative to centralized Muon, DeMuon adds two mechanisms absent from the centralized setting: peer-to-peer consensus through $\alpha\in(1,2]$9 and gradient tracking through $W\in\mathbb{R}^{N\times N}$0. Relative to DSGD, it replaces raw stochastic-gradient steps with momentum-smoothed, tracked, and orthogonalized matrix updates. Relative to DSGD_Clip, it does not rely on scalar clipping as the core response to heavy tails. Relative to conventional gradient-tracking methods, it retains the matrix geometry through $W\in\mathbb{R}^{N\times N}$1 rather than using vectorized Euclidean updates (He et al., 1 Oct 2025).

A subsequent paper, "SUDA-Muon: Structural Design Principles and Boundaries for Fully Decentralized Muon" (Zhang et al., 27 Apr 2026), treats DeMuon as the main reference point for what a fully decentralized Muon method already looks like in the literature. In that terminology, DeMuon is a “closely related gradient-tracking mechanism combined with Muon directions,” operating on general communication graphs without a central server. The later paper argues, however, that DeMuon is structurally different from its SUDA-based design because DeMuon applies gossip-style mixing directly to the primal variables after the Muon update and does not maintain the SUDA dual variable $W\in\mathbb{R}^{N\times N}$2 (Zhang et al., 27 Apr 2026).

That later analysis frames the difference as structural rather than cosmetic. Because the nonlinear matrix-sign operator $W\in\mathbb{R}^{N\times N}$3 does not commute with linear averaging, the ordering of communication, tracking, and polarization becomes a design principle. The SUDA paper identifies two non-modular boundaries relevant to DeMuon’s interpretation: internally, tracking-before-polarization is necessary to avoid non-stationary fixed points under heterogeneous objectives; externally, fully decentralized local-polarize-then-average cannot reproduce federated average-then-polarize and therefore may fail to exhibit linear speedup even on a complete graph with exact averaging (Zhang et al., 27 Apr 2026).

This reinterpretation does not invalidate DeMuon. Rather, it places DeMuon within a broader taxonomy of decentralized Muon methods and suggests that its behavior is constrained by the noncommutativity of matrix polarization and averaging. It should also be distinguished from DMuon, which is a distributed systems layer for exact Muon semantics with owner-based execution and near-AdamW overhead, not a graph-decentralized optimizer (Chen et al., 25 Jun 2026).

6. Empirical behavior, scope, and limitations

The original empirical evaluation of DeMuon is a decentralized transformer-pretraining study using a 3M-parameter GPT model on Multi30k for auto-regressive language modeling. Three graph topologies are considered: a complete graph, a directed exponential graph, and a ring graph. The baselines are DSGD, DSGD_Clip, and GT_NSGDm (He et al., 1 Oct 2025).

The reported pattern is that DeMuon converges rapidly at the beginning and reaches low validation loss in roughly 500 rounds. On directed exponential and ring graphs, it performs close to GT_NSGDm and clearly outperforms DSGD and DSGD_Clip. On the complete graph, it also converges quickly, but shows some later-round instability relative to GT_NSGDm while still outperforming DSGD and DSGD_Clip (He et al., 1 Oct 2025).

The later SUDA-Muon study provides a second empirical reference point. On CIFAR-100 with a 10-node ring and strong heterogeneity ($W\in\mathbb{R}^{N\times N}$4), SUDA--Muon-ED, SUDA--Muon-ATC-GT, and DeMuon perform similarly after 25 epochs: SUDA--Muon-ED achieves $W\in\mathbb{R}^{N\times N}$5, SUDA--Muon-ATC-GT achieves $W\in\mathbb{R}^{N\times N}$6, and DeMuon achieves $W\in\mathbb{R}^{N\times N}$7. In a longer 20-node, 100-epoch non-IID CIFAR-100 run, SUDA--Muon-ED reaches $W\in\mathbb{R}^{N\times N}$8 training loss and $W\in\mathbb{R}^{N\times N}$9 accuracy, while DeMuon reaches $W\ge 0$0 and $W\ge 0$1. On GPT-2 fine-tuning for Wikitext-2 in a mild near-IID regime, the three methods are nearly indistinguishable, with validation perplexities after four epochs all around $W\ge 0$2 and SUDA--Muon-ED slightly best at $W\ge 0$3 (Zhang et al., 27 Apr 2026).

The practical significance claimed for DeMuon is specific. It is the first direct decentralized extension of Muon with theory, it is designed for matrix-valued optimization under graph communication, and it accommodates heavy-tailed stochastic gradients. The stated limitations are also specific: the constant $W\ge 0$4 in the complexity bound is complicated, the schedules $W\ge 0$5 are theory-driven, the experiments are preliminary, and the internal Newton–Schulz implementation is not expanded in the algorithmic description because the analysis is carried out at the level of the abstract sign map $W\ge 0$6 (He et al., 1 Oct 2025).

In subsequent synthesis, DeMuon is characterized as a valid fully decentralized Muon method, but not the final word on decentralized Muon design (Zhang et al., 27 Apr 2026). A plausible implication is that its long-term significance lies less in being a terminal algorithmic form than in establishing the first rigorous bridge between Muon’s matrix-aware geometry and decentralized stochastic optimization over graphs.