Decentralized Normalized SGD (DNSGD)

• DNSGD is a decentralized optimization framework that normalizes local gradients before mixing to stabilize descent even under heavy-tailed noise.
• It incorporates network consensus and gradient tracking to mitigate local heterogeneity and ensure a reliable average descent direction across agents.
• DNSGD variants address both undirected and directed network models, providing optimal convergence guarantees in challenging nonconvex settings.

Decentralized Normalized Stochastic Gradient Descent (DNSGD) denotes a family of decentralized first-order methods for stochastic nonconvex optimization in which each agent normalizes a local gradient or gradient-tracking estimate before performing network mixing and descent. In the recent literature, DNSGD appears in closely related forms tailored to heavy-tailed gradient noise, row-stochastic directed networks, and relaxed-smooth objectives. Across these formulations, the common design principle is to combine normalization with consensus and, in several variants, gradient tracking so that local updates remain stable even when raw stochastic gradients are large or poorly behaved (Yu et al., 6 May 2025, Luo et al., 10 Sep 2025, Wang et al., 16 Jan 2026).

1. Problem class and formal setting

DNSGD is studied in decentralized stochastic optimization problems of the form

$$\min_{x\in\mathbb R^d} f(x)=\frac1n\sum_{i=1}^n f_i(x)$$

or, equivalently in the notation of another paper,

$$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$

Each agent holds a local objective and communicates only with graph neighbors (Yu et al., 6 May 2025, Luo et al., 10 Sep 2025, Wang et al., 16 Jan 2026).

The baseline smooth setting assumes that each local function is $L$-smooth and that $f(x)\ge f^*>-\infty$. In the relaxed-smooth setting, each $f_i$ satisfies $(L_0,L_1)$-smoothness: $$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$ for $\|x-y\|\le 1/L_1$. Under an additional bounded heterogeneity condition

$\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$

the global objective $f$ is itself $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$0-smooth with $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$1 (Luo et al., 10 Sep 2025).

The stochastic oracle assumptions differ across the DNSGD variants. In the heavy-tailed formulations, each node accesses an unbiased stochastic gradient with only a bounded $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$2-th moment,

$$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$3

for $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$4. The case $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$5 recovers bounded variance, while $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$6 covers heavy-tailed gradients with only a bounded $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$7 moment, where $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$8 (Wang et al., 16 Jan 2026). In the relaxed-smooth DNSGD paper, the oracle satisfies the standard bounded-variance condition

$$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$9

(Luo et al., 10 Sep 2025).

This places DNSGD at the intersection of decentralized optimization, stochastic nonconvex analysis, and robustness to unfavorable gradient noise. A plausible implication is that the term “DNSGD” is best understood as a methodological family rather than a single fixed recursion.

2. Network models and the role of normalization

The literature distinguishes between doubly stochastic undirected networks and row-stochastic directed networks. In the doubly stochastic case, communication is governed by a primitive weight matrix $L$0 or $L$1 satisfying $L$2 and $L$3, with spectral gap characterized by

$L$4

or equivalently $L$5 (Yu et al., 6 May 2025, Luo et al., 10 Sep 2025). In the directed case, the mixing matrix is row-stochastic,

$L$6

with Perron vector $L$7 and generalized spectral gap $L$8 defined through

$L$9

(Wang et al., 16 Jan 2026).

Normalization is the central mechanism that differentiates DNSGD from conventional decentralized SGD. In GT-NSGDm, the descent step uses

$f(x)\ge f^*>-\infty$0

and the rationale is stated explicitly: under heavy-tailed noise the raw stochastic gradient may occasionally blow up, while dividing by its norm stabilizes the step length so that no single gradient sample can dominate (Yu et al., 6 May 2025). In the relaxed-smooth DNSGD method, the normalized direction is

$f(x)\ge f^*>-\infty$1

which prevents an unbounded descent step when local gradients become large (Luo et al., 10 Sep 2025). In DNSGD-PD for row-stochastic networks, the method normalizes the local mini-batch gradient before mixing,

$f(x)\ge f^*>-\infty$2

with $f(x)\ge f^*>-\infty$3 small (Wang et al., 16 Jan 2026).

A common misconception is to treat normalization as interchangeable with clipping. The heavy-tailed decentralized analysis distinguishes the two: DSGD-Clip uses $f(x)\ge f^*>-\infty$4-clipping per step, whereas normalized methods are described as parameter-free beyond the usual optimization hyperparameters and as not requiring tuning of a clipping threshold (Yu et al., 6 May 2025). The relaxed-smooth DNSGD paper similarly contrasts normalization with gradient clipping and notes that DNSGD avoids the inconsistency clipping can cause among neighbors (Luo et al., 10 Sep 2025).

3. Core algorithmic architectures

The simplest DNSGD template consists of four elements: a local stochastic gradient estimate, normalization, consensus or gossip, and an auxiliary mechanism to reduce bias from heterogeneity. The exact realization depends on the network model.

In GT-NSGDm, each node maintains a momentum-averaged estimator $f(x)\ge f^*>-\infty$5, a gradient-tracking variable $f(x)\ge f^*>-\infty$6, and the local model $f(x)\ge f^*>-\infty$7. The update is

1. sample a noisy gradient $f(x)\ge f^*>-\infty$8,
2. update momentum $f(x)\ge f^*>-\infty$9,
3. perform gradient tracking

$f_i$0

1. apply normalized descent and mixing

$f_i$1

(Yu et al., 6 May 2025).

In the relaxed-smooth DNSGD method, each node maintains $f_i$2 and a local tracking estimate $f_i$3. Each iteration forms the normalized direction $f_i$4, applies $f_i$5 rounds of Chebyshev-accelerated averaging to $f_i$6, resamples a mini-batch stochastic gradient $f_i$7, and updates the tracking variable via one consensus step on $f_i$8 (Luo et al., 10 Sep 2025). The use of multi-consensus is integral rather than incidental: the paper specifies $f_i$9 to obtain exponentially small spectral error $(L_0,L_1)$0.

The row-stochastic directed variant DNSGD-PD incorporates Pull-Diag preconditioning. Each node maintains

$(L_0,L_1)$1

Consensus on $(L_0,L_1)$2 is carried out by

$(L_0,L_1)$3

and gradient tracking uses

$(L_0,L_1)$4

where $(L_0,L_1)$5 stores the diagonal entries of the consensus matrix (Wang et al., 16 Jan 2026). The paper states that, in addition to vanilla Push-Sum, the method uses a “Pull-Diag” preconditioning to correct for row-stochasticity.

Across these variants, gradient tracking addresses a specific failure mode of naive normalization. Without tracking, local normalization can distort the average descent direction when $(L_0,L_1)$6 varies significantly across nodes. The heavy-tailed GT-NSGDm exposition states this explicitly: heterogeneity in $(L_0,L_1)$7 can “push” the average away from a stationary point, whereas tracking enforces that $(L_0,L_1)$8 approaches the network-average gradient (Yu et al., 6 May 2025).

4. Convergence theory under heavy-tailed noise

The heavy-tailed DNSGD literature focuses on the expected gradient norm as the stationarity criterion. In GT-NSGDm, when the tail index $(L_0,L_1)$9 is known and the communication graph admits primitive and doubly stochastic weights, the method uses

$$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$0

and guarantees

$$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$1

The paper states that this is the first result in the literature guaranteeing convergence of the expected gradient norm at the optimal non-asymptotic rate under decentralized heavy-tailed nonconvex optimization, and that the rate matches the centralized lower bound for nonconvex heavy-tailed SGD (Yu et al., 6 May 2025).

When $$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$2 is unknown, GT-NSGDm fixes $$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$3 and retains $$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$4 as before, yielding

$$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$5

For $$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$6, this rate is stated to be topology independent and to admit a speedup factor $$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$7 in the number of nodes (Yu et al., 6 May 2025).

The directed-network DNSGD-PD result is expressed in $$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$8-stationarity form. Under the row-stochastic assumptions, with

$$\|\nabla f_i(x)-\nabla f_i(y)\| \le \bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,$$9

the output $\|x-y\|\le 1/L_1$0, chosen uniformly from $\|x-y\|\le 1/L_1$1, satisfies

$\|x-y\|\le 1/L_1$2

for every $\|x-y\|\le 1/L_1$3. The total sample complexity is

$\|x-y\|\le 1/L_1$4

and communication rounds satisfy

$\|x-y\|\le 1/L_1$5

(Wang et al., 16 Jan 2026).

The same paper states that matching lower bounds show the sample rate $\|x-y\|\le 1/L_1$6 is optimal under $\|x-y\|\le 1/L_1$7-BCM noise, while the communication rate $\|x-y\|\le 1/L_1$8 is near-optimal. For undirected networks, replacing Pull-Diag with standard double stochastic gossip and taking $\|x-y\|\le 1/L_1$9 yields exactly the DNSGD method, with the same sample complexity

$\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$0

and communication

$\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$1

again described as nearly tight (Wang et al., 16 Jan 2026).

The proof architecture also has a shared pattern. In the row-stochastic setting, the analysis introduces the weighted average

$\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$2

and a Lyapunov function combining $\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$3 with consensus and tracking errors. Normalization ensures bounded update directions, while a moment inequality under heavy tails yields

$\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$4

(Wang et al., 16 Jan 2026). In GT-NSGDm, the analysis instead decomposes descent into contributions from gradient-tracking consensus error, momentum estimation error, and normalization bias, and handles the heavy-tailed component via a vector-Martingale-difference bound (Yu et al., 6 May 2025).

5. DNSGD under relaxed smoothness

The relaxed-smooth DNSGD formulation extends normalization-based decentralized optimization beyond globally Lipschitz gradients. Its key analytical contribution is a Lyapunov function that couples optimality and consensus error: $\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$5 where

$\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$6

and

$\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$7

(Luo et al., 10 Sep 2025).

The paper emphasizes that classical descent proofs break because $\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$8-smoothness allows unbounded gradients. Under appropriate bounds on $\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,$9 and $f$0, its descent lemma gives

$f$1

Consensus and tracking are then controlled using bounds of the form

$f$2

with $f$3 when $f$4 (Luo et al., 10 Sep 2025).

Its main theorem states that if

$f$5

$f$6

$f$7

then a random average iterate $f$8 satisfies

$f$9

(Luo et al., 10 Sep 2025).

With sufficiently many initial consensus rounds $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$00, each agent achieves an $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$01-stationary point with sample complexity per agent

$$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$02

and total communication rounds

$$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$03

When $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$04, these reduce to

$$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$05

which the paper states matches known lower bounds up to logarithmic factors (Luo et al., 10 Sep 2025).

6. Empirical behavior, baselines, and significance

The empirical studies position DNSGD-style methods against both conventional decentralized SGD and more specialized robust baselines. In the heavy-tailed decentralized setting, GT-NSGDm is compared with DSGD, GT-DSGD, DSGD-Clip, and SClip-EF-Network. The reported qualitative picture is that DSGD and GT-DSGD explode or stall when variance is infinite, DSGD-Clip stabilizes but converges slowly, SClip-EF-Network is robust but slower on large models, and GT-NSGDm achieves the optimal exponent and strong $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$06-speedup (Yu et al., 6 May 2025).

The same paper reports two experiments. In nonconvex robust linear regression on synthetic tokenized data with $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$07 samples, $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$08, Tukey’s biweight loss, and $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$09 nodes, gradient noise is injected as Gaussian, Student-$$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$10 with $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$11, and Lévy $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$12-stable with $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$13. The metric is average estimation error $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$14 versus iterations, and GT-NSGDm together with SClip-EF-Network remains stable under all noise types while other methods diverge or fail to converge to optimum. In decentralized Transformer training on Multi30k, using a decoder-only GPT with approximately $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$15 million parameters, $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$16 nodes, batch size $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$17, and $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$18 epochs, the graph-wide average validation perplexity and loss show that only GT-NSGDm converges to near-zero loss, approximately $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$19, while all other baselines plateau above $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$20 (Yu et al., 6 May 2025).

The DNSGD-PD paper evaluates a 6-layer Transformer-XL LLM on Penn Treebank under heavy-tailed gradient noise. On directed ring and directed exponential graphs, DNSGD-PD is compared to MG-GT, and it is reported to converge faster in both sample and communication counts; increasing consensus rounds $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$21 improves performance until network mixing is sufficient; and the method shows near-linear speedup in $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$22. On undirected ring and Erdős–Rényi graphs, DNSGD is compared with DSGT and GT-NSGDm, and it is reported to outperform both baselines under heavy-tailed noise, again with near-linear scaling and optimal dependence on $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$23 (Wang et al., 16 Jan 2026).

The relaxed-smooth DNSGD paper studies MNIST image classification with a small 3-layer convolutional network over $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$24 agents connected by an Erdős–Rényi graph with edge probability $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$25. Using batch size $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$26, tuned step sizes, and Chebyshev rounds $$\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x), \qquad f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].$$27, DNSGD achieves lower average gradient norm versus number of samples and communications, and higher test accuracy for a given budget, compared with D-SGD, D-SGT, and D-NASA (Luo et al., 10 Sep 2025).

Taken together, these results suggest a consistent interpretation of DNSGD’s significance. Under heavy-tailed noise, normalization is used to control extreme stochastic gradients while preserving decentralized scalability; under relaxed smoothness, the same normalization mechanism stabilizes descent when gradient magnitudes can become large. In both regimes, the addition of gradient tracking and multi-step consensus is not merely an implementation detail but a structural device for recovering stationary-point guarantees on each local agent.