---
title: Decentralized Normalized SGD (DNSGD)
url: https://www.emergentmind.com/topics/decentralized-normalized-stochastic-gradient-descent-dnsgd
type: topic
---

# Decentralized Normalized SGD (DNSGD)

Decentralized Normalized Stochastic Gradient Descent (DNSGD) denotes a family of decentralized first-order methods for stochastic nonconvex optimization in which each agent normalizes a local gradient or gradient-tracking estimate before performing network mixing and descent. In the recent literature, DNSGD appears in closely related forms tailored to heavy-tailed gradient noise, row-stochastic directed networks, and relaxed-smooth objectives. Across these formulations, the common design principle is to combine normalization with consensus and, in several variants, gradient tracking so that local updates remain stable even when raw stochastic gradients are large or poorly behaved [2505.03736], [2509.08726], [2601.11435].

## 1. Problem class and formal setting

DNSGD is studied in decentralized stochastic optimization problems of the form
\[
\min_{x\in\mathbb R^d} f(x)=\frac1n\sum_{i=1}^n f_i(x)
\]
or, equivalently in the notation of another paper,
\[
\min_{x\in\mathbb R^d} f(x)=\frac1m\sum_{i=1}^m f_i(x),
\qquad
f_i(x)=\mathbb E_{\xi_i\sim\mathcal D_i}[F_i(x;\xi_i)].
\]
Each agent holds a local objective and communicates only with graph neighbors [2505.03736], [2509.08726], [2601.11435].

The baseline smooth setting assumes that each local function is \(L\)-smooth and that \(f(x)\ge f^*>-\infty\). In the relaxed-smooth setting, each \(f_i\) satisfies \((L_0,L_1)\)-smoothness:
\[
\|\nabla f_i(x)-\nabla f_i(y)\|
\le
\bigl(L_0+L_1\|\nabla f_i(x)\|\bigr)\|x-y\|,
\]
for \(\|x-y\|\le 1/L_1\). Under an additional bounded heterogeneity condition
\[
\|\nabla f_i(x)-\nabla f(x)\|\le \zeta,
\]
the global objective \(f\) is itself \((L_f,L_1)\)-smooth with \(L_f=L_0+L_1\zeta\) [2509.08726].

The stochastic oracle assumptions differ across the DNSGD variants. In the heavy-tailed formulations, each node accesses an unbiased stochastic gradient with only a bounded \(p\)-th moment,
\[
\mathbb E[\nabla F_i(x;\xi)\mid x]=\nabla f_i(x),
\qquad
\mathbb E[\|\nabla F_i(x;\xi)-\nabla f_i(x)\|^p]\le \sigma^p,
\]
for \(p\in(1,2]\). The case \(p=2\) recovers bounded variance, while \(p<2\) covers heavy-tailed gradients with only a bounded \((1+\alpha)\) moment, where \(\alpha=p-1\) [2601.11435]. In the relaxed-smooth DNSGD paper, the oracle satisfies the standard bounded-variance condition
\[
\mathbb E_{\xi_i}[\nabla F_i(x;\xi_i)]=\nabla f_i(x),
\qquad
\mathbb E[\|\nabla F_i(x;\xi_i)-\nabla f_i(x)\|^2]\le \sigma^2
\]
[2509.08726].

This places DNSGD at the intersection of decentralized optimization, stochastic nonconvex analysis, and robustness to unfavorable gradient noise. A plausible implication is that the term “DNSGD” is best understood as a methodological family rather than a single fixed recursion.

## 2. Network models and the role of normalization

The literature distinguishes between doubly stochastic undirected networks and row-stochastic directed networks. In the doubly stochastic case, communication is governed by a primitive weight matrix \(W\) or \(A\) satisfying \(W\mathbf 1=\mathbf 1\) and \(\mathbf 1^\top W=\mathbf 1^\top\), with spectral gap characterized by
\[
\lambda=\|W-\tfrac1n\mathbf1\mathbf1^\top\|_2<1
\]
or equivalently \(\gamma=1-\lambda_2(W)\) [2505.03736], [2509.08726]. In the directed case, the mixing matrix is row-stochastic,
\[
A\mathbf1=\mathbf1,
\qquad
\pi^\top A=\pi^\top,
\]
with Perron vector \(\pi>0\) and generalized spectral gap \(1-\beta\) defined through
\[
\beta=\big\|\Pi^{1/2}A\Pi^{-1/2}-\Pi^{-1/2}\mathbf1\pi^\top\Pi^{1/2}\big\|_2,
\qquad
\Pi=\operatorname{diag}(\pi)
\]
[2601.11435].

Normalization is the central mechanism that differentiates DNSGD from conventional decentralized SGD. In GT-NSGDm, the descent step uses
\[
x_i^{t+1}\leftarrow \sum_r w_{ir}\Bigl(x_r^t-\alpha\, y_r^t/\|y_r^t\|\Bigr),
\]
and the rationale is stated explicitly: under heavy-tailed noise the raw stochastic gradient may occasionally blow up, while dividing by its norm stabilizes the step length so that no single gradient sample can dominate [2505.03736]. In the relaxed-smooth DNSGD method, the normalized direction is
\[
u_i^t=\frac{v_i^t}{\|v_i^t\|}\quad\text{if }v_i^t\neq 0,
\]
which prevents an unbounded descent step when local gradients become large [2509.08726]. In DNSGD-PD for row-stochastic networks, the method normalizes the local mini-batch gradient before mixing,
\[
u_{t,i}=\frac{g_{t,i}}{\|g_{t,i}\|+\epsilon},
\]
with \(\epsilon>0\) small [2601.11435].

A common misconception is to treat normalization as interchangeable with clipping. The heavy-tailed decentralized analysis distinguishes the two: DSGD-Clip uses \(\ell_2\)-clipping per step, whereas normalized methods are described as parameter-free beyond the usual optimization hyperparameters and as not requiring tuning of a clipping threshold [2505.03736]. The relaxed-smooth DNSGD paper similarly contrasts normalization with gradient clipping and notes that DNSGD avoids the inconsistency clipping can cause among neighbors [2509.08726].

## 3. Core algorithmic architectures

The simplest DNSGD template consists of four elements: a local stochastic gradient estimate, normalization, consensus or gossip, and an auxiliary mechanism to reduce bias from heterogeneity. The exact realization depends on the network model.

In GT-NSGDm, each node maintains a momentum-averaged estimator \(v_i^t\), a gradient-tracking variable \(y_i^t\), and the local model \(x_i^t\). The update is
1. sample a noisy gradient \(g_i^t\),
2. update momentum \(v_i^t\leftarrow \beta v_i^{t-1}+(1-\beta)g_i^t\),
3. perform gradient tracking
\[
y_i^t\leftarrow \sum_r w_{ir}\bigl[y_r^{t-1}+(v_r^t-v_r^{t-1})\bigr],
\]
4. apply normalized descent and mixing
\[
x_i^{t+1}\leftarrow \sum_r w_{ir}\Bigl(x_r^t-\alpha\, y_r^t/\|y_r^t\|\Bigr)
\]
[2505.03736].

In the relaxed-smooth DNSGD method, each node maintains \(x_i^t\) and a local tracking estimate \(v_i^t\approx \nabla f_i(x_i^t)\). Each iteration forms the normalized direction \(u_i^t=v_i^t/\|v_i^t\|\), applies \(K\) rounds of Chebyshev-accelerated averaging to \(X^t-\eta U^t\), resamples a mini-batch stochastic gradient \(g_i^{t+1}\), and updates the tracking variable via one consensus step on \(V^t+G^{t+1}-G^t\) [2509.08726]. The use of multi-consensus is integral rather than incidental: the paper specifies \(K\approx (\ln m)/\sqrt\gamma\) to obtain exponentially small spectral error \(\rho\approx e^{-c\sqrt\gamma K}\).

The row-stochastic directed variant DNSGD-PD incorporates Pull-Diag preconditioning. Each node maintains
\[
x_{t,i}\in\mathbb R^d,\qquad v_{t,i}\in\mathbb R^d,\qquad
g_{t,i}=\frac1b\sum_{k=1}^b \nabla F_i(x_{t,i};\xi_{t,i,k}).
\]
Consensus on \(x\) is carried out by
\[
x_{t,i}\leftarrow \texttt{PullDiag}(x_{t,i}-\eta u_{t,i},A,K),
\]
and gradient tracking uses
\[
v_{t+1,i}
=\texttt{PullDiag}\Bigl(v_{t,i}
+\bigl[g_{t+1,i}/D_{t+1,ii}\bigr]
-\bigl[g_{t,i}/D_{t,ii}\bigr],A,K\Bigr),
\]
where \(D_t=\operatorname{diag}(A^{(t+1)K})\) stores the diagonal entries of the consensus matrix [2601.11435]. The paper states that, in addition to vanilla Push-Sum, the method uses a “Pull-Diag” preconditioning to correct for row-stochasticity.

Across these variants, gradient tracking addresses a specific failure mode of naive normalization. Without tracking, local normalization can distort the average descent direction when \(\|\nabla f_i\|\) varies significantly across nodes. The heavy-tailed GT-NSGDm exposition states this explicitly: heterogeneity in \(\|\nabla f_i\|\) can “push” the average away from a stationary point, whereas tracking enforces that \(y_i^t\) approaches the network-average gradient [2505.03736].

## 4. Convergence theory under heavy-tailed noise

The heavy-tailed DNSGD literature focuses on the expected gradient norm as the stationarity criterion. In GT-NSGDm, when the tail index \(p\) is known and the communication graph admits primitive and doubly stochastic weights, the method uses
\[
1-\beta=T^{-p/(3p-2)},
\qquad
\alpha=\Theta(T^{-1/2}),
\]
and guarantees
\[
\frac1{nT}\sum_{t=0}^{T-1}\sum_{i=1}^n \mathbb E[\|\nabla f(x_i^t)\|]
=
O\Bigl(\frac1{T^{(p-1)/(3p-2)}}\Bigr).
\]
The paper states that this is the first result in the literature guaranteeing convergence of the expected gradient norm at the optimal non-asymptotic rate under decentralized heavy-tailed nonconvex optimization, and that the rate matches the centralized lower bound for nonconvex heavy-tailed SGD [2505.03736].

When \(p\) is unknown, GT-NSGDm fixes \(\beta=1-1/\sqrt T\) and retains \(\alpha\) as before, yielding
\[
\frac1{nT}\sum \mathbb E[\|\nabla f(x_i^t)\|]
=
O\bigl(T^{-(p-1)/(2p)}\bigr).
\]
For \(p<2\), this rate is stated to be topology independent and to admit a speedup factor \(n^{1-1/p}\) in the number of nodes [2505.03736].

The directed-network DNSGD-PD result is expressed in \(\epsilon\)-stationarity form. Under the row-stochastic assumptions, with
\[
T=O(L\Delta/\epsilon^2),\qquad
\eta=\Theta(\epsilon/L),\qquad
b=\Theta\bigl((n^{-1}\sigma/\epsilon)^{p/(p-1)}\bigr),\qquad
K,\hat K=\tilde\Theta\bigl((1-\beta)^{-1}\bigr),
\]
the output \(\hat x_i\), chosen uniformly from \(\{x_{0,i},\dots,x_{T-1,i}\}\), satisfies
\[
\mathbb E\|\nabla f(\hat x_i)\|\le \epsilon
\]
for every \(i\). The total sample complexity is
\[
nTb
=
O\bigl(L\sigma^{p/(p-1)}\Delta\,\epsilon^{-(3p-2)/(p-1)}\bigr),
\]
and communication rounds satisfy
\[
T\times K=\tilde O\Bigl(\frac{L\Delta}{(1-\beta)\epsilon^2}\Bigr)
\]
[2601.11435].

The same paper states that matching lower bounds show the sample rate \(O(\epsilon^{-(3p-2)/(p-1)})\) is optimal under \(p\)-BCM noise, while the communication rate \(\tilde O((1-\beta)^{-1}\epsilon^{-2})\) is near-optimal. For undirected networks, replacing Pull-Diag with standard double stochastic gossip and taking \(D_t=I\) yields exactly the DNSGD method, with the same sample complexity
\[
O\bigl(L\sigma^{p/(p-1)}\Delta\,\epsilon^{-(3p-2)/(p-1)}\bigr)
\]
and communication
\[
\tilde O\bigl((1-\beta)^{-1/2}\epsilon^{-2}\bigr),
\]
again described as nearly tight [2601.11435].

The proof architecture also has a shared pattern. In the row-stochastic setting, the analysis introduces the weighted average
\[
\bar x_t=\sum_i \pi_i x_{t,i}
\]
and a Lyapunov function combining \(f(\bar x_t)\) with consensus and tracking errors. Normalization ensures bounded update directions, while a moment inequality under heavy tails yields
\[
\mathbb E\|g_{t,i}-\nabla f_i(x_{t,i})\|
\le
O\bigl(\sigma/b^{1-1/p}\bigr)
\]
[2601.11435]. In GT-NSGDm, the analysis instead decomposes descent into contributions from gradient-tracking consensus error, momentum estimation error, and normalization bias, and handles the heavy-tailed component via a vector-Martingale-difference bound [2505.03736].

## 5. DNSGD under relaxed smoothness

The relaxed-smooth DNSGD formulation extends normalization-based decentralized optimization beyond globally Lipschitz gradients. Its key analytical contribution is a Lyapunov function that couples optimality and consensus error:
\[
\Phi^t
=
f(\bar x^t)
+
\frac{3\eta}{\sqrt m\,(M_0+M_1\|\nabla f(\bar x^t)\|)}\|\Delta X^t\|
+
\frac{2\eta}{\sqrt m}\|\Delta V^t\|,
\]
where
\[
\bar x^t=\tfrac1m\sum_i x_i^t,\quad
\Delta X^t=X^t-\mathbf1(\bar x^t)^\top,\quad
\Delta V^t=V^t-\mathbf1(\bar x^t)^\top,
\]
and
\[
M_0=\sqrt{2(L_0^2+L_1^2\zeta^2)},\qquad
M_1=\sqrt2\,L_1,\qquad
L_f=L_0+L_1\zeta
\]
[2509.08726].

The paper emphasizes that classical descent proofs break because \((L_0,L_1)\)-smoothness allows unbounded gradients. Under appropriate bounds on \(\eta\) and \(b\), its descent lemma gives
\[
\mathbb E[f(\bar x^{t+1})]
\le
f(\bar x^t)
-\eta\Bigl(1-\tfrac{\eta L_1}{2}\Bigr)\|\nabla f(\bar x^t)\|
+\frac{2\sqrt2\,\eta}{\sqrt m\,(M_0+M_1\|\nabla f(\bar x^t)\|)}\|\Delta X^t\|
+\frac{\eta}{\sqrt m}\|\Delta V^t\|
+\frac{\eta^2L_f}{2}.
\]
Consensus and tracking are then controlled using bounds of the form
\[
\|\Delta X^{t+1}\|\le \rho(\|\Delta X^t\|+m\eta),
\qquad
\|\Delta V^{t+1}\|
\le
\rho\Bigl(\|\Delta V^t\|+\cdots+\frac{2\sqrt m\,\sigma}{\sqrt b}\Bigr),
\]
with \(\rho\approx \exp(-c\sqrt\gamma K)\) when \(K\gtrsim (\log m)/\sqrt\gamma\) [2509.08726].

Its main theorem states that if
\[
\eta=\min\Bigl\{\frac{\epsilon}{4L_f+1},\,\frac1{2L_1}\Bigr\},
\]
\[
b\ge \max\Bigl\{\frac{256(4L_f+1)^2\sigma^2}{mL_f^2\epsilon^2},\,
\frac{1024L_1^2\sigma^2}{mL_f^2}\Bigr\},
\]
\[
T\ge \max\Bigl\{8(4L_f+1)\Delta_\Phi \epsilon^{-2},\,
16L_1\Delta_\Phi \epsilon^{-1}\Bigr\},
\qquad
K\gtrsim \frac{\ln m}{\sqrt\gamma},
\]
then a random average iterate \(\hat x_i\) satisfies
\[
\mathbb E[\|\nabla f(\hat x_i)\|]\le \epsilon/2
\]
[2509.08726].

With sufficiently many initial consensus rounds \(\hat K\gtrsim 1/\sqrt\gamma\), each agent achieves an \(\epsilon\)-stationary point with sample complexity per agent
\[
Tb
=
\frac1m\Bigl(
\frac{L_f\sigma^2\Delta_f}{\epsilon^4}
+
\frac{\sigma^2}{\epsilon^2}
+
\frac{L_1^3\sigma^2\Delta_f}{L_f^2\epsilon}
+
\frac{L_1^2\sigma^2}{L_f^2}
\Bigr),
\]
and total communication rounds
\[
TK+\hat K
=
\widetilde O\Bigl(
\frac{L_f\Delta_f}{\sqrt\gamma\,\epsilon^2}
+
\frac{L_1\Delta_f}{\sqrt\gamma\,\epsilon}
\Bigr).
\]
When \(L_1=0\), these reduce to
\[
Tb
=
\frac1m\Bigl(
\frac{L_0\sigma^2\Delta_f}{\epsilon^4}
+
\frac{\sigma^2}{\epsilon^2}
\Bigr),
\qquad
TK+\hat K
=
\widetilde O\Bigl(\frac{L_0\Delta_f}{\sqrt\gamma\,\epsilon^2}\Bigr),
\]
which the paper states matches known lower bounds up to logarithmic factors [2509.08726].

## 6. Empirical behavior, baselines, and significance

The empirical studies position DNSGD-style methods against both conventional decentralized SGD and more specialized robust baselines. In the heavy-tailed decentralized setting, GT-NSGDm is compared with DSGD, GT-DSGD, DSGD-Clip, and SClip-EF-Network. The reported qualitative picture is that DSGD and GT-DSGD explode or stall when variance is infinite, DSGD-Clip stabilizes but converges slowly, SClip-EF-Network is robust but slower on large models, and GT-NSGDm achieves the optimal exponent and strong \(n\)-speedup [2505.03736].

The same paper reports two experiments. In nonconvex robust linear regression on synthetic tokenized data with \(1\,000\) samples, \(d=20\), Tukey’s biweight loss, and \(n=20\) nodes, gradient noise is injected as Gaussian, Student-\(t\) with \(\mathrm{df}=1.5\), and Lévy \(\alpha\)-stable with \(\alpha=1.5\). The metric is average estimation error \((1/n)\sum\|w_i^t-w_*\|\) versus iterations, and GT-NSGDm together with SClip-EF-Network remains stable under all noise types while other methods diverge or fail to converge to optimum. In decentralized Transformer training on Multi30k, using a decoder-only GPT with approximately \(3\) million parameters, \(n=8\) nodes, batch size \(64\), and \(12\) epochs, the graph-wide average validation perplexity and loss show that only GT-NSGDm converges to near-zero loss, approximately \(0.26\), while all other baselines plateau above \(5\) [2505.03736].

The DNSGD-PD paper evaluates a 6-layer Transformer-XL language model on Penn Treebank under heavy-tailed gradient noise. On directed ring and directed exponential graphs, DNSGD-PD is compared to MG-GT, and it is reported to converge faster in both sample and communication counts; increasing consensus rounds \(K\) improves performance until network mixing is sufficient; and the method shows near-linear speedup in \(n\). On undirected ring and Erdős–Rényi graphs, DNSGD is compared with DSGT and GT-NSGDm, and it is reported to outperform both baselines under heavy-tailed noise, again with near-linear scaling and optimal dependence on \(\epsilon\) [2601.11435].

The relaxed-smooth DNSGD paper studies MNIST image classification with a small 3-layer convolutional network over \(m=10\) agents connected by an Erdős–Rényi graph with edge probability \(p=0.6\). Using batch size \(256\), tuned step sizes, and Chebyshev rounds \(K=\hat K=2\), DNSGD achieves lower average gradient norm versus number of samples and communications, and higher test accuracy for a given budget, compared with D-SGD, D-SGT, and D-NASA [2509.08726].

Taken together, these results suggest a consistent interpretation of DNSGD’s significance. Under heavy-tailed noise, normalization is used to control extreme stochastic gradients while preserving decentralized scalability; under relaxed smoothness, the same normalization mechanism stabilizes descent when gradient magnitudes can become large. In both regimes, the addition of gradient tracking and multi-step consensus is not merely an implementation detail but a structural device for recovering stationary-point guarantees on each local agent.

Source: https://www.emergentmind.com/topics/decentralized-normalized-stochastic-gradient-descent-dnsgd