Incremental Consensus-Based Distributed SGD

• The paper presents a framework that interleaves multiple consensus steps with stochastic gradient updates to manage the trade-off between inter-agent agreement and global optimization.
• It establishes rigorous theoretical guarantees in both strongly convex and nonconvex settings using Lyapunov methods and consensus error analysis.
• Empirical results on CIFAR-10 demonstrate that increasing the number of consensus steps improves convergence and reduces model spread, despite higher communication costs.

Incremental consensus-based distributed stochastic gradient descent (i-CDSGD) is an algorithmic framework for collaborative deep learning in decentralized networked settings, explicitly designed to balance the trade-off between inter-agent consensus and global optimization. In i-CDSGD, multiple consensus (neighbor-averaging) steps are interleaved with stochastic gradient updates to manage the tension between reaching an agreement among agents and converging to the minimizer of the global empirical risk. The approach offers rigorous guarantees for both strongly convex and nonconvex objective landscapes and provides explicit control over communication complexity and consensus error via a tunable consensus parameter (Jiang et al., 2018).

1. Problem Formulation and Network Setting

The foundational problem addressed by i-CDSGD is decentralized minimization of a global empirical risk function

$F(x) = \frac{1}{N}\sum_{i=1}^N f_i(x),$

where each agent $i$ in a networked system holds a (possibly nonconvex) smooth local loss $f_i : \mathbb{R}^d \to \mathbb{R}$. The agents are nodes in an undirected, connected network $\mathcal{G}=(\mathcal{V}, \mathcal{E})$ with $N=|\mathcal{V}|$ agents, collaborating by exchanging messages only with immediate neighbors. Network communication is governed by a symmetric, doubly stochastic mixing matrix $W \in \mathbb{R}^{N \times N}$ that reflects the graph topology: $W_{ij} > 0$ iff $(i,j) \in \mathcal{E}$. The spectrum of $W$ is ordered as $$1 = \lambda_1(W) > \lambda_2(W) \geq \cdots \geq \lambda_N(W) > -1$$.

Each agent $i$0 maintains a local model $i$1 at iteration $i$2. Gradients are computed via stochastic oracles, with $i$3 an unbiased estimate of $i$4, $i$5 and variance $i$6.

2. Algorithmic Structure of i-CDSGD

i-CDSGD operates by performing $i$7 neighbor-averaging consensus steps at each iteration, followed by a local stochastic gradient update. Specifically, for agent $i$8 at iteration $i$9:

• Consensus Phase: Starting from $f_i : \mathbb{R}^d \to \mathbb{R}$0, repeat for $f_i : \mathbb{R}^d \to \mathbb{R}$1:

$f_i : \mathbb{R}^d \to \mathbb{R}$2

After $f_i : \mathbb{R}^d \to \mathbb{R}$3 consensus steps, set $f_i : \mathbb{R}^d \to \mathbb{R}$4.

• SGD Update: Draw a minibatch, compute $f_i : \mathbb{R}^d \to \mathbb{R}$5, and update:

$f_i : \mathbb{R}^d \to \mathbb{R}$6

where $f_i : \mathbb{R}^d \to \mathbb{R}$7 is the step size. The algorithmic steps are summarized in the following procedural outline:

Step	Operation	Parameter
Initialization	$f_i : \mathbb{R}^d \to \mathbb{R}$8, $f_i : \mathbb{R}^d \to \mathbb{R}$9, $\mathcal{G}=(\mathcal{V}, \mathcal{E})$0, $\mathcal{G}=(\mathcal{V}, \mathcal{E})$1	N/A
Consensus	$\mathcal{G}=(\mathcal{V}, \mathcal{E})$2 steps: $\mathcal{G}=(\mathcal{V}, \mathcal{E})$3	$\mathcal{G}=(\mathcal{V}, \mathcal{E})$4
Gradient	$\mathcal{G}=(\mathcal{V}, \mathcal{E})$5	$\mathcal{G}=(\mathcal{V}, \mathcal{E})$6

This procedure enables explicit interpolation between communication (controlled by $\mathcal{G}=(\mathcal{V}, \mathcal{E})$7) and local computation burden.

3. Theoretical Convergence Properties

The convergence properties of i-CDSGD are established for both strongly convex and nonconvex objective settings through Lyapunov function techniques that capture the suboptimality and consensus error.

3.1 Strongly Convex Regime

Under $\mathcal{G}=(\mathcal{V}, \mathcal{E})$8-strong convexity and $\mathcal{G}=(\mathcal{V}, \mathcal{E})$9-smoothness assumptions, with consensus spectral gap $N=|\mathcal{V}|$0 and step size $N=|\mathcal{V}|$1 satisfying suitable bounds, i-CDSGD exhibits linear convergence to a neighborhood of the global minimizer:

$N=|\mathcal{V}|$2

where $N=|\mathcal{V}|$3, $N=|\mathcal{V}|$4, and the bias term

$N=|\mathcal{V}|$5

shrinks exponentially in $N=|\mathcal{V}|$6.

3.2 Nonconvex Regime

If the global objective is nonconvex, the expected average squared gradient norm converges at a sublinear rate:

$N=|\mathcal{V}|$7

where $N=|\mathcal{V}|$8 is a relevant Lyapunov function and the consensus error remains governed by $N=|\mathcal{V}|$9 and network topology.

4. Consensus–Optimality Trade-offs

The core trade-off in i-CDSGD resides in the choice of $W \in \mathbb{R}^{N \times N}$0, the number of consensus steps between SGD updates:

• Larger $W \in \mathbb{R}^{N \times N}$1: Improves consensus error bound, $W \in \mathbb{R}^{N \times N}$2, and tightens optimization error $W \in \mathbb{R}^{N \times N}$3, producing a consensus among agents but increasing communication cost per iteration.
• Smaller $W \in \mathbb{R}^{N \times N}$4: Reduces communication but increases the steady-state bias, resulting in larger consensus and optimization errors.

The spectral gap $W \in \mathbb{R}^{N \times N}$5 increases with $W \in \mathbb{R}^{N \times N}$6, leading to exponentially sharper consensus and smaller error floors at the cost of higher per-iteration communication.

5. Communication and Computational Complexity

The per-iteration cost for each agent in i-CDSGD consists of:

• Communication: $W \in \mathbb{R}^{N \times N}$7 neighbor exchanges per iteration, $W \in \mathbb{R}^{N \times N}$8 data exchange.
• Computation: One stochastic gradient evaluation (cost $W \in \mathbb{R}^{N \times N}$9) and $W_{ij} > 0$0 local averaging steps.

The overall computation is $W_{ij} > 0$1, and total communication is $W_{ij} > 0$2. Increasing $W_{ij} > 0$3 linearly increases communication and local computation. The consensus error, by contrast, decreases exponentially in $W_{ij} > 0$4 (Jiang et al., 2018).

6. Empirical Illustration

In numerical experiments on CIFAR-10 with $W_{ij} > 0$5 agents arranged according to a sparse ring topology, and fixed step size $W_{ij} > 0$6, varying $W_{ij} > 0$7 demonstrates the practical consensus–optimality trade-off:

• $W_{ij} > 0$8 (standard CDSGD) converges to $W_{ij} > 0$9 test accuracy with inter-agent accuracy spread $(i,j) \in \mathcal{E}$0 over $(i,j) \in \mathcal{E}$1 epochs.
• $(i,j) \in \mathcal{E}$2 yields $(i,j) \in \mathcal{E}$3 accuracy and consensus spread of $(i,j) \in \mathcal{E}$4.
• $(i,j) \in \mathcal{E}$5 results in $(i,j) \in \mathcal{E}$6 accuracy and $(i,j) \in \mathcal{E}$7 spread.

Larger $(i,j) \in \mathcal{E}$8 values confer improved consensus and slightly higher accuracy at the cost of proportionally increased communication per iteration.

7. Summary and Implications

i-CDSGD integrates multiple consensus operations with local stochastic gradient updates to mediate the balance between distributed consensus and global optimization. The tunable consensus step parameter $(i,j) \in \mathcal{E}$9 provides explicit control over convergence speed, error floor, and communication intensity. Linear convergence up to an exponentially decaying bias can be guaranteed for strongly convex problems, and sublinear (in $W$0) convergence for nonconvex objectives, with error bounds precisely characterized in terms of network spectral properties and algorithmic parameters. This positions i-CDSGD as a flexible and theoretically grounded framework for distributed deep learning in network-constrained environments (Jiang et al., 2018).