Dual Local SGD for Distributed Learning

• Dual Local SGD is a distributed optimization algorithm that decouples local and global step sizes to reduce communication overhead and enhance convergence speed.
• It employs dual step sizes where larger local updates are aggregated with a proportionally scaled global update to correct variance and optimize time complexity.
• Experimental validations on deep learning benchmarks demonstrate its robustness to system heterogeneity and improved performance compared to traditional methods.

Dual Local SGD refers to a family of distributed stochastic optimization algorithms that decouple local and global step sizes to achieve optimal wall-clock efficiency in large-scale, communication-constrained learning. It corrects the canonical Local SGD and Federated Averaging (FedAvg) schemes (which are suboptimal in time complexity) by using two distinct step sizes—one for the local (on-device) SGD loops and another for global parameter aggregation. This framework not only closes the time-complexity gap with respect to Minibatch SGD and Hero SGD in the nonconvex regime but also generalizes to broader classes of asynchronous and local computation graphs. The following sections detail the methodology, theoretical properties, and significance of Dual Local SGD and its extensions (Fradin et al., 27 Sep 2025).

1. Motivation and Limitations of Canonical Local SGD

Local SGD/FedAvg permits each worker to execute multiple local updates before synchronizing with other nodes, substantially reducing communication frequency. In canonical Local SGD, the update scheme is: $x^{t+1} = \frac{1}{n} \sum_{i=1}^n z^{t}_{(i,K)}$ where each worker $i$ starts at $x^t$ and performs $K$ local SGD steps to reach $z^{t}_{(i,K)}$.

When analyzed for time complexity—including both per-gradient computation time $h$ and inter-node communication time $\tau$—canonical Local SGD is provably inferior to Minibatch SGD and Hero SGD. Despite attractive iteration complexity (per communication round), excessive scaling of the local step size and naive averaging lead to suboptimal wall-clock convergence, with the communication overhead nullifying the algorithm’s parallelization benefits (Fradin et al., 27 Sep 2025).

2. Algorithmic Design of Dual Local SGD

Dual Local SGD resolves the time-complexity inefficiency by introducing a decoupled update:

1. Local (Inner) Updates: Each worker $i$ performs $K$ SGD steps with step size $\eta_\ell$:

$$z^{t}_{(i,0)} = x^t, \quad z^{t}_{(i,j+1)} = z^{t}_{(i,j)} - \eta_\ell \nabla f\left(z^{t}_{(i,j)}; \xi^{t}_{(i,j)}\right), \quad j = 0, \ldots, K-1$$

1. Global Aggregation (Outer Update): The new global iterate is updated as

$$x^{t+1} = x^t - \eta_g \sum_{i=1}^n \sum_{j=0}^{K-1} \nabla f\left(z^{t}_{(i,j)}; \xi^{t}_{(i,j)}\right)$$

where $\eta_g$ is the global step size, typically set as $\eta_g = \eta_\ell / \sqrt{n}$, i.e., $\eta_\ell = \sqrt{n} \cdot \eta_g$.

This corrected scaling ensures that the aggregated gradient’s contribution at each round matches the variance reduction expected from parallelization; it avoids the “overcounting” effect present in the canonical Local SGD scheme (Fradin et al., 27 Sep 2025).

3. Optimal Time Complexity and Step Size Scheduling

Dual Local SGD achieves optimal time complexity up to logarithmic factors for both convex and nonconvex objectives. The overall wall-clock time to achieve $\epsilon$-stationarity is

$$T_{\text{Dual}} = \tau\cdot O\left(\frac{L \Delta}{\epsilon}\right) + h\cdot O\left( \frac{L \Delta}{\epsilon} + \frac{L \sigma^2 \Delta}{n \epsilon^2} \right)$$

where $L$ is the Lipschitz constant, $\Delta$ is the initial suboptimality, and $\sigma^2$ is the variance of the stochastic gradients.

Further improvements are obtained in Decaying Local SGD, which uses a schedule: $$\eta_j = \sqrt{ \frac{b}{(j+1)(\log K + 1)} } \cdot \eta_g$$ where $b = \max \{\sigma^2 / \epsilon, n\}$ and $j$ indexes the local step. This adaptive decay allows much larger step sizes in early local updates while controlling divergence as local steps increase, further reducing time-to-accuracy in adversarial or high-variance settings (Fradin et al., 27 Sep 2025).

4. Comparison with Minibatch SGD and Hero SGD

A summary comparison of time complexity bounds is as follows:

Method	Time Complexity (Convex/Nonconvex)	Scaling of Step Size
Minibatch SGD	$$\tau O(L \Delta / \epsilon) + h O(L\sigma^2 \Delta / (n\epsilon^2))$$	$\eta \sim 1/n$
Hero SGD	$$h O(L\Delta/\epsilon + L \sigma^2 \Delta/\epsilon^2)$$	N/A (single node)
Local SGD (naive)	Worse, due to over-scaled local steps	$\eta_\ell = n\cdot\eta$
Dual Local SGD	As in Minibatch/Hero up to log factors	$\eta_\ell = \sqrt{n}\cdot\eta_g$
Decaying Local SGD	As above, or better for “hard” instances	Adaptive, larger at early steps

Proper decoupling of $\eta_\ell$ and $\eta_g$ (i.e., not naively setting $\eta_\ell = n\eta_g$) is essential to avoid excessive drift and suboptimal convergence (Fradin et al., 27 Sep 2025).

5. Implications for Asynchronous and Hierarchical Methods

Dual Local SGD theory, together with step-size decoupling and adaptive decay, generalizes to asynchronous and computation-graph based schemes. The “Birch SGD” abstraction allows non-main branches (auxiliary local update branches) to follow adaptive or decaying step sizes for optimal complexity—subsuming a wide class of previously proposed local, asynchronous, or federated methods (Fradin et al., 27 Sep 2025).

In federated settings, efficient decoupling ensures robust performance in heterogeneous environments. Larger initial local steps amplify model progress while tuning the decay factor ensures stability across varying device speeds and participation rates.

6. Practical Implementation and Experimental Validation

Dual Local SGD and its variants have been evaluated on large-scale deep learning benchmarks, including deep residual networks on ImageNet. Results confirm:

• Achieving the same accuracy as Minibatch SGD and FedAvg with significantly improved wall-clock efficiency when computation/communication time scales as predicted.
• Robustness to system heterogeneity when local steps are adaptively scheduled.
• The effectiveness of decaying local step sizes in “hard” nonconvex regimes or when variance is large, confirming theoretical predictions.

Implementation requires only setting two step sizes, with defaults $\eta_\ell = \sqrt{n}\cdot\eta_g$ and a simple schedule for decaying $\eta_j$ during local loops. No change is needed to aggregation or communication primitives versus standard Local SGD.

7. Impact and Extensions

Dual Local SGD redefines the theoretical and practical landscape for parallel and federated optimization. By closing the time-complexity gap and providing a blueprint for step-size scaling, it directly informs the optimal design of local update rules across distributed, asynchronous, and hierarchical architectures. These insights apply directly to algorithmic frameworks in federated learning, cloud/distributed datacenter training, and large-scale data-parallel optimization, including settings with variable device participation or deep communication bottlenecks (Fradin et al., 27 Sep 2025).

Moreover, the corrected aggregation and step-size design principles in Dual Local SGD extend to the development of advanced methods employing error feedback, quantized communication, variance reduction, and heterogeneous device adaptivity. The decoupling of local and global optimization dynamics remains key to efficient and robust learning in communication-limited settings.