LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication

Abstract: In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such as Adam, have demonstrated strong practical performance in deep learning and centralized distributed settings. However, their convergence properties remain largely unexplored in decentralized settings involving multiple local training steps, such as federated learning. To address this limitation, we propose LoDAdaC, a unified multiple Local Training (MLT) Decentralized framework with Adam-type updates and Compressed communication (CC). LoDAdaC accommodates a broad class of optimizers for its local adaptive updates, including AMSGrad, Adam, and AdaGrad; it is compatible with standard (possibly biased) compressors such as low-bit quantization and sparsification. MLT and CC enable LoDAdaC to achieve multiplied reduction of communication cost, while the technique of adaptive updates enables fast convergence. We rigorously prove the combined advantage through complexity analysis. In addition, experiments on image classification and GPT-style LLM training validate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.

• The paper introduces LoDAdaC, a unified framework combining multiple local training, compressed communication, and adaptive optimizers for decentralized nonconvex optimization.
• It provides rigorous non-asymptotic convergence guarantees with per-agent iteration complexity of O(1/(nε^4)) and exponential communication savings via local updates and Top-k compression.
• Empirical evaluations on image classification and language modeling demonstrate faster convergence, near-linear scalability with agent count, and resilience to non-IID data.

LoDAdaC: Unified Local Training-Based Decentralized Optimization with Adaptive Gradients and Compressed Communication

Introduction and Context

The LoDAdaC framework (“Local training-based Decentralized framework with Adaptive gradients and Compressed communication”) (2604.09970) addresses the confluence of three algorithmic directions in decentralized learning: multiple local updates (MLT), compressed communication (CC), and adaptive gradient methods (e.g., Adam, AMSGrad, AdaGrad). While adaptive optimizers are ubiquitous in deep learning, there has been limited theoretical and algorithmic advancement that delivers their benefits in decentralized, communication-efficient topologies, especially under heterogeneous (i.e., non-IID) data. LoDAdaC provides a unified approach encompassing MLT and CC within a decentralized, adaptive stochastic optimization paradigm for nonconvex objectives.

Algorithmic Framework

LoDAdaC generalizes the decentralized parallel stochastic optimization problem with $n$ agents connected in a graph, each hosting private data distributions. Each node only communicates (possibly compressed) messages to its immediate neighbors. The key ideas in LoDAdaC are:

• Multiple Local Training (MLT): Each agent performs $K$ local optimizer steps (e.g., Adam, AMSGrad, or AdaGrad) using its own data between communication rounds.
• Compressed Communication (CC): Communication of updates is performed via $\eta$-contractive compressors, covering both unbiased and biased mechanisms such as quantization and Top-$k$ sparsification.
• Adaptive Optimizer Support: The core LODAdaC update generalizes the adaptive preconditioners found in Adam, AMSGrad, and AdaGrad. The local update can admit momentum, per-parameter learning rates, and other adaptive schemes.

The scheme requires minimal synchronization and offers robustness to network topology and data heterogeneity thanks to its consensus-driven mixing via doubly stochastic matrices. The algorithm subsumes non-adaptive decentralized SGD and decentralized momentum as degenerate cases.

Theoretical Analysis

A central technical contribution is the non-asymptotic convergence analysis of adaptive methods with both MLT and CC in the decentralized stochastic nonconvex case. The theoretical framework addresses several open challenges:

• Consensus Error Control: Tight control of the consensus error under simultaneously adaptive and compressed/proximal updates, without restrictive assumptions on the compressors (allows for biased compression) or mixing matrix (e.g., non-symmetric allowed).
• Unified Complexity Bounds: LoDAdaC provably achieves a per-agent iteration complexity of $O(1/(n\epsilon^4))$ and total communication complexity $O\left(\frac{1-\eta}{nK\epsilon^4}\right)$ for producing an $\epsilon$-stationary solution. Increasing $K$ (number of local updates per round) and reducing $\eta$ (improved compression) provide compounded communication savings.
• Optimizer-Agnostic Guarantees: The results hold uniformly across the range of supported optimizers (including vector/matrix AdaGrad, AMSGrad, Adam, etc.).
• Practical Step Sizes: Step size does not depend on network mixing matrix spectral properties or data compression ratio, offering linear speedup with the number of agents and ease of hyperparameter selection.

Empirical Evaluation

The numerical study validates both convergence rates and communication savings on image classification (FashionMNIST, CIFAR-10) and language modeling (nanoGPT on Shakespeare). A wide hyperparameter grid is surveyed: varying $K$, optimizer choice, agents/topologies, and Top-$K$0 compression.

• Optimizer Comparison: Adaptive methods (Adam, AMSGrad, AdaGrad) consistently converge faster and more stably than decentralized momentum SGD baselines (e.g., SQuARM-SGD), with particularly pronounced advantages for LLMs.

Figure 1: Comparative performance across optimizers on CIFAR-10, showing faster convergence and higher accuracy for adaptive methods vs momentum SGD.

• Communication Rounds and Volume: Increasing $K$1 and employing Top-$K$2 compression reduces the number of communication rounds and the total communication volume exponentially, with minimal or negligible impact on test performance.

Figure 2: CIFAR-10 accuracy and loss as $K$3 increases, demonstrating a drastic reduction in communication rounds while retaining performance.

Figure 3: Drastic reduction in total communication volume on CIFAR-10 using higher local steps and compression; $K$4 with Top-$K$5 reduces communication by 99.4\% compared to the uncompressed, $K$6 baseline.

• Scalability: Near-linear speedup is observed with respect to the agent count, for both ring and grid topologies, and across all optimizers.

Figure 4: Linear scaling of training loss and accuracy with increasing agent counts on CIFAR-10 under a ring topology using Adam.

• Data Heterogeneity Resilience: Significant heterogeneity (Dirichlet $K$7) slows convergence but does not incur substantial degradation in test accuracy or consensus, demonstrating empirical resilience under non-IID setups.

Figure 5: Robustness to non-IID data partitioning (Dirichlet $K$8) on CIFAR-10 with LoDAdaC.

Implications and Future Directions

LoDAdaC establishes theoretically-principled and empirically effective decentralization for adaptive optimizers in large-scale, bandwidth-constrained, and privacy-conscious scenarios. The framework not only matches the best-known communication and iteration complexities achieved by momentum-based methods, but—through adaptive preconditioning—achieves much faster empirical convergence with no additional communication cost. The analytical paradigm developed also yields tools for further analysis involving biased compressors, non-symmetric mixing, and a broader class of optimizers.

The practical implications are substantial for federated and decentralized learning contexts (e.g., cross-silo/edge collaborative learning, privacy-preserving networks, LLM pretraining in decentralized environments). A critical direction for future research is extending theoretical results beyond bounded-gradients (which is restrictive for deep networks), developing adaptive step-size strategies robust to high gradient variance, and testing scalability on larger models and graph topologies.

Conclusion

LoDAdaC provides a unified, theoretically sound, and practically compelling solution for decentralized, adaptive, and communication-efficient nonconvex stochastic optimization. It enables the deployment of state-of-the-art adaptive optimizers under minimal communication and synchronization, is resilient to topological and data heterogeneity, and is compatible with the implementation constraints of modern distributed and federated systems. The combination of provable efficiency gains and strong empirical performance substantiates the practicality of MLT+CC+adaptive strategies in real-world decentralized learning applications.