Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data (2305.04792v1)

Abstract: Decentralized learning enables the training of deep learning models over large distributed datasets generated at different locations, without the need for a central server. However, in practical scenarios, the data distribution across these devices can be significantly different, leading to a degradation in model performance. In this paper, we focus on designing a decentralized learning algorithm that is less susceptible to variations in data distribution across devices. We propose Global Update Tracking (GUT), a novel tracking-based method that aims to mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a $1-6\%$ improvement in test accuracy compared to other existing techniques.

• The paper introduces Global Update Tracking (GUT), a novel decentralized algorithm that tracks global model updates to manage heterogeneous data efficiently.
• It reduces communication overhead by halving data exchanges while improving test accuracy by 1-6% compared to previous methods.
• Theoretical analysis confirms a competitive non-asymptotic convergence rate, supporting its application in edge computing and privacy-sensitive scenarios.

Overview

The paper presents Global Update Tracking (GUT), a novel algorithm for decentralized learning over heterogeneous data distributions. Decentralized learning eschews the need for a central server by enabling the training of machine learning models across multiple devices or 'agents', each with its own local dataset. A common challenge in such techniques is dealing with non-Independently and Identically Distributed (non-IID) data, which tends to be the norm in decentralized settings. The paper offers a comprehensive solution to this issue through GUT, a tracking-based method that enhances the performance of decentralized algorithms without incurring additional communication costs.

Algorithmic Contributions

GUT addresses the communication overhead commonly associated with existing decentralized learning algorithms that adopt tracking mechanisms. The proposed algorithm operates by tracking the global model updates rather than individual gradients. This approach allows each agent to communicate only its model updates, which curbs the need to share both model parameters and tracking variables, effectively halving the communication requirements. The central novelty lies in the maintenance of a tracking variable that represents the model update, which is aligned with the consensus model's trajectory over time. The GUT method reports impressive results, providing a 1-6% increase in test accuracy over previously established techniques for decentralized learning.

Theoretical Insights

In addition to the empirical results, the authors deliver a theoretical analysis of GUT, proving its convergence rate. They establish the non-asymptotic convergence rate under standard assumptions, such as Lipschitz gradients and bounded variance. The analysis reveals that the algorithm meets the convergence rates of the best-known decentralized algorithms without extra computational burden. This facet is crucial in confirming the algorithm's applicability and reliability.

Empirical Evaluation

The thorough empirical evaluation includes a variety of datasets, such as CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette, along with different neural network architectures. The paper reports that the quasi-global momentum version of GUT, or QG-GUTm, consistently outperforms current benchmarks across various levels of data heterogeneity. Notably, it was shown to enhance the CIFAR-10 dataset classification accuracy significantly, even in highly heterogeneous scenarios. These robust experimental results firmly endorse the efficacy of GUT for decentralized learning on heterogeneous datasets.

Potential and Impact

The research outcomes offer a promising direction for leveraging the distributed datasets effectively while keeping communication costs low. The scalability and robustness of GUT to data heterogeneity make it an attractive solution for deploying machine learning models in edge computing and privacy-sensitive applications. As an enabling technology, GUT could significantly contribute to the expanded adoption of decentralized machine learning across various real-world applications, advancing the field towards more efficient and scalable learning paradigms.