Fast-Convergent Federated Learning (2007.13137v2)

Abstract: Federated learning has emerged recently as a promising solution for distributing machine learning tasks through modern networks of mobile devices. Recent studies have obtained lower bounds on the expected decrease in model loss that is achieved through each round of federated learning. However, convergence generally requires a large number of communication rounds, which induces delay in model training and is costly in terms of network resources. In this paper, we propose a fast-convergent federated learning algorithm, called FOLB, which performs intelligent sampling of devices in each round of model training to optimize the expected convergence speed. We first theoretically characterize a lower bound on improvement that can be obtained in each round if devices are selected according to the expected improvement their local models will provide to the current global model. Then, we show that FOLB obtains this bound through uniform sampling by weighting device updates according to their gradient information. FOLB is able to handle both communication and computation heterogeneity of devices by adapting the aggregations according to estimates of device's capabilities of contributing to the updates. We evaluate FOLB in comparison with existing federated learning algorithms and experimentally show its improvement in trained model accuracy, convergence speed, and/or model stability across various machine learning tasks and datasets.

• The paper introduces Federated Optimization with Lower Bound (FOLB), a novel algorithm that accelerates Federated Learning convergence through intelligent device sampling.
• Experiments show FOLB achieves significantly faster convergence, improved stability, and higher accuracy compared to FedAvg and FedProx.
• FOLB effectively handles heterogeneous environments and offers a path toward more robust, energy-efficient, and privacy-conscious distributed machine learning.

Fast-Convergent Federated Learning

In the paper titled "Fast-Convergent Federated Learning," the authors tackle the challenge of accelerating convergence rates in Federated Learning (FL), a distributed machine learning paradigm wherein model training is performed across multiple devices without data pooling on a central server. The FL model reduces privacy risks but faces challenges related to heterogeneity of data and systems, as well as high communication costs, which impeded rapid convergence. The paper introduces FOLB (Federated Optimization with Lower Bound), a novel algorithm designed to expedite convergence by intelligently sampling participating devices based on the expected improvement their local models can provide to the global model.

The paper first establishes a theoretical foundation for the proposed intelligent sampling approach by characterizing a lower bound on improvement achievable per round of federated learning under optimal device sampling. Specifically, devices are targeted based on an assessment of their gradient contributions towards reducing the global model loss. FOLB aims to achieve this bound, contrasting existing algorithms that sample devices uniformly, which tend naturally to slower convergence rates.

The authors present the formulation of a near-optimal selection probability distribution, termed LB-near-optimal distribution, which determines device selection based on the magnitude of the gradient inner products between local and global models. They show that this distribution optimizes expected loss reduction per communication round in FL, potentially with fewer devices involved than traditional methods, such as FedAvg or FedProx. The proposed algorithm circumvents the communication burden by leveraging gradient information efficiently, utilizing two separate sets of device samples for calibration of local updates and scaling parameters according to correlation with the estimated global gradient.

FOLB is demonstrated experimentally against FedAvg and FedProx across a spectrum of datasets including MNIST, FEMNIST, and public synthetic data. The results emphasize evident improvements in convergence speed, model stability, and accuracy. When varying proximal parameter values, device counts per round, model architectures, and levels of data non-IIDness, FOLB showcases consistently superior performance. Notably, FOLB requires significantly fewer iterations to reach comparable accuracy levels, a practical benefit where communication cost and training duration are critical.

The paper further explores the flexibility of FOLB in heterogeneous environments, where devices may exhibit varied computational capacities and communication delays. An adapted aggregation mechanism weights device updates not only by gradient impact but also by the optimality level of local solvers, allowing adjustments per device computation and communication profiles.

This research advances the theoretical and practical understanding of federated learning algorithm design, suggesting a pathway for future development in the optimization of distributed ML systems. The implications are substantial, hinting at more robust, energy-efficient, and privacy-conscious machine learning applications across diverse, real-world networked device environments. Future works are projected toward multi-period device selection methodologies to generate sustained performance gains while maintaining minimal communication overhead.

In conclusion, FOLB contributes a significant step forward in federated learning, providing a solid mathematical and empirical basis for intelligent device sampling methodologies that can be adapted to varying datasets and heterogeneities, potentially influencing developments in adaptive federated learning frameworks.