Federated Learning: Strategies for Improving Communication Efficiency (1610.05492v2)

Abstract: Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model. The typical clients in this setting are mobile phones, and communication efficiency is of the utmost importance. In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, where we learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling before sending it to the server. Experiments on both convolutional and recurrent networks show that the proposed methods can reduce the communication cost by two orders of magnitude.

• The paper demonstrates structured and sketched updates that significantly cut communication costs in federated learning.
• It details low-rank and random mask approaches alongside subsampling, quantization, and random rotations to compress model updates.
• Experimental results reveal up to 256x compression with minimal accuracy loss, enabling scalable, efficient federated training.

Federated Learning: Strategies for Improving Communication Efficiency

The paper "Federated Learning: Strategies for Improving Communication Efficiency" by Jakub Konenčny et al. addresses critical challenges and strategies in the context of Federated Learning (FL), where the objective is to train a centralized model using data distributed across numerous clients, such as mobile devices. The haLLMark of Federated Learning lies in its ability to enable collaborative model training while keeping the data localized on the client's device, thus ensuring data privacy and security.

Introduction and Problem Definition

Federated Learning presents unique computational constraints due to the large number of participating devices, each possessing diverse and non-i.i.d. data and typically constrained by slow and unreliable network connections. The paper identifies the primary bottleneck as uplink communication, emphasized by the asymmetrical nature of typical internet connections (e.g., 55.0 Mbps download vs. 18.9 Mbps upload).

The proposed algorithms aim to mitigate this communication bottleneck via two main strategies:

• Structured Updates: Learning an update from a restricted space that requires fewer parameters, and
• Sketched Updates: Compressing model updates through quantization, random rotations, and subsampling before transmitting them.

Structured Updates

The approach of structured updates confines the updates $H^i_t$ to a lower-dimensional subspace:

• Low-rank updates: These updates constrain matrices $H^i_t$ to be of low rank, achieved via the product $H^i_t = A^i_t B^i_t$, where $A^i_t \in \mathbb{R}^{d_1 \times k}$ and $B^i_t \in \mathbb{R}^{k \times d_2}$. This parameterization reduces the number of variables needed to represent the update, notably enhancing communication efficiency.
• Random mask updates: This sparsity-based approach restricts updates to a sparse matrix with a pre-defined pattern, generated using a random seed. Only non-zero elements and the seed are transmitted, significantly cutting down the transmitted data size.

Sketched Updates

Sketched updates involve compressing the full model update $H^i_t$ after computation:

• Subsampling: Randomly selecting and communicating a subset of $H^i_t$'s elements.
• Probabilistic quantization: Compressing updates via bit-wise quantization, where each update's value can be projected to a limited number of bits.
• Random rotations: Preprocessing updates with structured random rotations to reduce the quantization error.

Experimental Results

Experiments were conducted using Federated Learning to train deep neural networks on both CIFAR-10 and a large-scale Reddit dataset.

For CIFAR-10, structured random mask updates demonstrated superior performance over low-rank updates. The experiments revealed that subsampling and quantization, especially when paired with random rotations, maintained model performance while significantly reducing communication. For example, random rotations combined with $2$-bit quantization and $6.25\%$ subsampling achieved compression by a factor of $256$ with negligible loss in accuracy.

In the Reddit dataset, simulations involved training an LSTM for next-word prediction across $763,430$ clients. The evaluations showed that combining sketched updates techniques, particularly random rotations with aggressive subsampling and $2$-bit quantization, drastically reduced communication needs (by up to two orders of magnitude) while achieving substantial accuracy.

Implications and Future Directions

The proposed methods have shown practical utility in reducing communication overhead, essential for deploying Federated Learning systems at scale, particularly in scenarios where clients have limited upload bandwidth. The ability to maintain competitive model performance while drastically cutting communication costs underscores the practical feasibility of Federated Learning.

Future research might explore further optimizations in model sketching and quantization, adaptive client selection strategies to maximize computational efficiency, and extending the current methods to diverse model architectures and real-world datasets. These improvements could enhance the robustness and scalability of Federated Learning systems, advancing their integration into widespread applications.

This paper lays a solid foundation for addressing the communication inefficiencies in Federated Learning, paving the way for broader adoption and more resource-efficient implementations.