Federated Optimization:Distributed Optimization Beyond the Datacenter (1511.03575v1)

Abstract: We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are distributed (unevenly) over an extremely large number of \nodes, but the goal remains to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of utmost importance. A motivating example for federated optimization arises when we keep the training data locally on users' mobile devices rather than logging it to a data center for training. Instead, the mobile devices are used as nodes performing computation on their local data in order to update a global model. We suppose that we have an extremely large number of devices in our network, each of which has only a tiny fraction of data available totally; in particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, we assume that no device has a representative sample of the overall distribution. We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results. This work also sets a path for future research needed in the context of federated optimization.

• The paper introduces Federated Optimization as a new paradigm for distributed learning on non-IID, unbalanced data across numerous user devices.
• The paper adapts SVRG with flexible stepsizes, sparse data scaling, and adaptive aggregation to ensure rapid convergence with minimal communication rounds.
• Experimental results demonstrate that the proposed DSVRG method outperforms other algorithms like DANE and CoCoA, highlighting its efficiency in real-world settings.

Federated Optimization: Distributed Optimization Beyond the Datacenter

Introduction

The paper "Federated Optimization: Distributed Optimization Beyond the Datacenter" by Jakub Konečný et al. introduces the concept of Federated Optimization, which represents a new paradigm in distributed machine learning. The fundamental challenge addressed by the paper is optimizing a centralized model when data is distributed unevenly across an extremely large number of nodes, such as user devices like smartphones. This paper underscores communication efficiency as a critical factor in Federated Optimization.

Problem Formulation

The paper situates its problem within the established framework of finite-sum optimization problems, expressed as:

$$\min_{w \in \mathbb{R}^d} f(w) \quad \text{where} \quad f(w) = \frac{1}{n} \sum_{i=1}^n f_i(w),$$

where the objective function $f(w)$ is an average of several component functions $f_i(w)$, each dependent on data points $(x_i, y_i)$.

The traditional approach to distributed optimization is unsuitable for scenarios where data is massively distributed across user devices, is non-IID (non-Independent and Identically Distributed), and highly unbalanced.

Challenges in Federated Optimization

The paper identifies key characteristics that algorithms for Federated Optimization must handle:

• Massively Distributed Data: Data is stored across a very large number of nodes, with each node containing only a tiny fraction of the total dataset.
• Non-IID Data: The data on each node may follow different distributions, deviating significantly from a global distribution representation.
• Unbalanced Data: Nodes may hold vastly varying numbers of data points, complicating the training process.

Proposed Algorithm

The authors propose an algorithm inspired by connections between Stochastic Variance Reduced Gradient (SVRG) and Distributed Approximate Newton (DANE) methods. The algorithm outlined is a distributed version of SVRG that adapts for the unique challenges of Federated Optimization. Noteworthy modifications to the algorithm include:

1. Flexible Stepsize: Different stepsizes for each node, adjusted based on the local data size, $n_k$.
2. Sparse Data Handling: Scaling of stochastic updates by a diagonal matrix $S_k$ to mitigate biases due to uneven feature distribution.
3. Adaptive Aggregation: Adjustments in the aggregation procedure to account for data sparsity by incorporating a diagonal matrix $A$.

These modifications enhance robustness and stability in the federated learning setting, particularly with sparse data scenarios.

Experimental Results

The experimental results, demonstrated on a dataset of public social network posts, highlight the efficiency of the proposed method. The dataset contained posts from 10,000 authors, divided into training (75%) and testing (25%) sets. Logistic regression was employed to predict whether posts would receive comments, using a bag-of-words model with the 20,000 most frequent words.

The algorithm demonstrated superior performance compared to existing communication-efficient algorithms, such as DANE and DiSCO, which diverged in this setting. Notably, the proposed algorithm, referred to as DSVRG, reached optimality within a few communication rounds, unlike other algorithms including CoCoA and distributed gradient descent. The robustness of the algorithm is underscored by similar performance both with originally clustered and randomly reshuffled data.

Implications and Future Work

Federated Optimization is poised to grow in importance given the increasing computational power of mobile devices and rising concerns about data privacy. The approach promises to save network bandwidth by performing computation on local devices, offering privacy benefits by keeping data decentralized. However, substantial questions remain open, including:

• The need for public datasets that naturally align with the federated optimization framework, to facilitate broader engagement and validation.
• More rigorous experimental validation, particularly in complex scenarios like deep learning.
• Theoretical advances to solidify and enhance the proposed algorithm.
• Integrating differential privacy to strengthen data security and privacy in practical implementations.

In summary, while the paper provides a firm foundation for Federated Optimization and showcases a promising algorithmic solution, further research is essential to address outstanding challenges and fully realize the potential of this new paradigm in distributed machine learning.