Multi-objective Evolutionary Federated Learning (1812.07478v2)

Abstract: Federated learning is an emerging technique used to prevent the leakage of private information. Unlike centralized learning that needs to collect data from users and store them collectively on a cloud server, federated learning makes it possible to learn a global model while the data are distributed on the users' devices. However, compared with the traditional centralized approach, the federated setting consumes considerable communication resources of the clients, which is indispensable for updating global models and prevents this technique from being widely used. In this paper, we aim to optimize the structure of the neural network models in federated learning using a multi-objective evolutionary algorithm to simultaneously minimize the communication costs and the global model test errors. A scalable method for encoding network connectivity is adapted to federated learning to enhance the efficiency in evolving deep neural networks. Experimental results on both multilayer perceptrons and convolutional neural networks indicate that the proposed optimization method is able to find optimized neural network models that can not only significantly reduce communication costs but also improve the learning performance of federated learning compared with the standard fully connected neural networks.

• The paper introduces a bi-objective optimization framework that balances communication efficiency with global model accuracy.
• It adapts the sparse evolutionary training algorithm using an Erdos Rényi random graph to reduce neural network complexity and data transfer.
• Experimental results demonstrate competitive accuracies and significant communication cost reductions on both IID and non-IID datasets.

Overview of Multi-objective Evolutionary Federated Learning

The paper "Multi-objective Evolutionary Federated Learning" authored by Hangyu Zhu and Yaochu Jin presents an innovative approach to federated learning (FL), which is designed to overcome one of its principal drawbacks: high communication costs. Federated learning facilitates the development of a global model using distributed data located on user devices, thereby preserving privacy. However, it demands significant communication resources because data updates must be frequently exchanged between local client devices and the central server. This paper proposes a novel method utilizing a multi-objective evolutionary algorithm to minimize these communication costs while simultaneously optimizing model accuracy.

Key Contributions

The paper's central contribution is the formulation of federated learning as a bi-objective optimization problem, focusing on minimizing both communication costs and global model test errors. The authors introduce a modified sparse evolutionary training (SET) algorithm, which adjusts the connectivity of neural networks via an Erdos Rényi random graph to ensure scalability in federated settings. Specifically, the key contributions include:

1. Bi-Objective Optimization Framework: Establishing federated learning as a dual-objective problem addressed with an evolutionary algorithm to balance communication efficiency with model accuracy.
2. Modified SET Algorithm: Adapting the SET procedure to reduce the neural network's number of weights, thereby decreasing the data transfer required for each communication round. This modification enables adjustments to the network topology, ensuring a more efficient connection structure tailored for federated settings.
3. Experimental Evaluation: Comprehensive experiments utilizing MLP and CNN architectures under both IID and non-IID data distributions demonstrate that the proposed method provides a favorable trade-off between communication efficiency and model performance.

Results and Implications

The experiments indicate that the proposed framework and algorithms can significantly reduce model complexity and communication costs while maintaining, or even improving, learning accuracy compared to fully connected models. Notably:

• On the IID datasets, optimized models achieved slightly higher global test accuracies than standard models with fewer model connections, indicating a reduction of communication rounds and associated costs.
• On non-IID datasets, models designed through this optimization achieved comparable accuracy with a significant reduction in required parameters, thus addressing the disparity caused by non-IID data distribution.

The implications of these findings are twofold. Practically, it suggests that federated learning could be more broadly applied in settings with limited communication bandwidth by exploiting optimized network topologies. Theoretically, it provides a framework that bridges privacy-centric learning with cost-effective implementation strategies, highlighting a new perspective in federated optimization research.

Future Research Directions

Several research opportunities stem from this work:

• Scalability to Larger Networks: Although the framework demonstrated efficacy on smaller networks, there remains a need to explore scalability in larger architectures and diverse datasets.
• Impact of Adversarial Conditions: Future studies should consider how adversarial attacks might affect the robustness of models optimized using this method, given their implications for model integrity and privacy.
• Communication Dynamics: Investigating how network conditions, such as latency and packet loss, influence the performance of the proposed federated learning approach could yield insights into deployment in real-world applications.

Overall, this paper provides a significant step forward in the realization of practical federated learning by addressing critical challenges around communication efficiency and model optimization.