Accelerating DNN Training in Wireless Federated Edge Learning Systems (1905.09712v3)

Abstract: Training task in classical machine learning models, such as deep neural networks, is generally implemented at a remote cloud center for centralized learning, which is typically time-consuming and resource-hungry. It also incurs serious privacy issue and long communication latency since a large amount of data are transmitted to the centralized node. To overcome these shortcomings, we consider a newly-emerged framework, namely federated edge learning, to aggregate local learning updates at the network edge in lieu of users' raw data. Aiming at accelerating the training process, we first define a novel performance evaluation criterion, called learning efficiency. We then formulate a training acceleration optimization problem in the CPU scenario, where each user device is equipped with CPU. The closed-form expressions for joint batchsize selection and communication resource allocation are developed and some insightful results are highlighted. Further, we extend our learning framework to the GPU scenario. The optimal solution in this scenario is manifested to have the similar structure as that of the CPU scenario, recommending that our proposed algorithm is applicable in more general systems. Finally, extensive experiments validate the theoretical analysis and demonstrate that the proposed algorithm can reduce the training time and improve the learning accuracy simultaneously.

Accelerating DNN Training in Wireless Federated Edge Learning Systems

The paper "Accelerating DNN Training in Wireless Federated Edge Learning Systems" addresses the critical challenge of reducing training latency and resource consumption in Deep Neural Network (DNN) training by leveraging Federated Edge Learning (FEEL). Traditional centralized training methods entail significant data transmission to a cloud center, leading to privacy concerns, heavy resource usage, and high communication latency. FEEL offers a decentralized alternative where local updates are aggregated at the network edge, circumventing these drawbacks. The paper focuses on optimizing training efficiency in FEEL systems by balancing computation and communication costs.

The authors introduce a novel evaluation criterion called learning efficiency, defined as the ratio of the global loss decay to the end-to-end latency in the training process. This efficiency metric underpins the formulation of an optimization problem aimed at accelerating DNN training by optimally selecting batch size and allocating communication resources in both CPU and GPU scenarios.

Key elements of the paper include:

1. System Model Definition: The FEEL system model describes a scenario with multiple edge computing devices interacting with a central edge server. The model articulates the process whereby each device performs local computations based on its data subset before sharing gradient updates with the edge server. This setup is contrasted with conventional centralized data processing methods.
2. Problem Formulation: The training acceleration problem is divided into subproblems focusing on local gradient calculation, uploading, and global gradient download, each aiming to maximize learning efficiency subject to communication and computation resource constraints.
3. CPU and GPU Scenarios: The paper explores distinct approaches for CPU and GPU-based devices, noting that computational and memory capacities influence optimal training practices. For CPUs, batch size is correlated linearly and sub-linearly with training speed and communication rates, respectively. A parallel investigation is carried out for GPU-equipped devices where training latency and resource allocation mechanisms are adapted accordingly.
4. Mathematical Analysis and Solutions: Solutions involve closed-form expressions and iterative algorithms for joint batch size selection and communication resource allocation. The paper shows that optimal batch sizes dynamically adapt to wireless channel conditions, ensuring efficient use of resources.
5. Experimental Validation: Extensive numerical experiments involving popular DNN models and real datasets demonstrate the feasibility of the proposed schemes. The results indicate improved training speed and model accuracy, supporting the efficacy of the proposed federated edge learning strategies against several benchmark schemes.

The implications of this research are significant for the practical deployment of AI systems in wireless networks, especially in applications that demand private data handling and rapid model training. The paper outlines a pathway for reducing latency and resource burden in federated learning setups, potentially catalyzing more widespread adoption of edge computing paradigms in AI processes.

Looking forward, potential developments include addressing non-IID data challenges further, optimizing long-term training acceleration strategies, and exploring analogous solutions in non-orthogonal communication systems. This paper sets the foundation for integrating more sophisticated AI capabilities into wireless networks, enhancing their efficiency and responsiveness to dynamic environments.