Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Joint Compression and Deadline Optimization for Wireless Federated Learning (2305.03969v2)

Published 6 May 2023 in cs.IT, cs.DC, and math.IT

Abstract: Federated edge learning (FEEL) is a popular distributed learning framework for privacy-preserving at the edge, in which densely distributed edge devices periodically exchange model-updates with the server to complete the global model training. Due to limited bandwidth and uncertain wireless environment, FEEL may impose heavy burden to the current communication system. In addition, under the common FEEL framework, the server needs to wait for the slowest device to complete the update uploading before starting the aggregation process, leading to the straggler issue that causes prolonged communication time. In this paper, we propose to accelerate FEEL from two aspects: i.e., 1) performing data compression on the edge devices and 2) setting a deadline on the edge server to exclude the straggler devices. However, undesired gradient compression errors and transmission outage are introduced by the aforementioned operations respectively, affecting the convergence of FEEL as well. In view of these practical issues, we formulate a training time minimization problem, with the compression ratio and deadline to be optimized. To this end, an asymptotically unbiased aggregation scheme is first proposed to ensure zero optimality gap after convergence, and the impact of compression error and transmission outage on the overall training time are quantified through convergence analysis. Then, the formulated problem is solved in an alternating manner, based on which, the novel joint compression and deadline optimization (JCDO) algorithm is derived. Numerical experiments for different use cases in FEEL including image classification and autonomous driving show that the proposed method is nearly 30X faster than the vanilla FedAVG algorithm, and outperforms the state-of-the-art schemes.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics, pp. 1273–1282, PMLR, 2017.
  2. W. Saad, M. Bennis, and M. Chen, “A vision of 6g wireless systems: Applications, trends, technologies, and open research problems,” IEEE Netw., vol. 34, no. 3, pp. 134–142, 2019.
  3. G. Zhu, D. Liu, Y. Du, C. You, J. Zhang, and K. Huang, “Toward an intelligent edge: Wireless communication meets machine learning,” IEEE Commun. Mag., vol. 58, no. 1, pp. 19–25, 2020.
  4. J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar, “signSGD: Compressed optimisation for non-convex problems,” in Proc. International Conference on Machine Learning (ICLR), pp. 560–569, Vancouver, Canada, 2018.
  5. G. Zhu, Y. Du, D. Gündüz, and K. Huang, “One-bit over-the-air aggregation for communication-efficient federated edge learning: Design and convergence analysis,” IEEE Transactions on Wireless Communications, vol. 20, no. 3, pp. 2120–2135, 2020.
  6. S. P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi, “Error feedback fixes signsgd and other gradient compression schemes,” in Proc. International Conference on Machine Learning (ICML), pp. 3252–3261, Long Beach, USA, 2019.
  7. S. Zheng, C. Shen, and X. Chen, “Design and analysis of uplink and downlink communications for federated learning,” IEEE J. Sel. Areas Commun., vol. 39, no. 7, pp. 2150–2167, 2020.
  8. W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, “Terngrad: Ternary gradients to reduce communication in distributed deep learning,” Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 30, Long Beach, Canada, 2017.
  9. D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communication-efficient SGD via gradient quantization and encoding,” Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 30, Long Beach, Canada, 2017.
  10. Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” [Online]. Available: https://arxiv.org/abs/1712.01887, 2017.
  11. S. U. Stich, J.-B. Cordonnier, and M. Jaggi, “Sparsified SGD with memory,” Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 31, Montréal, Canada, 2018.
  12. J. Wangni, J. Wang, J. Liu, and T. Zhang, “Gradient sparsification for communication-efficient distributed optimization,” Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 31, Montréal, Canada, 2018.
  13. J. Zhang, N. Li, and M. Dedeoglu, “Federated learning over wireless networks: A band-limited coordinated descent approach,” in Proc. IEEE International Conference on Computer Communications (INFOCOM), pp. 1–10, Vancouver, Canada, 2021.
  14. B. Isik, F. Pase, D. Gunduz, T. Weissman, and M. Zorzi, “Sparse random networks for communication-efficient federated learning,” [Online]. Available: https://arxiv.org/abs/2209.15328, 2022.
  15. D. Wen, K.-J. Jeon, and K. Huang, “Federated dropout—a simple approach for enabling federated learning on resource constrained devices,” IEEE Wireless Commun. Lett., vol. 11, no. 5, pp. 923–927, 2022.
  16. A. Panda, S. Mahloujifar, A. N. Bhagoji, S. Chakraborty, and P. Mittal, “Sparsefed: Mitigating model poisoning attacks in federated learning with sparsification,” in Proc. International Conference on Artificial Intelligence and Statistics, pp. 7587–7624, 2022.
  17. S. Li, Q. Qi, J. Wang, H. Sun, Y. Li, and F. R. Yu, “Ggs: General gradient sparsification for federated learning in edge computing,” in Proc. IEEE International Conference on Communications (ICC), pp. 1–7, Dublin, Ireland, 2020.
  18. E. Ozfatura, K. Ozfatura, and D. Gündüz, “Time-correlated sparsification for communication-efficient federated learning,” in Proc. IEEE International Symposium on Information Theory (ISIT), pp. 461–466, Melbourne, Australia, 2021.
  19. J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” [Online]. Available: https://arxiv.org/abs/1610.05492, 2016.
  20. N. Shlezinger, M. Chen, Y. C. Eldar, H. V. Poor, and S. Cui, “Uveqfed: Universal vector quantization for federated learning,” IEEE Trans. Signal Process., vol. 69, pp. 500–514, 2020.
  21. Y. Oh, N. Lee, Y.-S. Jeon, and H. V. Poor, “Communication-efficient federated learning via quantized compressed sensing,” [Online]. Available: https://arxiv.org/abs/2111.15071, 2021.
  22. Y.-S. Jeon, M. M. Amiri, J. Li, and H. V. Poor, “A compressive sensing approach for federated learning over massive mimo communication systems,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 1990–2004, 2020.
  23. N. Mitchell, J. Ballé, Z. Charles, and J. Konečnỳ, “Optimizing the communication-accuracy trade-off in federated learning with rate-distortion theory,” [Online]. Available: https://arxiv.org/abs/2201.02664, 2022.
  24. L. Liu, J. Zhang, S. Song, and K. B. Letaief, “Hierarchical federated learning with quantization: Convergence analysis and system design,” accepted to appear in IEEE Trans. Wireless Commun., 2022.
  25. A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization,” in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 2021–2031, Palermo, Italy, 2020.
  26. D. Jhunjhunwala, A. Gadhikar, G. Joshi, and Y. C. Eldar, “Adaptive quantization of model updates for communication-efficient federated learning,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3110–3114, Toronto, Canada, 2021.
  27. C. Shen, J. Xu, S. Zheng, and X. Chen, “Resource rationing for wireless federated learning: concept, benefits, and challenges,” IEEE Commun. Mag., vol. 59, no. 5, pp. 82–87, 2021.
  28. P. Han, S. Wang, and K. K. Leung, “Adaptive gradient sparsification for efficient federated learning: An online learning approach,” in Proc. IEEE International Conference on Distributed Computing Systems (ICDCS), pp. 300–310, Singapore, 2020.
  29. M. Chen, D. Gündüz, K. Huang, W. Saad, M. Bennis, A. V. Feljan, and H. V. Poor, “Distributed learning in wireless networks: Recent progress and future challenges,” IEEE J. Sel. Areas Commun., 2021.
  30. Z. Chai, Y. Chen, A. Anwar, L. Zhao, Y. Cheng, and H. Rangwala, “FedAT: a high-performance and communication-efficient federated learning system with asynchronous tiers,” in Proc. of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–16, St. Louis, MO, 2021.
  31. M. Li, Y. Chen, Y. Wang, and Y. Pan, “Efficient asynchronous vertical federated learning via gradient prediction and double-end sparse compression,” in Proc. IEEE International Conference on Control, Automation, Robotics and Vision (ICARCV), pp. 291–296, Shenzhen, China, 2020.
  32. C. Xie, S. Koyejo, and I. Gupta, “Asynchronous federated optimization,” [Online]. Available: https://arxiv.org/abs/1903.03934, 2019.
  33. T. Nishio and R. Yonetani, “Client selection for federated learning with heterogeneous resources in mobile edge,” in Proc. IEEE international conference on communications (ICC), pp. 1–7, Shanghai, China, 2019.
  34. A. Reisizadeh, I. Tziotis, H. Hassani, A. Mokhtari, and R. Pedarsani, “Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity,” [Online]. Available: https://arxiv.org/abs/2012.14453, 2020.
  35. A. Reisizadeh, H. Taheri, A. Mokhtari, H. Hassani, and R. Pedarsani, “Robust and communication-efficient collaborative learning,” Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 32, Vancouver, Canada, 2019.
  36. J. Lee, H. Ko, and S. Pack, “Adaptive deadline determination for mobile device selection in federated learning,” IEEE Trans. Veh. Technol., vol. 71, no. 3, pp. 3367–3371, 2021.
  37. M. Chen, Z. Yang, W. Saad, C. Yin, H. V. Poor, and S. Cui, “A joint learning and communications framework for federated learning over wireless networks,” IEEE Trans. Wireless Commun., vol. 20, no. 1, pp. 269–283, 2020.
  38. Y. Wang, Y. Xu, Q. Shi, and T.-H. Chang, “Quantized federated learning under transmission delay and outage constraints,” IEEE J. Sel. Areas Commun., vol. 40, no. 1, pp. 323–341, 2021.
  39. L. Cui, X. Su, Y. Zhou, and J. Liu, “Optimal rate adaption in federated learning with compressed communications,” in Proc. IEEE Conference on Computer Communications (INFOCOM), pp. 1459–1468, 2022.
  40. F. Lai, X. Zhu, H. V. Madhyastha, and M. Chowdhury, “Oort: Efficient federated learning via guided participant selection,” in USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 19–35, 2021.
  41. M. Zhang, G. Zhu, S. Wang, J. Jiang, Q. Liao, C. Zhong, and S. Cui, “Communication-efficient federated edge learning via optimal probabilistic device scheduling,” IEEE Trans. Wireless Commun., vol. 21, no. 10, pp. 8536–8551, 2022.
  42. J. Ren, Y. He, D. Wen, G. Yu, K. Huang, and D. Guo, “Scheduling in Cellular Federated Edge Learning with Importance and Channel Awareness,” [Online]. Available: https://arxiv.org/abs/2004.00490/, 2020.
  43. N. H. Tran, W. Bao, A. Zomaya, M. N. Nguyen, and C. S. Hong, “Federated learning over wireless networks: Optimization model design and analysis,” in Proc. IEEE Conference on Computer Communications (INFOCOM), pp. 1387–1395, Paris, France, 2019.
  44. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, Las Vegas, USA, 2016.
  45. S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečnỳ, H. B. McMahan, V. Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,” [Online]. Available: https://arxiv.org/abs/1812.01097, 2018.
  46. A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images,” 2009.
  47. A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361, Providence, USA, 2012.
  48. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” [Online]. Available: https://arxiv.org/abs/1409.1556, 2014.
  49. Z. Jiang, Y. Xu, H. Xu, Z. Wang, and C. Qian, “Adaptive control of client selection and gradient compression for efficient federated learning,” in Proc. IEEE Conference on Computer Communications (INFOCOM), New York, USA, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Maojun Zhang (25 papers)
  2. Yang Li (1142 papers)
  3. Dongzhu Liu (15 papers)
  4. Richeng Jin (32 papers)
  5. Guangxu Zhu (88 papers)
  6. Caijun Zhong (84 papers)
  7. Tony Q. S. Quek (237 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now