Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Massive Digital Over-the-Air Computation for Communication-Efficient Federated Edge Learning (2405.15969v2)

Published 24 May 2024 in cs.IT, eess.SP, and math.IT

Abstract: Over-the-air computation (AirComp) is a promising technology converging communication and computation over wireless networks, which can be particularly effective in model training, inference, and more emerging edge intelligence applications. AirComp relies on uncoded transmission of individual signals, which are added naturally over the multiple access channel thanks to the superposition property of the wireless medium. Despite significantly improved communication efficiency, how to accommodate AirComp in the existing and future digital communication networks, that are based on discrete modulation schemes, remains a challenge. This paper proposes a massive digital AirComp (MD-AirComp) scheme, that leverages an unsourced massive access protocol, to enhance compatibility with both current and next-generation wireless networks. MD-AirComp utilizes vector quantization to reduce the uplink communication overhead, and employs shared quantization and modulation codebooks. At the receiver, we propose a near-optimal approximate message passing-based algorithm to compute the model aggregation results from the superposed sequences, which relies on estimating the number of devices transmitting each code sequence, rather than trying to decode the messages of individual transmitters. We apply MD-AirComp to the federated edge learning (FEEL), and show that it significantly accelerates FEEL convergence compared to state-of-the-art while using the same amount of communication resources. To support further research and ensure reproducibility, we have made our code available at https://github.com/liqiao19/MD-AirComp.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. L. Qiao, Z. Gao, Z. Li, and D. Gündüz, “Unsourced massive access-based digital over-the-air computation for efficient federated edge learning,” in Proc. IEEE Int. Symp. Inform. Theory (ISIT), Taipei, Taiwan, 2023, pp. 2003-2008.
  2. X. Chen, D. W. K. Ng, W. Yu, E. G. Larsson, N. Al-Dhahir, and R. Schober, “Massive access for 5G and beyond,” IEEE J. Select. Areas Commun., vol. 39, no. 3, pp. 615-637, Mar. 2021.
  3. D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, and H. Vincent Poor, “Federated learning for Internet of Things: A comprehensive survey,” IEEE Commun. Surv. Tut., vol. 23, no. 3, pp. 1622-1658, thirdquarter 2021.
  4. G. Scutari, F. Facchinei, L. Lampariello, S. Sardellitti, and P. Song, “Parallel and distributed methods for constrained nonconvex optimization-Part II: Applications in communications and machine learning,” IEEE Trans. Signal Process., vol. 65, no. 8, pp. 1945-1960, Apr. 2017.
  5. D. Gündüz, D. B. Kurka, M. Jankowski, M. M. Amiri, E. Ozfatura, and S. Sreekumar, “Communicate to learn at the edge,” IEEE Commun. Mag., vol. 58, no. 12, pp. 14-19, Dec. 2020.
  6. M. B. Mashhadi, M. Mahdavimoghadam, R. Tafazolli, and W. Saad, “Collaborative learning with a drone orchestrator,” IEEE Trans. Veh. Technol., doi: 10.1109/TVT.2023.3303630.
  7. M. M. Amiri and D. Gündüz, “Federated learning over wireless fading channels”, IEEE Trans. Wireless Commun., vol. 19, no. 5, pp. 3546-3557, May 2020.
  8. G. Zhu, Y. Wang, and K. Huang, “Broadband analog aggregation for low-latency federated edge learning,” IEEE Trans. Wireless Commun., vol. 19, no. 1, pp. 491-506, Jan. 2020.
  9. M. M. Amiri and D. Gündüz, “Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air”, IEEE Trans. Signal Process., vol. 68, pp. 2155-2169, Mar. 2020.
  10. Y. Sun, S. Zhou, Z. Niu, and D. Gündüz, “Time-correlated sparsification for efficient over-the-air model aggregation in wireless federated learning,” in Proc. IEEE Int. Conf. on Commun. (ICC), Seoul, Republic of Korea, 2022, pp. 3388-3393.
  11. K. Yang, T. Jiang, Y. Shi, and Z. Ding, “Federated learning via over-the-air computation”, IEEE Trans. Wireless Commun., vol. 19, no. 3, pp. 2022-2035, Mar. 2020.
  12. Y. -S. Jeon, M. M. Amiri, J. Li, and H. V. Poor, “A compressive sensing approach for federated learning over massive MIMO communication systems,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 1990-2004, Mar. 2021.
  13. Y. Shao, D. Gündüz, and S. C. Liew, “Bayesian over-the-air computation,” IEEE J. Select. Areas Commun., vol. 41, no. 3, pp. 589-606, Mar. 2023.
  14. Y. Shao, D. Gündüz, and S. C. Liew, “Federated edge learning with misaligned over-the-air computation,” IEEE Trans. Wireless Commun., vol. 21, no. 6, pp. 3951-3964, Jun. 2022.
  15. Y. Sun, S. Zhou, Z. Niu, and D. Gündüz, “Dynamic scheduling for over-the-air federated edge learning with energy constraints,” IEEE J. Sel. Areas Commun., vol. 40, no. 1, pp. 227-242, Jan. 2022.
  16. M. Chen, D. Gündüz, K. Huang, W. Saad, M. Bennis, A. V. Feljan, and H. V. Poor, “Distributed learning in wireless networks: Recent progress and future challenges,” IEEE J. Sel. Areas Commun., vol. 39, no. 12, pp. 3579-3605, Dec. 2021.
  17. A. Şahin and R. Yang, “A survey on over-the-air computation,” IEEE Commun. Surv. Tutor., vol. 25, no. 3, pp. 1877-1908, thirdquarter 2023.
  18. A. S. Berahas, R. Bollapragada, N. S. Keskar, and E. Wei, “Balancing communication and computation in distributed optimization,” IEEE Trans. Automat. Contr., vol. 64, no. 8, pp. 3141-3155, Aug. 2019.
  19. S. F. Yilmaz, B. Hasırcıoğlu, and D. Gündüz, “Over-the-air ensemble inference with model privacy”, in Proc. IEEE Int. Symp. Inf. Theory (ISIT), June 2022.
  20. Z. Liu, Q. Lan, A. E. Kalør, P. Popovski, and K. Huang, “Over-the-air multi-view pooling for distributed sensing”. ArXiv preprint arXiv:2302.09771.
  21. 3GPP, “5G NR overall description Stage-2,” 3rd Generation Partnership Project (3GPP), TS 38.300 V15.3.1, Oct. 2018.
  22. D. Jhunjhunwala, A. Gadhikar, G. Joshi, and Y. C. Eldar, “Adaptive quantization of model updates for communication-efficient federated learning,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), Toronto, ON, Canada, 2021.
  23. L. Qu, S. Song, and C. -Y. Tsui, “FedDQ: Communication-efficient federated learning with descending quantization,” in Proc. Global Commun. Conf. (Globecom), Rio de Janeiro, Brazil, 2022, pp. 281-286.
  24. Y. Wang, Y. Xu, Q. Shi, and T.-H. Chang, “Quantized federated learning under transmission delay and outage constraints,” IEEE J. Select. Areas Commun., vol. 40, no. 1, pp. 323-341, Jan. 2022.
  25. N. Shlezinger, M. Chen, Y. C. Eldar, H. V. Poor, and S. Cui, “UVeQFed: Universal vector quantization for federated learning,” IEEE Trans. Signal Process., vol. 69, pp. 500-514, 2021.
  26. Y. Oh, Y. -S. Jeon, M. Chen and W. Saad, “FedVQCS: Federated learning via vector quantized compressed sensing,” IEEE Trans. Wireless Commun., doi: 10.1109/TWC.2023.3291877.
  27. G. Zhu, Y. Du, D. Gündüz, and K. Huang, “One-bit over-the-air aggregation for communication-efficient federated edge learning: Design and convergence analysis,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 2120-2135, Mar. 2021.
  28. A. Şahin, “Distributed learning over a wireless network with non-coherent majority vote computation,” IEEE Trans. Wireless Commun.,, vol. 22, no. 11, pp. 8020-8034, Nov. 2023.
  29. A. Şahin, “Over-the-air computation based on balanced number systems for federated edge learning,” IEEE Trans. Wireless Commun., doi: 10.1109/TWC.2023.3320116.
  30. S. Razavikia, J. M. B. Da Silva, and C. Fischione, “ChannelComp: A general method for computation by communications,” IEEE Trans. Commun., doi: 10.1109/TCOMM.2023.3324999.
  31. ——, “SumComp coding: Digital over-the-air computation via the ring of integers,” arXiv preprint arXiv:2310.20504, 2023.
  32. ——, “Blind federated learning via over-the-air q-QAM,” arXiv preprint arXiv:2311.04253, 2023.
  33. L. Liu and W. Yu, “Massive connectivity with massive MIMO—Part I: Device activity detection and channel estimation,” IEEE Trans. Signal Process., vol. 66, no. 11, pp. 2933-2946, Jun. 2018.
  34. Z. Chen, F. Sohrabi, Y.-F. Liu, and W. Yu, “Phase transition analysis for covariance based massive random access with massive MIMO,” IEEE Trans. Inf. Theory, vol. 68, no. 3, pp. 1696-1715, Mar. 2022.
  35. Z. Wang, Y.-F. Liu, and L. Liu, “Covariance-based joint device activity and delay detection in asynchronous mMTC,” IEEE Signal Process. Lett., vol. 29, pp. 538–542, Jan. 2022.
  36. Y. Polyanskiy, “A perspective on massive random-access”, in Proc. IEEE Int. Symp. Inform. Theory (ISIT), pp. 1-5, Jun. 2017.
  37. K. -H. Ngo, G. Durisi, A. G. i. Amat, P. Popovski, A. E. Kalør, and B. Soret, “Unsourced multiple access with common alarm messages: Network slicing for massive and critical IoT,” IEEE Trans. Wireless Commun.,, vol. 72, no. 2, pp. 907-923, Feb. 2024.
  38. X. Shao, X. Chen, D. W. K. Ng, C. Zhong, and Z. Zhang, “Cooperative activity detection: Sourced and unsourced massive random access paradigms,” IEEE Trans. Signal Process., vol. 68, pp. 6578-6593, 2020.
  39. J. Hamer, M. Mohri, and A. T. Suresh, “Fedboost: A communication-efficient algorithm for federated learning”, in Proc. Int. Conf. Mach. Learn. (ICML), 2020, pp. 3973-3983.
  40. T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Proc. Mach. Learn. Syst., 2020, pp. 429-450.
  41. J. Wang, Q. Liu, H. Liang, G. Joshi, and V. H. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2020, pp. 7611-7623.
  42. S. U. Stich, J. B. Cordonnier, and M. Jaggi, “Sparsified SGD with memory,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), pp. 4452-4463, Dec. 2018.
  43. R. J. Baxley, B. T. Walkenhorst, and G. Acosta-Marum, “Complex gaussian ratio distribution with applications for error rate calculation in fading channels with imperfect CSI,” in Proc. Global Commun. Conf. (Globecom), Miami, FL, USA, 2010, pp. 1-5.
  44. D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,” Proc. Nat. Acad. Sci. USA, vol. 106, no. 45, pp. 18914-18919, 2009.
  45. X. Meng, S. Wu, L. Kuang, and J. Lu, “An expectation propagation perspective on approximate message passing,” IEEE Signal Process. Lett., vol. 22, no. 8, pp. 1194-1197, Aug. 2015.
  46. F. Krzakala, M. Mézard, F. Sausset, Y. Sun, and L. Zdeborová, “Probabilistic reconstruction in compressed sensing: Algorithms, phase diagrams, threshold achieving matrices,” J. Stat. Mech., Theor. Exp., vol. 2012, no. 8, p. P08009, Aug. 2012.
  47. L. Liu, S. Huang, and B. M. Kurkoski, “Memory AMP,” IEEE Trans. Inf. Theory, vol. 68, no. 12, pp. 8015-8039, Dec. 2022.
  48. Y. Wang, L. Lin, and J. Chen, “Communication-efficient adaptive federated learning”, in Proc. Int. Conf. Mach. Learn. (ICML), 2022, pp. 22802-22838.
  49. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, 2016, pp. 770-778.
  50. D. Arthur and S. Vassilvitskii, “K-means++ the advantages of careful seeding,” Proc. 18th Ann. ACM-SIAM Symp. Discrete Algorithms, pp. 1027-1035, 2007.
  51. I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” 2016, arXiv:1608.03983.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Li Qiao (27 papers)
  2. Zhen Gao (163 papers)
  3. Mahdi Boloursaz Mashhadi (30 papers)
  4. Deniz Gündüz (144 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now