Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
96 tokens/sec
Gemini 2.5 Pro Premium
44 tokens/sec
GPT-5 Medium
18 tokens/sec
GPT-5 High Premium
18 tokens/sec
GPT-4o
105 tokens/sec
DeepSeek R1 via Azure Premium
83 tokens/sec
GPT OSS 120B via Groq Premium
475 tokens/sec
Kimi K2 via Groq Premium
259 tokens/sec
2000 character limit reached

Asynchronous Online Federated Learning with Reduced Communication Requirements (2303.15226v2)

Published 27 Mar 2023 in cs.LG, cs.DC, math.OC, and stat.ML

Abstract: Online federated learning (FL) enables geographically distributed devices to learn a global shared model from locally available streaming data. Most online FL literature considers a best-case scenario regarding the participating clients and the communication channels. However, these assumptions are often not met in real-world applications. Asynchronous settings can reflect a more realistic environment, such as heterogeneous client participation due to available computational power and battery constraints, as well as delays caused by communication channels or straggler devices. Further, in most applications, energy efficiency must be taken into consideration. Using the principles of partial-sharing-based communications, we propose a communication-efficient asynchronous online federated learning (PAO-Fed) strategy. By reducing the communication overhead of the participants, the proposed method renders participation in the learning task more accessible and efficient. In addition, the proposed aggregation mechanism accounts for random participation, handles delayed updates and mitigates their effect on accuracy. We prove the first and second-order convergence of the proposed PAO-Fed method and obtain an expression for its steady-state mean square deviation. Finally, we conduct comprehensive simulations to study the performance of the proposed method on both synthetic and real-life datasets. The simulations reveal that in asynchronous settings, the proposed PAO-Fed is able to achieve the same convergence properties as that of the online federated stochastic gradient while reducing the communication overhead by 98 percent.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. F. Gauthier, V. C. Gogineni, S. Werner, Y.-F. Huang, and A. Kuh, “Resource-aware asynchronous online federated learning for nonlinear regression,” arXiv preprint arXiv:2111.13931, 2021.
  2. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: challenges, methods, and future directions,” IEEE Signal Process. Mag., vol. 37, no. 3, pp. 50–60, May 2020.
  3. J. Konečnỳ, H. B. McMahan, D. Ramage, and P. Richtárik, “Federated optimization: distributed machine learning for on-device intelligence,” arXiv preprint arXiv:1610.02527, Oct. 2016.
  4. O. Dekel, P. M. Long, and Y. Singer, “Online multitask learning,” in Int. Conf. Comput. Learn. Theory, 2006, pp. 453–467.
  5. L. Li, Y. Fan, M. Tse, and K.-Y. Lin, “A review of applications in federated learning,” Computers & Ind. Eng., vol. 149, p. 106854, 2020.
  6. T. Zhang, L. Gao, C. He, M. Zhang, B. Krishnamachari, and A. S. Avestimehr, “Federated learning for the internet of things: applications, challenges, and opportunities,” IEEE Internet Things Mag., vol. 5, no. 1, pp. 24–29, May 2022.
  7. Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” arXiv preprint arXiv:1806.00582, 2018.
  8. Z. Yang, M. Chen, W. Saad, C. S. Hong, and M. Shikh-Bahaei, “Energy Efficient Federated Learning Over Wireless Communication Networks,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 1935–1949, Mar. 2021.
  9. Z. Zhao, C. Feng, W. Hong, J. Jiang, C. Jia, T. Q. S. Quek, and M. Peng, “Federated Learning With Non-IID Data in Wireless Networks,” IEEE Trans. Wireless Commun., vol. 21, no. 3, pp. 1927–1942, Mar. 2022.
  10. E. Ozfatura, K. Ozfatura, and D. Gündüz, “FedADC: accelerated federated learning with drift control,” in Proc. IEEE Int. Symp. Inf. Theory, Jul. 2021, pp. 467–472.
  11. X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” arXiv preprint arXiv:1907.02189, Jul. 2019.
  12. J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: strategies for improving communication efficiency,” arXiv preprint arXiv:1610.05492, 2016.
  13. Z. Yang, M. Chen, W. Saad, C. S. Hong, and M. Shikh-Bahaei, “Energy efficient federated learning over wireless communication networks,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 1935–1949, Nov. 2020.
  14. Z. Lian, W. Wang, and C. Su, “COFEL: Communication-efficient and optimized federated learning with local differential privacy,” in Proc. IEEE Int. Conf. Commun., Jun. 2021, pp. 1–6.
  15. Y. Lu, Z. Liu, and Y. Huang, “Parameters compressed mechanism in federated learning for edge computing,” in Proc. IEEE Int. Conf. Cyber Secur. Cloud Comput., Jun. 2021, pp. 161–166.
  16. X. Fan, Y. Wang, Y. Huo, and Z. Tian, “Communication-efficient federated learning through 1-bit compressive sensing and analog aggregation,” in Proc. IEEE Int. Conf. Commun. Workshops, 2021, pp. 1–6.
  17. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” Artificial intel. statis., pp. 1273–1282, Apr. 2017.
  18. Y. Chen, Z. Chai, Y. Cheng, and H. Rangwala, “Asynchronous federated learning for sensor data with concept drift,” arXiv preprint arXiv:2109.00151, Sep. 2021.
  19. C. Xie, S. Koyejo, and I. Gupta, “Asynchronous federated optimization,” arXiv preprint arXiv:1903.03934, Mar. 2019.
  20. Y. Chen, Y. Ning, M. Slawski, and H. Rangwala, “Asynchronous online federated learning for edge devices with non-iid data,” in Proc. IEEE Int. Conf. Big Data, Dec. 2020, pp. 15–24.
  21. Z. Wang, Z. Zhang, Y. Tian, Q. Yang, H. Shan, W. Wang, and T. Q. Quek, “Asynchronous Federated Learning over Wireless Communication Networks,” IEEE Trans. Wireless Commun., Mar. 2022.
  22. Z. Chai, Y. Chen, L. Zhao, Y. Cheng, and H. Rangwala, “Fedat: a communication-efficient federated learning method with asynchronous tiers under non-iid data,” arXiv preprint arXiv:2010.05958, Oct. 2020.
  23. Y. Chen, X. Sun, and Y. Jin, “Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 10, pp. 4229–4238, Dec. 2019.
  24. Z. Wang, Z. Zhang, and J. Wang, “Asynchronous federated learning over wireless communication networks,” in Proc. IEEE Int. Conf. Commun., Jun. 2021, pp. 1–7.
  25. X. Qiu, T. Parcollet, D. J. Beutel, T. Topal, A. Mathur, and N. D. Lane, “Can federated learning save the planet?” arXiv preprint arXiv:2010.06537, Oct. 2020.
  26. V. C. Gogineni, S. Werner, Y.-F. Huang, and A. Kuh, “Communication-efficient online federated learning framework for nonlinear regression,” IEEE Int. Conf. Acoust., Speech and Signal Process., May 2022.
  27. “Robust and communication-efficient federated learning from non-i.i.d. data,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 9, pp. 3400–3413, Sep. 2020.
  28. J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv preprint arXiv:1610.05492, Oct. 2016.
  29. M. Chen, N. Shlezinger, H. V. Poor, Y. C. Eldar, and S. Cui, “Communication-efficient federated learning,” Proc. Nat. Acad. Sciences, vol. 118, no. 17, Apr. 2021.
  30. Z. Chai, A. Ali, S. Zawad, S. Truex, A. Anwar, N. Baracaldo, Y. Zhou, H. Ludwig, F. Yan, and Y. Cheng, “Tifl: A tier-based federated learning system,” Jun. 2020, pp. 125–136.
  31. X. Zhang, Y. Liu, J. Liu, A. Argyriou, and Y. Han, “D2D-Assisted Federated Learning in Mobile Edge Computing Networks,” Mar. 2021, pp. 1–7.
  32. W. Wu, L. He, W. Lin, R. Mao, C. Maple, and S. Jarvis, “SAFA: a semi-asynchronous protocol for fast federated learning with low overhead,” IEEE Trans. Computers, vol. 70, no. 5, pp. 655–668, May 2020.
  33. R. Arablouei, K. Doğançay, S. Werner, and Y.-F. Huang, “Adaptive distributed estimation based on recursive least-squares and partial diffusion,” IEEE Trans. Signal Process., vol. 62, no. 14, pp. 3510–3522, Jul. 2014.
  34. P. Bouboulis, S. Pougkakiotis, and S. Theodoridis, “Efficient KLMS and KRLS algorithms: a random Fourier feature perspective,” in Proc. IEEE Stat. Signal Process. Workshop, Jun. 2016, pp. 1–5.
  35. A. Rahimi, B. Recht et al., “Random features for large-scale kernel machines.” in Proc. Conf. on Neural Inf. Proc. Syst., vol. 3, no. 4, Dec. 2007, pp. 1–5.
  36. W. Liu, P. P. Pokharel, and J. C. Principe, “The kernel least-mean-square algorithm,” IEEE Trans. Signal Process., vol. 56, no. 2, pp. 543–554, Jan. 2008.
  37. V. C. Gogineni, V. R. Elias, W. A. Martins, and S. Werner, “Graph diffusion kernel LMS using random Fourier features,” 54th Asilomar Conf. Signals, Syst., Computers, pp. 1528–1532, Nov. 2020.
  38. R. D. Yates, Y. Sun, D. R. Brown, S. K. Kaul, E. Modiano, and S. Ulukus, “Age of information: an introduction and survey,” IEEE J. Sel. Areas Commun., vol. 39, no. 5, pp. 1183–1210, Mar. 2021.
  39. H. H. Yang, A. Arafa, T. Q. Quek, and H. V. Poor, “Age-based scheduling policy for federated learning in mobile edge networks,” in Proc. IEEE Int. Conf. Acoust., Speech and Signal Process., May 2020, pp. 8743–8747.
  40. C.-H. Hu, Z. Chen, and E. G. Larsson, “Scheduling and aggregation design for asynchronous federated learning over wireless networks,” IEEE J. Sel. Areas Commun., vol. 41, no. 4, pp. 874–886, Jul. 2023.
  41. V. C. Gogineni, S. P. Talebi, and S. Werner, “Performance of clustered multitask diffusion lms suffering from inter-node communication delays,” IEEE Trans. on Circuits and Syst. II: Express Briefs, vol. 68, no. 7, pp. 2695–2699, 2021.
  42. R. H. Koning, H. Neudecker, and T. Wansbeek, “Block Kronecker products and the vecb operator,” Linear algebra and its applications, vol. 149, pp. 165–184, Apr. 1991.
  43. S. Dane, “CalCOFI, Over 60 years of oceanographic data,” available at: https://www.kaggle.com/sohier/calcofi?select=bottle.csv.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now