Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Survey on Model-heterogeneous Federated Learning: Problems, Methods, and Prospects (2312.12091v2)

Published 19 Dec 2023 in cs.DC

Abstract: As privacy concerns continue to grow, federated learning (FL) has gained significant attention as a promising privacy-preserving technology, leading to considerable advancements in recent years. Unlike traditional machine learning, which requires central data collection, FL keeps data localized on user devices. However, conventional FL assumes that all clients operate with identical model structures initialized by the server. In real-world applications, system heterogeneity is common, with clients possessing varying computational capabilities. This disparity can hinder training for resource-limited clients and result in inefficient resource use for those with greater processing power. To address this challenge, model-heterogeneous FL has been introduced, enabling clients to train models of varying complexity based on their hardware resources. This paper reviews state-of-the-art approaches in model-heterogeneous FL, analyzing their strengths and weaknesses, while identifying open challenges and future research directions. To the best of our knowledge, this is the first survey to specifically focus on model-heterogeneous FL.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (30)
  1. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics.   PMLR, 2017, pp. 1273–1282.
  2. A. Reisizadeh, I. Tziotis, H. Hassani, A. Mokhtari, and R. Pedarsani, “Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity,” IEEE JASC, vol. 3, no. 2, pp. 197–205, 2022.
  3. D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, and H. Vincent Poor, “Federated learning for internet of things: A comprehensive survey,” IEEE Commun Surv Tutor, vol. 23, no. 3, pp. 1622–1658, 2021.
  4. P. Boobalan, S. P. Ramu, Q.-V. Pham, K. Dev, S. Pandya, P. K. R. Maddikunta, T. R. Gadekallu, and T. Huynh-The, “Fusion of federated learning and industrial internet of things: A survey,” Computer Networks, vol. 212, p. 109048, 2022.
  5. A. Imteaj, U. Thakker, S. Wang, J. Li, and M. H. Amini, “A survey on federated learning for resource-constrained iot devices,” IEEE IoT-J, vol. 9, no. 1, pp. 1–24, 2022.
  6. K. Pfeiffer, M. Rapp, R. Khalili, and J. Henkel, “Federated learning for computationally constrained heterogeneous devices: A survey,” ACM Comput. Surv., vol. 55, no. 14s, jul 2023.
  7. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
  8. J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” IJCV, vol. 129, pp. 1789–1819, 2021.
  9. D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” arXiv preprint arXiv:1910.03581, 2019.
  10. J. Zhang, S. Guo, X. Ma, H. Wang, W. Xu, and F. Wu, “Parameterized knowledge transfer for personalized federated learning,” NeurIPS, vol. 34, pp. 10 092–10 104, 2021.
  11. T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” NeurIPS, vol. 33, pp. 2351–2363, 2020.
  12. Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in ICML, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139.   PMLR, 18–24 Jul 2021, pp. 12 878–12 889.
  13. L. Zhang, D. Wu, and X. Yuan, “Fedzkt: Zero-shot knowledge transfer towards resource-constrained federated learning with heterogeneous on-device models,” in IEEE ICDCS 2024, 2022, pp. 928–938.
  14. L. Zhang, L. Shen, L. Ding, D. Tao, and L.-Y. Duan, “Fine-tuning global model via data-free knowledge distillation for non-iid federated learning,” in CVPR, June 2022, pp. 10 174–10 183.
  15. H. Chen, H. Vikalo et al., “The best of both worlds: Accurate global and personalized models through federated learning with data-free hyper-knowledge distillation,” arXiv preprint arXiv:2301.08968, 2023.
  16. Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “Fedproto: Federated prototype learning across heterogeneous clients,” in AAAI, vol. 36, no. 8, 2022, pp. 8432–8440.
  17. D. Makhija, X. Han, N. Ho, and J. Ghosh, “Architecture agnostic federated learning for neural networks,” in ICML.   PMLR, 2022, pp. 14 860–14 870.
  18. J. R. Regatti, S. Lu, A. Gupta, and N. Shroff, “Conditional moment alignment for improved generalization in federated learning,” in FL Workshop in NeurIPS 2022), 2022.
  19. E. Diao, J. Ding, and V. Tarokh, “Heterofl: Computation and communication efficient federated learning for heterogeneous clients,” in ICLR, 2020.
  20. S. Alam, L. Liu, M. Yan, and M. Zhang, “Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction,” NeurIPS, vol. 35, pp. 29 677–29 690, 2022.
  21. A. Li, J. Sun, P. Li, Y. Pu, H. Li, and Y. Chen, “Hermes: An efficient federated learning framework for heterogeneous mobile clients,” ser. MobiCom ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 420–437.
  22. Y. Niu, S. Prakash, S. Kundu, S. Lee, and S. Avestimehr, “Federated learning of large models at the edge via principal sub-model training,” arXiv preprint arXiv:2208.13141, 2022.
  23. A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li, “Lotteryfl: Empower edge intelligence with personalized and communication-efficient federated learning,” in 2021 IEEE/ACM SEC, 2021, pp. 68–79.
  24. Y. Jiang, S. Wang, V. Valls, B. J. Ko, W.-H. Lee, K. K. Leung, and L. Tassiulas, “Model pruning enables efficient federated learning on edge devices,” IEEE Trans. Neural Netw. Learn. Syst., pp. 1–13, 2022.
  25. O. Litany, H. Maron, D. Acuna, J. Kautz, G. Chechik, and S. Fidler, “Federated learning with heterogeneous architectures using graph hypernetworks,” arXiv preprint arXiv:2201.08459, 2022.
  26. M. Rapp, R. Khalili, K. Pfeiffer, and J. Henkel, “Distreal: Distributed resource-aware learning in heterogeneous systems,” in AAAI, vol. 36, no. 7, 2022, pp. 8062–8071.
  27. D. Yao, W. Pan, M. J. O’Neill, Y. Dai, Y. Wan, H. Jin, and L. Sun, “Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization,” arXiv preprint arXiv:2111.14655, 2021.
  28. C. Ma, J. Li, M. Ding, H. H. Yang, F. Shu, T. Q. S. Quek, and H. V. Poor, “On safeguarding privacy and security in the framework of federated learning,” IEEE Network, vol. 34, no. 4, pp. 242–248, 2020.
  29. M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to Byzantine-Robust federated learning,” in USENIX Security 20, Aug. 2020, pp. 1605–1622.
  30. C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun, “Splitfed: When federated learning meets split learning,” in AAAI, vol. 36, no. 8, 2022, pp. 8485–8493.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Boyu Fan (9 papers)
  2. Siyang Jiang (8 papers)
  3. Xiang Su (14 papers)
  4. Pan Hui (155 papers)
  5. Sasu Tarkoma (58 papers)
Citations (4)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets