Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CaBaFL: Asynchronous Federated Learning via Hierarchical Cache and Feature Balance (2404.12850v2)

Published 19 Apr 2024 in cs.LG and cs.DC

Abstract: Federated Learning (FL) as a promising distributed machine learning paradigm has been widely adopted in Artificial Intelligence of Things (AIoT) applications. However, the efficiency and inference capability of FL is seriously limited due to the presence of stragglers and data imbalance across massive AIoT devices, respectively. To address the above challenges, we present a novel asynchronous FL approach named CaBaFL, which includes a hierarchical Cache-based aggregation mechanism and a feature Balance-guided device selection strategy. CaBaFL maintains multiple intermediate models simultaneously for local training. The hierarchical cache-based aggregation mechanism enables each intermediate model to be trained on multiple devices to align the training time and mitigate the straggler issue. In specific, each intermediate model is stored in a low-level cache for local training and when it is trained by sufficient local devices, it will be stored in a high-level cache for aggregation. To address the problem of imbalanced data, the feature balance-guided device selection strategy in CaBaFL adopts the activation distribution as a metric, which enables each intermediate model to be trained across devices with totally balanced data distributions before aggregation. Experimental results show that compared with the state-of-the-art FL methods, CaBaFL achieves up to 9.26X training acceleration and 19.71\% accuracy improvements.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Sebastian Caldas et al. 2018b. CoRR abs/1812.01097 (2018). arXiv:1812.01097 http://arxiv.org/abs/1812.01097
  2. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 (2018).
  3. FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling. In 2020 IEEE International Conference on Big Data (Big Data). 5017–5026. https://doi.org/10.1109/BigData50022.2020.9377960
  4. A general theory for federated optimization with asynchronous and heterogeneous clients updates. Journal of Machine Learning Research 24, 110 (2023), 1–43.
  5. An Efficient Framework for Clustered Federated Learning. IEEE Transactions on Information Theory 68, 12 (2022), 8076–8091. https://doi.org/10.1109/TIT.2022.3192506
  6. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  7. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019).
  8. Scheduling and aggregation design for asynchronous federated learning over wireless networks. IEEE Journal on Selected Areas in Communications 41, 4 (2023), 874–886.
  9. Ming Hu et al. 2023b. AIoTML: A Unified Modeling Language for AIoT-Based Cyber-Physical Systems. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD) 40, 11 (2023), 3545–3558.
  10. Ming Hu et al. 2024. FedMut: Generalized Federated Learning via Stochastic Mutation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 12528–12537.
  11. Husnoo et al. 2023. FeDiSa: A Semi-asynchronous Federated Learning Framework for Power System Fault and Cyberattack Discrimination. arXiv preprint arXiv:2303.16956 (2023).
  12. Jin et al. 2022. Personalized edge intelligence via federated self-knowledge distillation. IEEE Transactions on Parallel and Distributed Systems 34, 2 (2022), 567–580.
  13. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning 14, 1–2 (2021), 1–210.
  14. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of International Conference on Machine Learning. 5132–5143.
  15. Federated reinforcement learning: Linear speedup under markovian sampling. In International Conference on Machine Learning. PMLR, 10997–11057.
  16. Consensus control for decentralized deep learning. In International Conference on Machine Learning. PMLR, 5686–5696.
  17. Learning multiple layers of features from tiny images. (2009).
  18. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012).
  19. Lee et al. 2021. Adaptive transmission scheduling in wireless networks for asynchronous federated learning. IEEE Journal on Selected Areas in Communications 39, 12 (2021), 3673–3687.
  20. Sample-level data selection for federated learning. In IEEE INFOCOM 2021-IEEE Conference on Computer Communications. IEEE, 1–10.
  21. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37, 3 (2020), 50–60.
  22. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2 (2020), 429–450.
  23. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning. PMLR, 3043–3052.
  24. Privacy-Aware Access Control in IoT-Enabled Healthcare: A Federated Deep Learning Approach. IEEE Internet of Things Journal 10, 4 (2023), 2893–2902.
  25. Liu et al. 2021. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1013–1023.
  26. FedSA: A semi-asynchronous federated learning mechanism in heterogeneous edge computing. IEEE Journal on Selected Areas in Communications 39, 12 (2021), 3654–3672.
  27. Communication-efficient learning of deep networks from decentralized data. In Proceedings of Artificial intelligence and statistics. 1273–1282.
  28. Federated learning for wireless communications: Motivation, opportunities, and challenges. IEEE Communications Magazine 58, 6 (2020), 46–51.
  29. Wireless network intelligence at the edge. Proc. IEEE 107, 11 (2019), 2204–2239.
  30. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
  31. Sunitha Safavat and Danda B Rawat. 2023. Asynchronous Federated Learning for Intrusion Detection in Vehicular Cyber-Physical Systems. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 1–6.
  32. Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data. IEEE Transactions on Neural Networks and Learning Systems 31, 9 (2020), 3400–3413.
  33. Asynchronous Federated Continual Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5054–5062.
  34. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
  35. Federated Deep Reinforcement Learning for Recommendation-Enabled Edge Caching in Mobile Edge-Cloud Computing Networks. IEEE Journal on Selected Areas in Communications 41, 3 (2023), 690–705.
  36. SAFA: A semi-asynchronous protocol for fast federated learning with low overhead. IEEE Trans. Comput. 70, 5 (2020), 655–668.
  37. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934 (2019).
  38. Jinjun Xiong and Huamin Chen. 2020. Challenges for Building a Cloud Native Scalable and Trustable Multi-Tenant AIoT Platform. In Proc. of International Conference on Computer-Aided Design (ICCAD).
  39. Asynchronous federated learning on heterogeneous devices: A survey. Computer Science Review 50 (2023), 100595.
  40. Distributed learning over unreliable networks. In International Conference on Machine Learning. PMLR, 7202–7212.
  41. Zhang et al. 2021b. A survey on federated learning. Knowledge-Based Systems 216 (2021), 106775.
  42. Zhang et al. 2023a. FedMDS: An Efficient Model Discrepancy-Aware Semi-Asynchronous Clustered Federated Learning Framework. IEEE Transactions on Parallel and Distributed Systems 34, 3 (2023), 1007–1019.
  43. End-to-End Federated Learning for Autonomous Driving Vehicles. In Proceedings of International Joint Conference on Neural Networks (IJCNN). 1–8.
  44. Efficient federated learning for cloud-based AIoT applications. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 40, 11 (2020), 2211–2223.
  45. Communication-efficient federated continual learning for distributed learning system with Non-IID data. Science China Information Sciences 66, 2 (2023), 122102.
  46. Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data. IEEE Transactions on Parallel and Distributed Systems 33, 12 (2022), 3291–3305.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Zeke Xia (6 papers)
  2. Ming Hu (110 papers)
  3. Dengke Yan (3 papers)
  4. Xiaofei Xie (104 papers)
  5. Tianlin Li (43 papers)
  6. Anran Li (24 papers)
  7. Junlong Zhou (3 papers)
  8. Mingsong Chen (53 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets