Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices using a Computing Power Aware Scheduler (2309.14675v2)

Published 26 Sep 2023 in cs.LG and cs.DC

Abstract: Cross-silo federated learning offers a promising solution to collaboratively train robust and generalized AI models without compromising the privacy of local datasets, e.g., healthcare, financial, as well as scientific projects that lack a centralized data facility. Nonetheless, because of the disparity of computing resources among different clients (i.e., device heterogeneity), synchronous federated learning algorithms suffer from degraded efficiency when waiting for straggler clients. Similarly, asynchronous federated learning algorithms experience degradation in the convergence rate and final model accuracy on non-identically and independently distributed (non-IID) heterogeneous datasets due to stale local models and client drift. To address these limitations in cross-silo federated learning with heterogeneous clients and data, we propose FedCompass, an innovative semi-asynchronous federated learning algorithm with a computing power-aware scheduler on the server side, which adaptively assigns varying amounts of training tasks to different clients using the knowledge of the computing power of individual clients. FedCompass ensures that multiple locally trained models from clients are received almost simultaneously as a group for aggregation, effectively reducing the staleness of local models. At the same time, the overall training process remains asynchronous, eliminating prolonged waiting periods from straggler clients. Using diverse non-IID heterogeneous distributed datasets, we demonstrate that FedCompass achieves faster convergence and higher accuracy than other asynchronous algorithms while remaining more efficient than synchronous algorithms when performing federated learning on heterogeneous clients. The source code for FedCompass is available at https://github.com/APPFL/FedCompass.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems, 1:374–388, 2019.
  2. FedAT: A high-performance and communication-efficient federated learning system with asynchronous tiers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp.  1–16, 2021.
  3. Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation. IEEE transactions on neural networks and learning systems, 31(10):4229–4238, 2019.
  4. Asynchronous online federated learning for edge devices with non-iid data. In 2020 IEEE International Conference on Big Data (Big Data), pp.  15–24. IEEE, 2020.
  5. Towards asynchronous federated learning for heterogeneous edge-powered internet of things. Digital Communications and Networks, 7(3):317–326, 2021.
  6. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pp.  424–432. Springer, 2016.
  7. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp.  168–172. IEEE, 2018.
  8. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288, 2019.
  9. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on Computer Vision and Pattern Recognition, pp.  248–255. Ieee, 2009.
  10. Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297–302, 1945.
  11. Time efficient federated learning with semi-asynchronous communication. In 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS), pp.  156–163. IEEE, 2020.
  12. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.  770–778, 2016.
  13. Enabling end-to-end secure federated learning in biomedical research on heterogeneous computing environments with appflx. arXiv preprint arXiv:2312.08701, 2023.
  14. Fedshuffle: Recipes for better use of local work in federated learning. arXiv preprint arXiv:2204.13169, 2022.
  15. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
  16. Fedar: Activity and resource-aware federated learning model for distributed mobile robots. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pp.  1153–1160. IEEE, 2020.
  17. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  18. End-to-end privacy preserving deep learning on multi-institutional medical imaging. Nature Machine Intelligence, 3(6):473–484, 2021.
  19. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp.  5132–5143. PMLR, 2020.
  20. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
  21. Learning multiple layers of features from tiny images. 2009.
  22. Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
  23. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60, 2020.
  24. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.
  25. APPFLx: Providing privacy-preserving cross-silo federated learning as a service. arXiv preprint arXiv:2308.08786, 2023.
  26. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  27. A state-of-the-art survey on solving non-iid data in federated learning. Future Generation Computer Systems, 135:244–258, 2022.
  28. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  29. Communication-efficient federated learning for wireless edge intelligence in IoT. IEEE Internet of Things Journal, 7(7):5986–5994, 2019.
  30. Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics, pp.  3581–3607. PMLR, 2022.
  31. Client selection for federated learning with heterogeneous resources in mobile edge. In ICC 2019-2019 IEEE international Conference on Communications (ICC), pp.  1–7. IEEE, 2019.
  32. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. Advances in Neural Information Processing Systems, 35:5315–5334, 2022.
  33. Federated learning enables big data for rare cancer boundary detection. Nature Communications, 13(1):7346, 2022.
  34. TorchIO: a Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. Computer Methods and Programs in Biomedicine, 208:106236, 2021.
  35. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
  36. Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity. IEEE Journal on Selected Areas in Information Theory, 3(2):197–205, 2022.
  37. APPFL: open-source software framework for privacy-preserving federated learning. In 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp.  1074–1083. IEEE, 2022.
  38. Secure aggregation for buffered asynchronous federated learning. arXiv preprint arXiv:2110.02177, 2021.
  39. Sebastian U Stich. Local SGD converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018.
  40. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on Machine Learning, pp.  6105–6114. PMLR, 2019.
  41. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9, 2018.
  42. Guan Wang. Interpret federated learning with shapley values. arXiv preprint arXiv:1905.04519, 2019.
  43. Measure contribution of participants in federated learning. In 2019 IEEE international conference on Big Data (Big Data), pp.  2597–2604. IEEE, 2019.
  44. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems, 33:7611–7623, 2020.
  45. Efficient federated learning for fault diagnosis in industrial cloud-edge computing. Computing, 103(10):2319–2337, 2021.
  46. SAFA: A semi-asynchronous protocol for fast federated learning with low overhead. IEEE Transactions on Computers, 70(5):655–668, 2020.
  47. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934, 2019.
  48. Asynchronous federated learning on heterogeneous devices: A survey. arXiv preprint arXiv:2109.04269, 2021.
  49. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019.
  50. Parallel restarted with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.  5693–5700, 2019.
  51. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pp.  7252–7261. PMLR, 2019.
  52. CSAFL: A clustered semi-asynchronous federated learning framework. In 2021 International Joint Conference on Neural Networks (IJCNN), pp.  1–10. IEEE, 2021.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets