Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedFa: A Fully Asynchronous Training Paradigm for Federated Learning (2404.11015v2)

Published 17 Apr 2024 in cs.LG, cs.AI, and cs.DC

Abstract: Federated learning has been identified as an efficient decentralized training paradigm for scaling the machine learning model training on a large number of devices while guaranteeing the data privacy of the trainers. FedAvg has become a foundational parameter update strategy for federated learning, which has been promising to eliminate the effect of the heterogeneous data across clients and guarantee convergence. However, the synchronization parameter update barriers for each communication round during the training significant time on waiting, slowing down the training procedure. Therefore, recent state-of-the-art solutions propose using semi-asynchronous approaches to mitigate the waiting time cost with guaranteed convergence. Nevertheless, emerging semi-asynchronous approaches are unable to eliminate the waiting time completely. We propose a full asynchronous training paradigm, called FedFa, which can guarantee model convergence and eliminate the waiting time completely for federated learning by using a few buffered results on the server for parameter updating. Further, we provide theoretical proof of the convergence rate for our proposed FedFa. Extensive experimental results indicate our approach effectively improves the training performance of federated learning by up to 6x and 4x speedup compared to the state-of-the-art synchronous and semi-asynchronous strategies while retaining high accuracy in both IID and Non-IID scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (25)
  1. Generalization in nli: Ways (not) to go beyond simple heuristics, 2021.
  2. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:1611.04482, 2016.
  3. Leaf: A benchmark for federated settings. arXiv preprint:1812.01097, 2018.
  4. Fedat: A communication-efficient federated learning method with asynchronous tiers under non-iid data. ArXivorg, 2020.
  5. Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703, 2022.
  6. A general theory for federated optimization with asynchronous and heterogeneous clients updates. Journal of Machine Learning Research, 24(110):1–43, 2023.
  7. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  8. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
  9. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  10. Alex K. and H. Geoffrey. Learning multiple layers of features from tiny images. 2009.
  11. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  12. Cryptonite: A framework for flexible time-series secure aggregation with online fault tolerance. Cryptology ePrint Archive, Paper 2020/1561, 2020.
  13. Sharper convergence guarantees for asynchronous sgd for distributed and federated learning. Advances in Neural Information Processing Systems, 35:17202–17215, 2022.
  14. Federated optimization in heterogeneous networks. Proceedings of MLSys, 2:429–450, 2020.
  15. Perturbed iterate analysis for asynchronous stochastic optimization. arXiv preprint arXiv:1507.06970, 2015.
  16. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  17. Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics, pages 3581–3607. PMLR, 2022.
  18. Turbo-aggregate: Breaking the quadratic aggregation barrier in secure federated learning. IEEE Journal on Selected Areas in Information Theory, 2021.
  19. How asynchronous can federated learning be? In 2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS), pages 1–11. IEEE, 2022.
  20. Sebastian U. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018.
  21. Asynchronous federated learning with reduced number of rounds and with differential privacy from less aggregated gaussian noise. arXiv preprint arXiv:2007.09208, 2020.
  22. Tackling the objective inconsistency problem in heterogeneous federated optimization. NeurIPS, 2020.
  23. Asynchronous federated learning on heterogeneous devices: A survey. arXiv preprint arXiv:2109.04269, 2021.
  24. Momentum-driven adaptive synchronization model for distributed dnn training on hpc clusters. Journal of Parallel and Distributed Computing, 159:65–84, 2022.
  25. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.

Summary

We haven't generated a summary for this paper yet.

X Twitter Logo Streamline Icon: https://streamlinehq.com