Papers
Topics
Authors
Recent
Search
2000 character limit reached

FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission

Published 1 Mar 2024 in cs.LG, cs.DC, and cs.NI | (2403.00881v1)

Abstract: Communication overhead is a significant bottleneck in federated learning (FL), which has been exaggerated with the increasing size of AI models. In this paper, we propose FedRDMA, a communication-efficient cross-silo FL system that integrates RDMA into the FL communication protocol. To overcome the limitations of RDMA in wide-area networks (WANs), FedRDMA divides the updated model into chunks and designs a series of optimization techniques to improve the efficiency and robustness of RDMA-based communication. We implement FedRDMA atop the industrial federated learning framework and evaluate it on a real-world cross-silo FL scenario. The experimental results show that \sys can achieve up to 3.8$\times$ speedup in communication efficiency compared to traditional TCP/IP-based FL systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. “Leveraging large language models for multiple choice question answering,” arXiv preprint arXiv:2210.12353, 2022.
  2. “From images to textual prompts: Zero-shot visual question answering with frozen large language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10867–10877.
  3. “Sentiment analysis in the era of large language models: A reality check,” arXiv preprint arXiv:2305.15005, 2023.
  4. “Text classification via large language models,” arXiv preprint arXiv:2305.08377, 2023.
  5. “Large language models for text classification: From zero-shot learning to fine-tuning,” Open Science Foundation, 2023.
  6. “Natural language processing for ehr-based computational phenotyping,” IEEE/ACM transactions on computational biology and bioinformatics, vol. 16, no. 1, pp. 139–153, 2018.
  7. “Llms accelerate annotation for medical information extraction,” in Machine Learning for Health (ML4H). PMLR, 2023, pp. 82–100.
  8. “Finbert: A large language model for extracting information from financial text,” Contemporary Accounting Research, vol. 40, no. 2, pp. 806–841, 2023.
  9. “Finbert: A pretrained language model for financial communications,” arXiv preprint arXiv:2006.08097, 2020.
  10. Paul Voigt and Axel Von dem Bussche, “The eu general data protection regulation (gdpr),” A Practical Guide, 1st Ed., Cham: Springer International Publishing, vol. 10, no. 3152676, pp. 10–5555, 2017.
  11. “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019.
  12. “A survey on federated learning systems: Vision, hype and reality for data privacy and protection,” IEEE Transactions on Knowledge and Data Engineering, 2021.
  13. “Federated few-shot learning for mobile nlp,” in Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 2023, pp. 1–17.
  14. “Efficient federated learning for modern nlp,” in Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 2023, pp. 1–16.
  15. “Federated fine-tuning of billion-sized language models across mobile devices,” arXiv preprint arXiv:2308.13894, 2023.
  16. “Federated large language model: A position paper,” arXiv preprint arXiv:2307.08925, 2023.
  17. “Llm-based federated recommendation,” arXiv preprint arXiv:2402.09959, 2024.
  18. “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, pp. 9, 2019.
  19. “Fast distributed deep learning over rdma,” in Proceedings of the Fourteenth EuroSys Conference 2019, 2019, pp. 1–14.
  20. “Accelerating distributed deep learning using multi-path rdma in data center networks,” in Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR), 2021, pp. 88–100.
  21. “Rdma over commodity ethernet at scale,” in Proceedings of the 2016 ACM SIGCOMM Conference, 2016, pp. 202–215.
  22. “A review of applications in federated learning,” Computers & Industrial Engineering, vol. 149, pp. 106854, 2020.
  23. “A survey on federated learning,” Knowledge-Based Systems, vol. 216, pp. 106775, 2021.
  24. “The future of digital health with federated learning,” NPJ digital medicine, vol. 3, no. 1, pp. 119, 2020.
  25. “Federated learning: Challenges, methods, and future directions,” IEEE signal processing magazine, vol. 37, no. 3, pp. 50–60, 2020.
  26. “Cross-silo federated learning: Challenges and opportunities,” 2022.
  27. “Chatgpt for good? on opportunities and challenges of large language models for education,” Learning and individual differences, vol. 103, pp. 102274, 2023.
  28. “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology, 2023.
  29. “Emergent abilities of large language models,” arXiv preprint arXiv:2206.07682, 2022.
  30. “Evaluation of point-to-point network performance of hpc clusters at the level of udp, tcp, and mpi,” in IV Simposio Científico y Tecnológico en Computación, Caracas, Venezuela, 2016.
  31. “Low-parameter federated learning with large language models,” arXiv preprint arXiv:2307.13896, 2023.
  32. “Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  33. “Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning,” in Conference on Lifelong Learning Agents. PMLR, 2023, pp. 456–469.
  34. “Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes,” arXiv preprint arXiv:2312.06353, 2023.
  35. “Error compensated quantized sgd and its applications to large-scale distributed optimization,” in International Conference on Machine Learning. PMLR, 2018, pp. 5325–5333.
  36. “signsgd: Compressed optimisation for non-convex problems,” in International Conference on Machine Learning. PMLR, 2018, pp. 560–569.
  37. “Hermes: an efficient federated learning framework for heterogeneous mobile clients,” in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021, pp. 420–437.
  38. “Pyramidfl: A fine-grained client selection framework for efficient federated learning,” .
  39. “Oort: Efficient federated learning via guided participant selection.,” 2021.
  40. Duowen Liu, “Accelerating intra-party communication in vertical federated learning with rdma,” in Proceedings of the 1st Workshop on Distributed Machine Learning, 2020, pp. 14–20.
  41. Don Towsley, “The stutter go back-n arq protocol,” IEEE transactions on Communications, vol. 27, no. 6, pp. 869–875, 1979.
  42. “A priority-driven flow control mechanism for real-time traffic in multiprocessor networks,” IEEE Transactions on Parallel and Distributed Systems, vol. 9, no. 7, pp. 664–678, 1998.
  43. Sally Floyd, “Tcp and explicit congestion notification,” ACM SIGCOMM Computer Communication Review, vol. 24, no. 5, pp. 8–23, 1994.
  44. Marek Irland, “Buffer management in a packet switch,” IEEE transactions on Communications, vol. 26, no. 3, pp. 328–337, 1978.
  45. “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics. PMLR, 2017, pp. 1273–1282.
  46. “Grpc: A communication cooperation mechanism in distributed systems,” ACM SIGOPS Operating Systems Review, vol. 27, no. 3, pp. 75–86, 1993.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free