Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
108 tokens/sec
GPT-4o
67 tokens/sec
Gemini 2.5 Pro Pro
54 tokens/sec
o3 Pro
13 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
24 tokens/sec
2000 character limit reached

A Survey on Efficient Federated Learning Methods for Foundation Model Training (2401.04472v3)

Published 9 Jan 2024 in cs.LG, cs.AI, and cs.DC

Abstract: Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs, and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT.

PDF HTML Abstract
File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat with Paper (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Qsgd: Communication-efficient sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems, volume 30, 2017.
  2. Slora: Federated parameter efficient fine-tuning of language models, 2023.
  3. Federated learning review: Fundamentals, enabling technologies, and future applications. Information Processing & Management, 59(6), 2022.
  4. Flower: A friendly federated learning research framework, 2020.
  5. On the opportunities and risks of foundation models, 2021.
  6. Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23. IJCAI Org., 2023.
  7. A guide to sharing open healthcare data under the general data protection regulation. Scientific Data, 10(1), 2023.
  8. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2020.
  9. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
  10. Fate-llm: A industrial grade federated learning framework for large language models, 2023.
  11. Openfl: the open federated learning library. Physics in Medicine & Biology, 67(21), 2022.
  12. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
  13. Substra: a framework for privacy-preserving, traceable and collaborative machine learning, 2019.
  14. Google. Tensorflow federated. ., 2019.
  15. Knowledge distillation in vision transformers: A critical review, 2023.
  16. Fedml: A research library and benchmark for federated machine learning, 2020.
  17. Distiller: A systematic study of model distillation methods in natural language processing, 2021.
  18. FjORD: Fair and accurate federated learning under heterogeneous targets with ordered dropout. In Advances in Neural Information Processing Systems, 2021.
  19. Lora: Low-rank adaptation of large language models, 2021.
  20. Distributed pruning towards tiny neural networks in federated learning, 2022.
  21. Papaya: Practical, private, and scalable federated learning, 2021.
  22. How can we train deep learning models across clouds and continents? an experimental study, 2023.
  23. Sparse random networks for communication-efficient federated learning, 2022.
  24. Visual prompt tuning, 2022.
  25. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  26. Ammus : A survey of transformer-based pretrained models in natural language processing, 2021.
  27. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023.
  28. Block pruning for faster transformers. In Proceedings of the 2021 Conference on EMNLP, Online and Punta Cana, Dominican Republic, November 2021. ACL.
  29. The power of scale for parameter-efficient prompt tuning, 2021.
  30. Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets, 2020.
  31. Soteriafl: A unified framework for private federated learning with communication compression. In Advances in Neural Information Processing Systems, volume 35, 2022.
  32. Fate: An industrial grade platform for collaborative learning with data protection. JMLR, 22(1), 2021.
  33. From distributed machine learning to federated learning: a survey. Knowledge and Information Systems, 64(4), 2022.
  34. Towards graph foundation models: A survey and beyond, 2023.
  35. The flan collection: Designing data and methods for effective instruction tuning, 2023.
  36. Fedclip: Fast generalization and personalization for clip in federated learning, 2023.
  37. Ibm federated learning: an enterprise framework white paper v0.1, 2020.
  38. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research. PMLR, 2017.
  39. Distributed learning with compressed gradient differences, 2019.
  40. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. In Advances in Neural Information Processing Systems, 2021.
  41. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials, 23(3), 2021.
  42. NVIDIA. Federated learning for healthcare using nvidia clara. ., 2021.
  43. OpenAI. Gpt-4 technical report, 2023.
  44. A hybrid deep learning architecture for privacy-preserving mobile analytics. IEEE Internet of Things Journal, 7(5), 2020.
  45. Federated adversarial domain adaptation. In ICLR, 2020.
  46. Audio-visual model distillation using acoustic images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020.
  47. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
  48. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108. PMLR, 2020.
  49. Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023.
  50. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
  51. Privacy-preserving deep learning. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, September 2015.
  52. Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning, 2023.
  53. Fedbert: When federated learning meets pre-training. ACM Transactions on Intelligent Systems and Technology, 13(4), 2022.
  54. Flint: A platform for federated learning integration, 2023.
  55. Federated fine-tuning of llms on the very edge: The good, the bad, the ugly, 2023.
  56. Federatedscope: A flexible federated learning platform for heterogeneity. Proceedings of the VLDB Endowment, 16(5), 2023.
  57. Federated learning of gboard language models with differential privacy. 2023.
  58. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In Proceedings of the Web Conference 2021, WWW ’21. ACM, 2021.
  59. Uniaudio: An audio foundation model toward universal audio generation, 2023.
  60. He Yang. H-FL: A hierarchical communication-efficient and privacy-protected architecture for federated learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. IJCAI Org., 2021.
  61. Green federated learning, 2023.
  62. Federated foundation models: Privacy-preserving and collaborative learning for large models, 2023.
  63. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021.
  64. A survey on federated learning. Knowledge-Based Systems, 216, 2021.
  65. Towards building the federated gpt: Federated instruction tuning, 2023.
  66. FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023. ACL, 2023.
  67. Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2022.
  68. Adaptive quantization for deep neural network. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.
  69. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt, 2023.
  70. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
  71. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’52. ACM, 2019.
  72. When foundation model meets federated learning: Motivations, challenges, and future directions, 2023.
  73. PySyft: A Library for Easy Federated Learning. Springer International Publishing, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Herbert Woisetschläger (8 papers)
  2. Alexander Isenko (4 papers)
  3. Shiqiang Wang (79 papers)
  4. Ruben Mayer (44 papers)
  5. Hans-Arno Jacobsen (62 papers)
Citations (12)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets