Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models (2401.06432v2)

Published 12 Jan 2024 in cs.LG and cs.DC

Abstract: Foundation models (FMs) adapt well to specific domains or tasks with fine-tuning, and federated learning (FL) enables the potential for privacy-preserving fine-tuning of the FMs with on-device local data. For federated fine-tuning of FMs, we consider the FMs with small to medium parameter sizes of single digit billion at maximum, referred to as on-device FMs (ODFMs) that can be deployed on devices for inference but can only be fine-tuned with parameter efficient methods. In our work, we tackle the data and system heterogeneity problem of federated fine-tuning of ODFMs by proposing a novel method using heterogeneous low-rank approximations (LoRAs), namely HetLoRA. First, we show that the naive approach of using homogeneous LoRA ranks across devices face a trade-off between overfitting and slow convergence, and thus propose HetLoRA, which allows heterogeneous ranks across client devices and efficiently aggregates and distributes these heterogeneous LoRA modules. By applying rank self-pruning locally and sparsity-weighted aggregation at the server, HetLoRA combines the advantages of high and low-rank LoRAs, which achieves improved convergence speed and final performance compared to homogeneous LoRA. Furthermore, HetLoRA offers enhanced computation efficiency compared to full fine-tuning, making it suitable for federated fine-tuning across heterogeneous devices.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Slora: Federated parameter efficient fine-tuning of language models. CoRR, abs/2308.06522.
  2. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
  3. Practical secure aggregation for federated learning on user-held data. In NIPS Workshop on Private Multi-Party Machine Learning.
  4. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  5. Differentially private bias-term only fine-tuning of foundation models. arXiv preprint arXiv:2210.00036.
  6. Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers. CoRR, abs/2211.08025.
  7. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  8. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378.
  9. Google. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  10. Google. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.1040.
  11. Pfedprompt: Learning personalized prompt for vision-language models in federated learning. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 1364–1374, New York, NY, USA. Association for Computing Machinery.
  12. Promptfl: Let federated participants cooperatively learn prompts instead of models — federated learning in age of foundation model. CoRR, abs/2208.11625.
  13. Venkatesan Guruswami and Ravi Kannan. 2012. Lecture notes in computer science theory for the information age.
  14. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning (ICML).
  15. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR).
  16. Large language models are zero-shot reasoners. In The 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
  17. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090.
  18. Guiding the last layer in federated learning with pre-trained models. In Workshop of Federated Learning and Analytics in Practice@ICML.
  19. The power of scale for parameter-efficient prompt tuning. In Empirical Methods in Natural Language Processing (EMNLP).
  20. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
  21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  22. Federated multilingual models for medical transcript analysis. In Conference on Health, Inference, and Learning (CHIL), pages 147–162.
  23. Communication-Efficient Learning of Deep Networks from Decentralized Data. International Conference on Artificial Intelligenece and Statistics (AISTATS).
  24. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:submit/4812508.
  25. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.
  26. Federated optimization for heterogeneous networks. In Proceedings of the 3rd MLSys Conference.
  27. Chateval: A tool for chatbot evaluation. Proceedings of NAACL-HLT.
  28. Ofir Ben Shoham and Nadav Rappoport. 2023. Federated learning of medical concepts embedding using behrt. arXiv preprint arXiv:2305.13052.
  29. Fit: Parameter efficient few-shot transfer learning for personalized and federated image classification. International Conference on Learning Representations (ICLR).
  30. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  31. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark. Association for Computational Linguistics.
  32. A field guide to federated optimization. arXiv preprint arXiv:2107.06917.
  33. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 37(6):1205 – 1221.
  34. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 34:16158–16170.
  35. lo-fi: distributed fine-tuning without communication. Transactions on Machine Learning Research (TMLR).
  36. Beyond goldfish memory: Long-term open-domain conversation. arXiv preprint arXiv:2107.07567.
  37. Fedlora: Model-heterogeneous personalized federated learning with lora tuning. arXiv preprint arXiv:2310.13283.
  38. Federated foundation models: Privacy-preserving and collaborative learning for large models. arXiv preprint arXiv:2305.11414.
  39. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199.
  40. Towards building the federated gpt: Federated instruction tuning. CoRR, abs/2305.05644.
  41. Adaptive budget allocation for parameter-efficient fine-tuning. In The 11th International Conference on Learning Representations (ICLR).
  42. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL).
  43. Fedlegal: The first real-world federated learning benchmark for legal nlp. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  44. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. Findings of the Association for Computational Linguistics (ACL).
  45. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Yae Jee Cho (15 papers)
  2. Luyang Liu (20 papers)
  3. Zheng Xu (73 papers)
  4. Aldi Fahrezi (2 papers)
  5. Gauri Joshi (73 papers)
Citations (27)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets