Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning (2310.13283v2)

Published 20 Oct 2023 in cs.LG and cs.DC

Abstract: Federated learning (FL) is an emerging machine learning paradigm in which a central server coordinates multiple participants (clients) collaboratively to train on decentralized data. In practice, FL often faces statistical, system, and model heterogeneities, which inspires the field of Model-Heterogeneous Personalized Federated Learning (MHPFL). With the increased interest in adopting LLMs in FL, the existing MHPFL methods cannot achieve acceptable computational and communication costs, while maintaining satisfactory model performance. To bridge this gap, we propose a novel and efficient model-heterogeneous personalized Federated learning framework based on LoRA tuning (pFedLoRA). Inspired by the popular LoRA method for fine-tuning pre-trained LLMs with a low-rank model (a.k.a., an adapter), we design a homogeneous small adapter to facilitate federated client's heterogeneous local model training with our proposed iterative training for global-local knowledge exchange. The homogeneous small local adapters are aggregated on the FL server to generate a global adapter. We theoretically prove the convergence of pFedLoRA. Extensive experiments on two benchmark datasets demonstrate that pFedLoRA outperforms six state-of-the-art baselines, beating the best method by 1.35% in test accuracy, 11.81 times computation overhead reduction and 7.41 times communication cost saving.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Jin-Hyun Ahn et al. 2019. Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data. In Proc. PIMRC. IEEE, Istanbul, Turkey, 1–6.
  2. Jin-Hyun Ahn et al. 2020. Cooperative Learning VIA Federated Distillation OVER Fading Channels. In Proc. ICASSP. IEEE, Barcelona, Spain, 8856–8860.
  3. Samiul Alam et al. 2022. FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction. In Proc. NeurIPS. , virtual.
  4. Hongyan Chang et al. 2021. Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer. In Proc. NeurIPS Workshop. , virtual.
  5. Jiangui Chen et al. 2021. FedMatch: Federated Learning Over Heterogeneous Question Answering Data. In Proc. CIKM. ACM, virtual, 181–190.
  6. Sijie Cheng et al. 2021. FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion. CoRR abs/2110.11027 (2021).
  7. Yae Jee Cho et al. 2022. Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning. In Proc. IJCAI. ijcai.org, virtual, 2881–2887.
  8. Liam Collins et al. 2021. Exploiting Shared Representations for Personalized Federated Learning. In Proc. ICML, Vol. 139. PMLR, virtual, 2089–2099.
  9. Enmao Diao. 2021. HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. In Proc. ICLR. OpenReview.net, Virtual Event, Austria, 1.
  10. Chaoyang He et al. 2020. Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge. In Proc. NeurIPS. , virtual.
  11. S. Horváth. 2021. FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout. In Proc. NIPS. OpenReview.net, Virtual, 12876–12889.
  12. Edward J. Hu et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR. OpenReview.net, Virtual, 1.
  13. Wenke Huang et al. 2022a. Few-Shot Model Agnostic Federated Learning. In Proc. MM. ACM, Lisboa, Portugal, 7309–7316.
  14. Wenke Huang et al. 2022b. Learn from Others and Be Yourself in Heterogeneous Federated Learning. In Proc. CVPR. IEEE, virtual, 10133–10143.
  15. Sohei Itahara et al. 2023. Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data. IEEE Trans. Mob. Comput. 22, 1 (2023), 191–205.
  16. Jaehee Jang et al. 2022. FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks. In Proc. ICPP. ACM, virtual, 76:1–76:10.
  17. Eunjeong Jeong et al. 2018. Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data. In Proc. NeurIPS Workshop on Machine Learning on the Phone and other Consumer Devices. , virtual.
  18. Yuang Jiang et al. 2022. Model Pruning Enables Efficient Federated Learning on Edge Devices. TNNLS 1, 1 (2022), 1.
  19. Peter Kairouz et al. 2021. Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning 14, 1–2 (2021), 1–210.
  20. Alex Krizhevsky et al. 2009. Learning multiple layers of features from tiny images. Toronto, ON, Canada, .
  21. Daliang Li and Junpu Wang. 2019. FedMD: Heterogenous Federated Learning via Model Distillation. In Proc. NeurIPS Workshop. , virtual.
  22. Qinbin Li et al. 2021. Practical One-Shot Federated Learning for Cross-Silo Setting. In Proc. IJCAI. ijcai.org, virtual, 1484–1490.
  23. Paul Pu Liang et al. 2020. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 1, 1 (2020).
  24. Tao Lin et al. 2020. Ensemble Distillation for Robust Model Fusion in Federated Learning. In Proc. NeurIPS. , virtual.
  25. Chang Liu et al. 2022. Completely Heterogeneous Federated Learning. CoRR abs/2210.15865 (2022).
  26. Xiaofeng Lu et al. 2022. Heterogeneous Model Fusion Federated Learning Mechanism Based on Model Mapping. IEEE Internet Things J. 9, 8 (2022), 6058–6068.
  27. Disha Makhija et al. 2022. Architecture Agnostic Federated Learning for Neural Networks. In Proc. ICML, Vol. 162. PMLR, virtual, 14860–14870.
  28. Brendan McMahan et al. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proc. AISTATS, Vol. 54. PMLR, Fort Lauderdale, FL, USA, 1273–1282.
  29. Jaehoon Oh et al. 2022. FedBABU: Toward Enhanced Representation for Federated Image Classification. In Proc. ICLR. OpenReview.net, virtual.
  30. Krishna Pillutla et al. 2022. Federated Learning with Partial Model Personalization. In Proc. ICML, Vol. 162. PMLR, virtual, 17716–17758.
  31. Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. CoRR abs/1609.04747 (2016), 1.
  32. Felix Sattler et al. 2021. FEDAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning. IEEE Trans. Neural Networks Learn. Syst. 1, 1 (2021), 1–13.
  33. Felix Sattler et al. 2022. CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding. IEEE Trans. Netw. Sci. Eng. 9, 4 (2022), 2025–2038.
  34. Aviv Shamsian et al. 2021. Personalized Federated Learning using Hypernetworks. In Proc. ICML, Vol. 139. PMLR, virtual, 9489–9502.
  35. Tao Shen et al. 2020. Federated Mutual Learning. CoRR abs/2006.16765 (2020).
  36. Alysa Ziying Tan et al. 2022a. Towards Personalized Federated Learning. IEEE Trans. Neural Networks Learn. Syst. 1, 1 (2022), 1–17. https://doi.org/10.1109/TNNLS.2022.3160699
  37. Yue Tan et al. 2022b. FedProto: Federated Prototype Learning across Heterogeneous Clients. In Proc. AAAI. AAAI Press, virtual, 8432–8440.
  38. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605.
  39. Chuhan Wu et al. 2022. Communication-efficient federated learning via knowledge distillation. Nature Communications 13, 1 (2022), 2032.
  40. Mang Ye et al. 2023. Heterogeneous Federated Learning: State-of-the-art and Research Challenges. CoRR abs/2307.10616 (2023), 1.
  41. FedGH: Heterogeneous Federated Learning with Generalized Global Header. In Proceedings of the 31st ACM International Conference on Multimedia (ACM MM’23). ACM, Canada, 11.
  42. Fuxun Yu et al. 2021. Fed2: Feature-Aligned Federated Learning. In Proc. KDD. ACM, virtual, 2066–2074.
  43. Sixing Yu et al. 2022. Resource-aware Federated Learning using Knowledge Extraction and Multi-model Fusion. CoRR abs/2208.07978 (2022).
  44. Jie Zhang et al. 2021. Parameterized Knowledge Transfer for Personalized Federated Learning. In Proc. NeurIPS. OpenReview.net, virtual, 10092–10104.
  45. Lan Zhang et al. 2022. FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models. In Proc. ICDCS. IEEE, virtual, 928–938.
  46. Zhilu Zhang and Mert R. Sabuncu. 2018. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels. In Proc. NeurIPS. Curran Associates Inc., Montréal, Canada, 8792–8802.
  47. Zhuangdi Zhu et al. 2021. Data-Free Knowledge Distillation for Heterogeneous Federated Learning. In Proc. ICML, Vol. 139. PMLR, virtual, 12878–12889.
  48. Zhuangdi Zhu et al. 2022. Resilient and Communication Efficient Learning for Heterogeneous Federated Systems. In Proc. ICML, Vol. 162. PMLR, virtual, 27504–27526.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Liping Yi (8 papers)
  2. Han Yu (218 papers)
  3. Gang Wang (406 papers)
  4. Xiaoguang Liu (19 papers)
  5. Xiaoxiao Li (144 papers)
Citations (3)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets