Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation (2407.16139v1)

Published 23 Jul 2024 in cs.LG

Abstract: In traditional Federated Learning approaches like FedAvg, the global model underperforms when faced with data heterogeneity. Personalized Federated Learning (PFL) enables clients to train personalized models to fit their local data distribution better. However, we surprisingly find that the feature extractor in FedAvg is superior to those in most PFL methods. More interestingly, by applying a linear transformation on local features extracted by the feature extractor to align with the classifier, FedAvg can surpass the majority of PFL methods. This suggests that the primary cause of FedAvg's inadequate performance stems from the mismatch between the locally extracted features and the classifier. While current PFL methods mitigate this issue to some extent, their designs compromise the quality of the feature extractor, thus limiting the full potential of PFL. In this paper, we propose a new PFL framework called FedPFT to address the mismatch problem while enhancing the quality of the feature extractor. FedPFT integrates a feature transformation module, driven by personalized prompts, between the global feature extractor and classifier. In each round, clients first train prompts to transform local features to match the global classifier, followed by training model parameters. This approach can also align the training objectives of clients, reducing the impact of data heterogeneity on model collaboration. Moreover, FedPFT's feature transformation module is highly scalable, allowing for the use of different prompts to tailor local features to various tasks. Leveraging this, we introduce a collaborative contrastive learning task to further refine feature extractor quality. Our experiments demonstrate that FedPFT outperforms state-of-the-art methods by up to 7.08%.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Debiasing model updates for improving personalized federated training. In International Conference on Machine Learning, pages 21–31. PMLR, 2021.
  2. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
  3. Efficient personalized federated learning via sparse model-adaptation. arXiv preprint arXiv:2305.02776, 2023.
  4. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022.
  5. Exploiting shared representations for personalized federated learning. pages 2089–2099, 2021.
  6. Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning. arXiv e-prints, pages arXiv–2310, 2023.
  7. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557–3568, 2020.
  8. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  9. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  10. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  11. Visual prompt tuning. In European Conference on Computer Vision, 2022.
  12. Personalized edge intelligence via federated self-knowledge distillation. IEEE Transactions on Parallel and Distributed Systems, 34(2):567–580, 2022.
  13. Learning multiple layers of features from tiny images. 2009.
  14. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html, 5, 2010.
  15. Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  16. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
  17. Visual prompt based personalized federated learning. Transactions on Machine Learning Research, 2023.
  18. Global and local prompts cooperation via optimal transport for federated learning. arXiv preprint arXiv:2403.00041, 2024.
  19. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357–6368. PMLR, 2021.
  20. FedBN: Federated learning on non-IID features via local batch normalization. In International Conference on Learning Representations, 2021.
  21. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5319–5329, 2023.
  22. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
  23. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
  24. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
  25. J. Luo and S. Wu. Adapt to adaptation: Learning personalization for cross-silo federated learning. In L. D. Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 2166–2173. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.
  26. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10092–10101, 2022.
  27. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017.
  28. Multi-task federated learning for personalised deep neural networks in edge computing. IEEE Transactions on Parallel and Distributed Systems, 33(3):630–641, 2021.
  29. Prior: Personalized prior for reactivating the information overlooked in federated learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 28378–28392. Curran Associates, Inc., 2023.
  30. Federated adaptive prompt tuning for multi-domain collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15117–15125, 2024.
  31. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 34:23309–23320, 2021.
  32. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394–21405, 2020.
  33. L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  34. Does learning from decentralized non-IID unlabeled data benefit from self supervision? In The Eleventh International Conference on Learning Representations, 2023.
  35. Bold but cautious: Unlocking the potential of personalized federated learning through cautiously aggressive collaboration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19375–19384, October 2023.
  36. pfedgf: Enabling personalized federated learning via gradient fusion. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 639–649. IEEE, 2022.
  37. Efficient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19159–19168, 2023.
  38. Gpfl: Simultaneously learning global and personalized feature information for personalized federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5041–5051, October 2023.
  39. Aligning before aggregating: Enabling communication efficient cross-domain federated learning via consistent feature extraction. IEEE Transactions on Mobile Computing, 23(5):5880–5896, 2024.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Xinghao Wu (12 papers)
  2. Jianwei Niu (42 papers)
  3. Xuefeng Liu (64 papers)
  4. Mingjia Shi (14 papers)
  5. Guogang Zhu (10 papers)
  6. Shaojie Tang (99 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now