Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Personalized Federated Learning with Feature Alignment and Classifier Collaboration (2306.11867v1)

Published 20 Jun 2023 in cs.LG and cs.DC

Abstract: Data heterogeneity is one of the most challenging issues in federated learning, which motivates a variety of approaches to learn personalized models for participating clients. One such approach in deep neural networks based tasks is employing a shared feature representation and learning a customized classifier head for each client. However, previous works do not utilize the global knowledge during local representation learning and also neglect the fine-grained collaboration between local classifier heads, which limit the model generalization ability. In this work, we conduct explicit local-global feature alignment by leveraging global semantic knowledge for learning a better representation. Moreover, we quantify the benefit of classifier combination for each client as a function of the combining weights and derive an optimization problem for estimating optimal weights. Finally, extensive evaluation results on benchmark datasets with various heterogeneous data scenarios demonstrate the effectiveness of our proposed method. Code is available at https://github.com/JianXu95/FedPAC

PDF HTML Abstract

Personalized Federated Learning with Feature Alignment and Classifier Collaboration

Federated learning (FL) has emerged as a promising technique for enabling collaborative model training over decentralized clients without necessitating the sharing of raw data. However, data heterogeneity across clients often poses significant challenges to achieving satisfactory performance with a single global model. To address these challenges, the paper "Personalized Federated Learning with Feature Alignment and Classifier Collaboration" proposes a novel framework designed to improve personalized models by leveraging global feature alignment and classifier collaboration.

Overview of the Framework

The framework introduced aligns feature representations by employing global feature centroids to regularize local training. This feature alignment aims to reduce diversity across localized feature extractors, thus facilitating better aggregation and model generalization. Furthermore, the framework optimizes local classifiers through inter-client collaboration, allowing each client to benefit from classifier heads derived from similar clients. The collaboration is quantified by a weighted combination of classifier heads, with the weights optimized based on each client's performance metrics.

Theoretical Insights

The paper rigorously analyzes the bias-variance trade-offs associated with classifier collaboration. It demonstrates that the quadratic formulation of this trade-off aids in estimating optimal weights for classifier heads, thereby enhancing personalization without overfitting or underfitting. The feature alignment technique is further backed by insights into reducing testing loss, showcasing the interplay between local and global representations.

Numerical Results and Observations

Evaluation results from experiments on benchmark datasets like EMNIST, Fashion-MNIST, CIFAR-10, and CINIC-10 validate the effectiveness of the proposed method. Notably, FedPAC improves average model accuracy by up to 5%, compared to existing personalized FL methods, under different heterogeneity setups. These improvements are consistent not only when data distributions are non-IID but also under label distribution skew scenarios, highlighting the robustness and adaptability of the approach.

Implications and Future Directions

The implications of FedPAC are both practical and theoretical. Practically, it offers a compelling solution for domains requiring personalized models, such as healthcare systems where data may be siloed in different hospitals. Theoretically, it advances the understanding of personalized model training in FL, prompting exploration into generalized model aggregation methods. Future research could delve into decentralized systems, dynamic data distributions, and privacy-enhanced collaborative learning.

In summary, the paper makes substantial contributions to federated learning literature by introducing mechanisms that effectively balance global and personalized training objectives, presenting a feasible pathway to achieving improved model performance across heterogeneous clients. As federated learning continues to gain traction, such frameworks will play vital roles in advancing personalized applications across various domains.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Federated learning based on dynamic regularization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a.
  2. Debiasing model updates for improving personalized federated training. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.  21–31. PMLR, 2021b.
  3. Personalized federated learning with gaussian processes. arXiv preprint arXiv:2106.15482, 2021.
  4. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
  5. Waffle: Weighted averaging for personalized federated learning. arXiv preprint arXiv:2110.06978, 2021.
  6. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798–1828, 2013.
  7. Rich Caruana. Multitask learning: A knowledge-based source of inductive bias. In Paul E. Utgoff (ed.), Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts, Amherst, MA, USA, June 27-29, 1993, pp.  41–48. Morgan Kaufmann, 1993.
  8. On bridging generic and personalized federated learning. arXiv preprint arXiv:2107.00778, 2021.
  9. On bridging generic and personalized federated learning for image classification. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
  10. Personalized federated learning for heterogeneous clients with clustered knowledge transfer. arXiv preprint arXiv:2109.08119, 2021.
  11. EMNIST: extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, May 14-19, 2017, pp.  2921–2926. IEEE, 2017.
  12. Exploiting shared representations for personalized federated learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.  2089–2099. PMLR, 2021.
  13. Imre Csiszár. The method of types [information theory]. IEEE Transactions on Information Theory, 44(6):2505–2523, 1998.
  14. Implicit gradient alignment in distributed and federated learning. CoRR, abs/2106.13897, 2021.
  15. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461, 2020.
  16. Personalized federated learning with moreau envelopes. In Conference on Neural Information Processing Systems, NeurIPS, 2020.
  17. Flexible clustered federated learning for client-level data distribution shift. arXiv preprint arXiv:2108.09749, 2021.
  18. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  19. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021.
  20. An efficient framework for clustered federated learning. arXiv preprint arXiv:2006.04088, 2020.
  21. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05516, 2020.
  22. Fedml: A research library and benchmark for federated machine learning. CoRR, abs/2007.13518, 2020.
  23. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, 2012.
  24. Federated visual classification with real-world data distribution. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, 2020.
  25. Personalized cross-silo federated learning on non-iid data. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021, pp.  7865–7873. AAAI Press, 2021.
  26. Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488, 2019.
  27. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1–210, 2021.
  28. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.03606, 2020a.
  29. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp.  5132–5143. PMLR, 2020b.
  30. Alex Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. University of Toronto, 2009.
  31. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
  32. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp.  794–797. IEEE, 2020.
  33. Deep learning. Nature, 521(7553):436–444, 2015.
  34. Prototypical contrastive learning of unsupervised representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
  35. Model-contrastive federated learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp.  10713–10722. Computer Vision Foundation / IEEE, 2021b.
  36. Federated learning on non-iid data silos: An experimental study. In 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, May 9-12, 2022, pp.  965–978. IEEE, 2022.
  37. Federated learning: Challenges, methods, and future directions. IEEE Signal Process. Mag., 37(3):50–60, 2020a.
  38. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, 2020b.
  39. Ditto: Fair and robust federated learning through personalization. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.  6357–6368. PMLR, 2021c.
  40. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
  41. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
  42. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.
  43. Federated multi-task learning under a mixture of distributions. arXiv preprint arXiv:2108.10252, 2021.
  44. Personalized federated learning through local memorization. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp.  15070–15092, 2022.
  45. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), volume 54, pp.  1273–1282, 2017.
  46. Prototype guided federated learning of visual feature representations. arXiv preprint arXiv:2105.08982, 2021.
  47. Fedproc: Prototypical contrastive federated learning on non-iid data. CoRR, abs/2109.12273, 2021.
  48. Bias-variance reduced local SGD for less heterogeneous federated learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021.
  49. Fedbabu: Toward enhanced representation for federated image classification. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
  50. Transferrable prototypical networks for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2239–2247. Computer Vision Foundation / IEEE, 2019.
  51. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Trans. Neural Networks Learn. Syst., 32(8):3710–3722, 2021.
  52. Personalized federated learning using hypernetworks. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.  9489–9502. PMLR, 2021.
  53. Federated multi-task learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp.  4424–4434, 2017.
  54. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp.  4077–4087, 2017.
  55. Towards personalized federated learning. arXiv preprint arXiv:2103.00710, 2021a.
  56. Fedproto: Federated prototype learning over heterogeneous devices. CoRR, abs/2105.00243, 2021b.
  57. Fedgp: Correlation-based active client selection for heterogeneous federated learning. CoRR, abs/2103.13822, 2021.
  58. A mathematical framework for quantifying transferability in multi-source transfer learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp.  26103–26116, 2021.
  59. Optimizing federated learning on non-iid data with reinforcement learning. In 39th IEEE Conference on Computer Communications, INFOCOM 2020, Toronto, ON, Canada, July 6-9, 2020, pp.  1698–1707. IEEE, 2020.
  60. Addressing class imbalance in federated learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021, pp.  10165–10173. AAAI Press, 2021.
  61. Node selection toward faster convergence for federated learning on non-iid data. CoRR, abs/2105.07066, 2021.
  62. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
  63. Empirical evaluation of rectified activations in convolutional network, 2015.
  64. Maximal correlation regression. IEEE Access, 8:26591–26601, 2020. doi: 10.1109/ACCESS.2020.2971386.
  65. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol., 10(2), January 2019. ISSN 2157–6904.
  66. Fedfm: Anchor-based feature matching for data heterogeneity in federated learning, 2022. URL https://arxiv.org/abs/2210.07615.
  67. Fedmix: Approximation of mixup under mean augmented federated learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  68. Parameterized knowledge transfer for personalized federated learning. arXiv preprint arXiv:2111.02862, 2021a.
  69. Personalized federated learning with first order model optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b.
  70. Federated learning with non-iid data. arXiv:1806.00582, 2018.
  71. Fedfa: Federated learning with feature anchors to align feature and classifier for heterogeneous data, 2022. URL https://arxiv.org/abs/2211.09299.
  72. Federated learning on non-iid data: A survey. Neurocomputing, 465:371–390, 2021a.
  73. Data-free knowledge distillation for heterogeneous federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.  12878–12889. PMLR, 2021b.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Jian Xu (209 papers)
  2. Xinyi Tong (10 papers)
  3. Shao-Lun Huang (48 papers)
Citations (83)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - JianXu95/FedPAC: Simplified Implementation of FedPAC (50 stars)