Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning (2404.02478v1)

Published 3 Apr 2024 in cs.LG and cs.AI

Abstract: Standard federated learning approaches suffer when client data distributions have sufficient heterogeneity. Recent methods addressed the client data heterogeneity issue via personalized federated learning (PFL) - a class of FL algorithms aiming to personalize learned global knowledge to better suit the clients' local data distributions. Existing PFL methods usually decouple global updates in deep neural networks by performing personalization on particular layers (i.e. classifier heads) and global aggregation for the rest of the network. However, preselecting network layers for personalization may result in suboptimal storage of global knowledge. In this work, we propose FedSelect, a novel PFL algorithm inspired by the iterative subnetwork discovery procedure used for the Lottery Ticket Hypothesis. FedSelect incrementally expands subnetworks to personalize client parameters, concurrently conducting global aggregations on the remaining parameters. This approach enables the personalization of both client parameters and subnetwork structure during the training process. Finally, we show that FedSelect outperforms recent state-of-the-art PFL algorithms under challenging client data heterogeneity settings and demonstrates robustness to various real-world distributional shifts. Our code is available at https://github.com/lapisrocks/fedselect.

PDF HTML Abstract

An Overview of FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning

The paper introduces FedSelect, a novel algorithm in the domain of Personalized Federated Learning (PFL) which enhances model personalization by customizing the selection of parameters for fine-tuning while simultaneously incorporating global knowledge across clients. This addresses a persistent challenge in federated learning, where heterogeneous data distributions across clients can lead to suboptimal global models.

Problem Context and Motivation

Federated Learning (FL) enables multiple clients to cooperatively train a central model without sharing local data, a crucial capability for privacy-concerned applications. However, significant heterogeneity in client data distributions poses a severe challenge as it may lead to a divergence between local updates and the aggregated global model, compromising overall performance. PFL approaches tackle this issue by personalizing parts or the entirety of the model to suit individual clients' data distributions better. Traditional PFL methods often decouple model parameters into global and personalized parts at a coarse level (e.g., layer-wise adjustments), which could result in inadequate knowledge sharing.

FedSelect refines this process by leveraging the Lottery Ticket Hypothesis (LTH) to progressively adapt client-specific subnetworks, thus optimizing both personalization and aggregation. This insight is derived from the hypothesis that not all layers or parameters are equally important for capturing local distribution characteristics.

Methodology

FedSelect uses a gradient-focused approach to identify which parameters to personalize. Instead of pre-selecting layers as previous methods have done, FedSelect evaluates the magnitude of parameter updates during training to decide personalization. Parameters with larger updates are deemed more essential for personalization, whereas those with minor updates remain part of the global aggregation. This parameter-wise attentiveness allows for a more granular and potentially more accurate personalization over traditional layer-wise approaches.

FedSelect operates as follows:

  • Local Training: Each client trains its local model, and the parameters are classified into global and personalized based on their updates' magnitude.
  • Parameter Selection: Inspired by LTH, which posits that dense, over-parameterized networks contain smaller, trainable subnetworks, FedSelect identifies and refines these subnetworks incrementally.
  • Aggregation and Broadcast: Parameters identified as global are aggregated across clients and redistributed, while personalized subnetworks are refined locally. This iterative refinement leads to enhanced adaptability to {distributional shifts}.

Experimental Results

Experiments with benchmark datasets such as CIFAR-10, CIFAR10-C, Mini-ImageNet, and OfficeHome demonstrated that FedSelect consistently achieves superior performance compared to state-of-the-art methods, particularly in scenarios involving feature and label shifts. Its robust performance is attributed to the dynamic selection of personalized subnetworks, offering a flexible balance between local personalization and global model quality.

Implications and Future Directions

The findings suggest that fine-grained parameter selection strategies can substantially improve the performance of PFL in heterogeneous data environments. By adopting a methodology informed by LTH, FedSelect presents a pathway towards more efficient use of network capacity, retaining essential global knowledge while allowing significant personalization.

Future research directions include exploring the theoretical underpinnings of personalized subnetwork discovery, extending the functionality of FedSelect to more complex and larger-scale federated settings, and investigating the implications of dynamic network reconfiguration over time from both a computational and communication efficiency perspective.

In conclusion, FedSelect exemplifies an innovative approach to federated learning personalization, showing promise for applications where data privacy and personalization are critical. It advances the narrative that personalization and global cohesiveness are not mutually exclusive but can be jointly optimized to achieve superior learning outcomes.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021.
  2. Personalized federated learning with gaussian processes. Advances in Neural Information Processing Systems, 34:8392–8406, 2021.
  3. Federated residual learning. ArXiv, abs/2003.12880, 2020.
  4. Federated learning: A survey on enabling technologies, protocols, and applications. IEEE Access, 8:140699–140725, 2020.
  5. Federated learning with personalization layers, 2019.
  6. Federated meta-learning with fast convergence and efficient communication, 2019.
  7. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pages 2089–2099. PMLR, 2021.
  8. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  9. Flexible clustered federated learning for client-level data distribution shift. IEEE Transactions on Parallel and Distributed Systems, 33:2661–2674, 2021.
  10. Personalized federated learning: A meta-learning approach, 2020.
  11. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
  12. Linear mode connectivity and the lottery ticket hypothesis, 2020.
  13. An efficient framework for clustered federated learning. IEEE Transactions on Information Theory, 68:8076–8091, 2020.
  14. Deep residual learning for image recognition, 2015.
  15. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
  16. Federated visual classification with real-world data distribution, 2020.
  17. Improving federated learning personalization via model agnostic meta learning, 2023.
  18. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  19. Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
  20. Surgical fine-tuning improves adaptation to distribution shifts, 2023.
  21. Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets, 2020a.
  22. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 35:3347–3366, 2019.
  23. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60, 2020b.
  24. Federated optimization in heterogeneous networks, 2020c.
  25. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357–6368. PMLR, 2021a.
  26. Fedbn: Federated learning on non-iid features via local batch normalization. In International Conference on Learning Representations, 2021b.
  27. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
  28. On privacy and personalization in cross-silo federated learning, 2022.
  29. Three approaches for personalization with applications to federated learning, 2020.
  30. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109–165, 1989.
  31. Communication-efficient learning of deep networks from decentralized data. In Proc. of Int’l Conf. Artificial Intelligence and Statistics (AISTATS), 2017.
  32. Fedltn: Federated learning for sparse and personalized lottery ticket networks. In Computer Vision – ECCV 2022, pages 69–85, Cham, 2022. Springer Nature Switzerland.
  33. FedBABU: Toward enhanced representation for federated image classification. In International Conference on Learning Representations, 2022.
  34. Federated learning with partial model personalization. In International Conference on Machine Learning, 2022.
  35. Comparing rewinding and fine-tuning in neural network pruning, 2020.
  36. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, pages 9489–9502. PMLR, 2021.
  37. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific Reports, 10(12598), 2020.
  38. Federated multi-task learning. ArXiv, abs/1705.10467, 2017.
  39. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 34:23309–23320, 2021.
  40. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  41. Deep hashing network for unsupervised domain adaptation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  42. Matching networks for one shot learning. In Neural Information Processing Systems, 2017.
  43. Addressing class imbalance in federated learning, 2020.
  44. Personalized federated learning with feature alignment and classifier collaboration. In The Eleventh International Conference on Learning Representations, 2023.
  45. How transferable are features in deep neural networks? In Neural Information Processing Systems, 2014.
  46. Salvaging federated learning by local adaptation, 2022.
  47. Federated learning with non-iid data. 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Rishub Tamirisa (5 papers)
  2. Chulin Xie (27 papers)
  3. Wenxuan Bao (14 papers)
  4. Andy Zhou (23 papers)
  5. Ron Arel (4 papers)
  6. Aviv Shamsian (23 papers)
Citations (3)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube