Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data (2401.08977v2)

Published 17 Jan 2024 in cs.LG and cs.AI

Abstract: Federated Long-Tailed Learning (Fed-LT), a paradigm wherein data collected from decentralized local clients manifests a globally prevalent long-tailed distribution, has garnered considerable attention in recent times. In the context of Fed-LT, existing works have predominantly centered on addressing the data imbalance issue to enhance the efficacy of the generic global model while neglecting the performance at the local level. In contrast, conventional Personalized Federated Learning (pFL) techniques are primarily devised to optimize personalized local models under the presumption of a balanced global data distribution. This paper introduces an approach termed Federated Local and Generic Model Training in Fed-LT (FedLoGe), which enhances both local and generic model performance through the integration of representation learning and classifier alignment within a neural collapse framework. Our investigation reveals the feasibility of employing a shared backbone as a foundational framework for capturing overarching global trends, while concurrently employing individualized classifiers to encapsulate distinct refinements stemming from each client's local features. Building upon this discovery, we establish the Static Sparse Equiangular Tight Frame Classifier (SSE-C), inspired by neural collapse principles that naturally prune extraneous noisy features and foster the acquisition of potent data representations. Furthermore, leveraging insights from imbalance neural collapse's classifier norm patterns, we develop Global and Local Adaptive Feature Realignment (GLA-FR) via an auxiliary global classifier and personalized Euclidean norm transfer to align global features with client preferences. Extensive experimental results on CIFAR-10/100-LT, ImageNet, and iNaturalist demonstrate the advantage of our method over state-of-the-art pFL and Fed-LT approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
  2. Credit card fraud detection using machine learning techniques: A comparative analysis. In 2017 international conference on computing networking and informatics (ICCNI), pp.  1–9. IEEE, 2017.
  3. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN), pp.  1–9. IEEE, 2020.
  4. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32, 2019.
  5. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=I1hQbx10Kxn.
  6. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 35(4):83–93, 2020.
  7. Personalized retrogress-resilient federated learning toward imbalanced medical data. IEEE Transactions on Medical Imaging, 41(12):3663–3674, 2022a.
  8. Towards federated long-tailed learning. arXiv preprint arXiv:2206.14988, 2022b.
  9. Spectral co-distillation for personalized federated learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  10. Grp-fed: Addressing client imbalance in federated learning via global-regularized personalization. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pp.  451–458. SIAM, 2022.
  11. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pp. 2089–2099. PMLR, 2021.
  12. Dispfl: Towards communication-efficient personalized federated learning via decentralized sparse training. arXiv preprint arXiv:2206.00187, 2022.
  13. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.  7314–7322, 2023.
  14. Neural collapse in deep linear network: From balanced to imbalanced data. arXiv preprint arXiv:2301.00437, 2023.
  15. Statistical and machine learning models in credit scoring: A systematic literature survey. Applied Soft Computing, 91:106263, 2020.
  16. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461, 2020.
  17. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020.
  18. Spherefed: Hyperspherical federated learning. In European Conference on Computer Vision, pp.  165–184. Springer, 2022.
  19. Federated model aggregation via self-supervised priors for highly imbalanced medical image classification. arXiv preprint arXiv:2307.14959, 2023.
  20. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557–3568, 2020.
  21. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021.
  22. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJl-b3RcF7.
  23. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586–19597, 2020.
  24. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp.  315–323. JMLR Workshop and Conference Proceedings, 2011.
  25. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34:7232–7241, 2021.
  26. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arXiv:2110.02796, 2021.
  27. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  28. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217, 2019.
  29. Adjusting decision boundary for class imbalanced learning. IEEE Access, 8:81674–81685, 2020.
  30. A broad study of pre-training for domain generalization and adaptation. In European Conference on Computer Vision, pp.  621–638. Springer, 2022.
  31. Federated learning on clinical benchmark data: performance assessment. Journal of medical Internet research, 22(10):e20891, 2020.
  32. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  33. Long-tailed visual recognition via gaussian clouded logit adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  6929–6938, 2022a.
  34. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020a.
  35. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357–6368. PMLR, 2021a.
  36. Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arXiv:2212.12206, 2022b.
  37. Fed{bn}: Federated learning on non-{iid} features via local batch normalization. In International Conference on Learning Representations, 2021b. URL https://openreview.net/pdf?id=6YEQUn0QICG.
  38. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10991–11000, 2020b.
  39. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. arXiv preprint arXiv:2303.10058, 2023.
  40. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  2537–2546, 2019.
  41. Digeo: Discriminative geometry-aware learning for generalized few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3208–3218, 2023.
  42. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  43. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619, 2020.
  44. FedBABU: Toward enhanced representation for federated image classification. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=HuaYQfggn5u.
  45. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
  46. Federated learning with partial model personalization. In International Conference on Machine Learning, pp. 17716–17758. PMLR, 2022.
  47. Long-tailed federated learning via aggregated meta mapping. In 2023 IEEE International Conference on Image Processing (ICIP), pp.  2010–2014. IEEE, 2023.
  48. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8):3710–3722, 2020.
  49. Fedic: Federated learning on non-iid and long-tailed data via calibrated distillation. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pp.  1–6. IEEE, 2022a.
  50. Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. arXiv preprint arXiv:2204.13399, 2022b.
  51. Cd2-pfed: Cyclic distillation-guided channel decoupling for model personalization in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10041–10050, 2022.
  52. An agnostic approach to federated learning with class imbalance. In International Conference on Learning Representations, 2021.
  53. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394–21405, 2020.
  54. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  55. Equalization loss v2: A new gradient balance approach for long-tailed object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  1685–1694, 2021.
  56. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225–27238, 2022.
  57. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning, pp. 21478–21505. PMLR, 2022.
  58. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  8769–8778, 2018.
  59. Proper reuse of image classification features improves object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  13628–13637, 2022.
  60. Logit calibration for non-iid and long-tailed data in federated learning. In 2022 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), pp.  782–789. IEEE, 2022.
  61. Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  10165–10173, 2021.
  62. Mma regularization: Decorrelating weights of neural networks by maximizing the minimal angles. Advances in Neural Information Processing Systems, 33:19099–19110, 2020.
  63. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO), pp.  2174–2178. IEEE, 2021.
  64. Integrating local real data with global gradient prototypes for classifier re-balancing in federated long-tailed learning. arXiv preprint arXiv:2301.10394, 2023a.
  65. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in Neural Information Processing Systems, 35:37991–38002, 2022.
  66. Neural collapse inspired feature-classifier alignment for few-shot class incremental learning. arXiv preprint arXiv:2302.03004, 2023b.
  67. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems, 35:11547–11560, 2022.
  68. Fed-cbs: A heterogeneity-aware client sampling mechanism for federated learning via class-imbalance reduction. In International Conference on Machine Learning, pp. 41354–41381. PMLR, 2023.
  69. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pp. 26311–26329. PMLR, 2022a.
  70. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning, pp. 26293–26310. PMLR, 2022b.
  71. Understanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  19550–19560, 2023.
  72. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820–29834, 2021.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets