Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Learning with Bilateral Curation for Partially Class-Disjoint Data (2405.18972v1)

Published 29 May 2024 in cs.LG and cs.DC

Abstract: Partially class-disjoint data (PCDD), a common yet under-explored data formation where each client contributes a part of classes (instead of all classes) of samples, severely challenges the performance of federated algorithms. Without full classes, the local objective will contradict the global objective, yielding the angle collapse problem for locally missing classes and the space waste problem for locally existing classes. As far as we know, none of the existing methods can intrinsically mitigate PCDD challenges to achieve holistic improvement in the bilateral views (both global view and local view) of federated learning. To address this dilemma, we are inspired by the strong generalization of simplex Equiangular Tight Frame~(ETF) on the imbalanced data, and propose a novel approach called FedGELA where the classifier is globally fixed as a simplex ETF while locally adapted to the personal distributions. Globally, FedGELA provides fair and equal discrimination for all classes and avoids inaccurate updates of the classifier, while locally it utilizes the space of locally missing classes for locally existing classes. We conduct extensive experiments on a range of datasets to demonstrate that our FedGELA achieves promising performance~(averaged improvement of 3.9% to FedAvg and 1.5% to best baselines) and provide both local and global convergence guarantees. Source code is available at:https://github.com/MediaBrain-SJTU/FedGELA.git.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
  2. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.
  3. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022.
  4. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In ISBI, pages 168–172, 2018.
  5. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921–2926. IEEE, 2017.
  6. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pages 2089–2099. PMLR, 2021.
  7. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288, 2019.
  8. Fedskip: Combatting statistical heterogeneity with federated skip aggregation. In 2022 IEEE International Conference on Data Mining (ICDM), pages 131–140, 2022. doi: 10.1109/ICDM54844.2022.00023.
  9. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021.
  10. Endemic goiter and endemic thyroid disorders. World journal of surgery, 15(2):205–215, 1991.
  11. Jonathan Huang. Maximum likelihood estimation of dirichlet distribution parameters. CMU Technique Report, pages 1–9, 2005.
  12. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Representations, 2021.
  13. Federated learning without full labels: A survey. arXiv preprint arXiv:2303.14453, 2023.
  14. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14:1–210, 2021.
  15. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pages 5132–5143, 2020.
  16. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
  17. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713–10722, 2021.
  18. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965–978. IEEE, 2022.
  19. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020a.
  20. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020b.
  21. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 995–1005, 2021.
  22. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  23. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
  24. Sgd and hogwild! convergence without the bounded gradients assumption. In International Conference on Machine Learning, pages 3750–3758. PMLR, 2018.
  25. Fedbabu: Toward enhanced representation for federated image classification. In 10th International Conference on Learning Representations, ICLR 2022. International Conference on Learning Representations (ICLR), 2022.
  26. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
  27. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026–8037, 2019.
  28. William Shakespeare et al. William Shakespeare: the complete works. Barnes & Noble Publishing, 1989.
  29. Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019a.
  30. Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232, 2019b.
  31. Sparsified sgd with memory. Advances in Neural Information Processing Systems, 31, 2018.
  32. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394–21405, 2020.
  33. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8432–8440, 2022.
  34. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint arXiv:2210.04620, 2022.
  35. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9, 2018.
  36. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. The Journal of Machine Learning Research, 22:9709–9758, 2021.
  37. Tackling the objective inconsistency problem in heterogeneous federated optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7611–7623. Curran Associates, Inc., 2020.
  38. Local adaptivity in federated learning: Convergence and consistency. arXiv preprint arXiv:2106.02305, 2021.
  39. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575–2584, 2020.
  40. Disentangling trainability and generalization in deep neural networks. In International Conference on Machine Learning, pages 10462–10472. PMLR, 2020.
  41. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? In Advances in Neural Information Processing Systems, 2022.
  42. Edge-cloud polarization and collaboration: A comprehensive survey for ai. IEEE Transactions on Knowledge and Data Engineering, 35(7):6866–6886, 2022.
  43. Personalized federated learning with inferred collaboration graphs. 2023a.
  44. Feddisco: Federated learning with discrepancy-aware collaboration. arXiv preprint arXiv:2305.19229, 2023b.
  45. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5693–5700, 2019.
  46. What do we mean by generalization in federated learning? In ICLR, 2022.
  47. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pages 26311–26329. PMLR, 2022a.
  48. Semi-supervised domain generalization for medical image analysis. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1–5. IEEE, 2022b.
  49. Grace: A generalized and personalized federated learning method for medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14–24. Springer, 2023a.
  50. Federated domain generalization with generalization adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3954–3963, 2023b.
  51. Communication-efficient algorithms for statistical optimization. Advances in neural information processing systems, 25, 2012.
  52. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367–27377. PMLR, 2022.
  53. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820–29834, 2021a.
  54. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning, pages 12878–12889. PMLR, 2021b.
Citations (12)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets