Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring Vacant Classes in Label-Skewed Federated Learning (2401.02329v3)

Published 4 Jan 2024 in cs.LG and cs.CV

Abstract: Label skews, characterized by disparities in local label distribution across clients, pose a significant challenge in federated learning. As minority classes suffer from worse accuracy due to overfitting on local imbalanced data, prior methods often incorporate class-balanced learning techniques during local training. Although these methods improve the mean accuracy across all classes, we observe that vacant classes-referring to categories absent from a client's data distribution-remain poorly recognized. Besides, there is still a gap in the accuracy of local models on minority classes compared to the global model. This paper introduces FedVLS, a novel approach to label-skewed federated learning that integrates both vacant-class distillation and logit suppression simultaneously. Specifically, vacant-class distillation leverages knowledge distillation during local training on each client to retain essential information related to vacant classes from the global model. Moreover, logit suppression directly penalizes network logits for non-label classes, effectively addressing misclassifications in minority classes that may be biased toward majority classes. Extensive experiments validate the efficacy of FedVLS, demonstrating superior performance compared to previous state-of-the-art (SOTA) methods across diverse datasets with varying degrees of label skews. Our code is available at https://github.com/krumpguo/FedVLS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263, 2021.
  2. A survey for federated learning evaluations: Goals and measures. arXiv preprint arXiv:2308.11841, 2023.
  3. Calfat: Calibrated federated adversarial training with label skewness. In Proc. NeurIPS, 2022.
  4. The best of both worlds: Accurate global and personalized models through federated learning with data-free hyper-knowledge distillation. In Proc. ICLR, 2023.
  5. Autoaugment: Learning augmentation policies from data. In Proc. CVPR, 2019.
  6. Class-balanced loss based on effective number of samples. In Proc. CVPR, pp.  9268–9277, 2019.
  7. Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29:141–142, 2012.
  8. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
  9. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data. IEEE Transactions on Mobile Computing, 22:191–205, 2021.
  10. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. In Proc. NeurIPS Workshops, 2018.
  11. Fedexp: Speeding up federated averaging via extrapolation. In Proc. ICLR, 2023.
  12. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14:1–210, 2021.
  13. Scaffold: Stochastic controlled averaging for federated learning. In Proc. ICML, pp.  5132–5143, 2020.
  14. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016.
  15. Krizhevsky, A. Learning multiple layers of features from tiny images. Master’s thesis, University of Tront, 2009.
  16. Tiny imagenet visual recognition challenge. CS 231N, 7:3, 2015.
  17. Preservation of the global knowledge by not-true distillation in federated learning. In Proc. NeurIPS, pp.  38461–38474, 2022.
  18. Fedmd: Heterogenous federated learning via model distillation. In Proc. NeurIPS Workshops, 2019.
  19. Model-contrastive federated learning. In Proc. CVPR, pp.  10713–10722, 2021.
  20. Federated learning on non-iid data silos: An experimental study. In Proc. ICDE, pp.  965–978, 2022.
  21. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37:50–60, 2020a.
  22. Federated optimization in heterogeneous networks. In Proc. MLSys, pp.  429–450, 2020b.
  23. On the convergence of fedavg on non-iid data. 2019.
  24. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proc. KDD, pp.  995–1005, 2021.
  25. Ensemble distillation for robust model fusion in federated learning. In Proc. NeurIPS, pp.  2351–2363, 2020.
  26. Towards method of horizontal federated learning: A survey. In Proc. BigDIA, pp.  259–266, 2022.
  27. Dfrd: Data-free robustness distillation for heterogeneous federated learning. In Proc. NeurIPS, 2023.
  28. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. In Proc. NeurIPS, pp.  5972–5984, 2021.
  29. Curvature-balanced feature manifold learning for long-tailed classification. In Proc. CVPR, pp.  15824–15835, 2023.
  30. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282, 2017.
  31. Long-tail learning via logit adjustment. In Proc. ICLR, 2021.
  32. Fedproc: Prototypical contrastive federated learning on non-iid data. Future Generation Computer Systems, 143:93–104, 2023.
  33. Generalized federated learning via sharpness aware minimization. In Proc. ICML, pp.  18250–18280, 2022.
  34. A comparative evaluation of fedavg and per-fedavg algorithms for dirichlet distributed heterogeneous data. arXiv preprint arXiv:2309.01275, 2023.
  35. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. CVPR, pp.  4510–4520, 2018.
  36. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 10:12598, 2020.
  37. Federated learning with classifier shift for class imbalance. arXiv preprint arXiv:2304.04972, 2023.
  38. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. In Proc. ICLR, 2023a.
  39. Understanding and mitigating dimensional collapse in federated learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b.
  40. Equalization loss for long-tailed object recognition. In Proc. CVPR, pp.  11662–11671, 2020.
  41. Dafkd: Domain-aware federated knowledge distillation. In Proc. CVPR, pp.  20412–20421, 2023a.
  42. Federated skewed label learning with logits fusion. arXiv preprint arXiv:2311.08202, 2023b.
  43. Survey of knowledge distillation in federated edge learning. arXiv preprint arXiv:2301.05849, 2023.
  44. Fed-grab: Federated long-tailed learning with self-adjusting gradient balancer. arXiv preprint arXiv:2310.07587, 2023.
  45. Achieving linear speedup with partial worker participation in non-iid federated learning. In Proc. ICLR, 2021.
  46. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3):1–44, 2023.
  47. Inverse distance aggregation for federated learning with non-iid data. In Proc. MICCAI Workshops, pp.  150–159, 2020.
  48. Global balanced experts for federated long-tailed learning. In Proc. ICCV, pp.  4815–4825, 2023.
  49. Federated learning with label distribution skew via logits calibration. In Proc. ICML, pp.  26311–26329, 2022.
  50. A survey on class imbalance in federated learning. arXiv preprint arXiv:2303.11673, 2023.
  51. Federated learning on non-iid data: A survey. Neurocomputing, 465:371–390, 2021a.
  52. Data-free knowledge distillation for heterogeneous federated learning. In Proc. ICML, pp.  12878–12889, 2021b.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets