Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators (2307.05358v3)

Published 11 Jul 2023 in cs.LG and cs.AI

Abstract: Federated learning has become a popular method to learn from decentralized heterogeneous data. Federated semi-supervised learning (FSSL) emerges to train models from a small fraction of labeled data due to label scarcity on decentralized clients. Existing FSSL methods assume independent and identically distributed (IID) labeled data across clients and consistent class distribution between labeled and unlabeled data within a client. This work studies a more practical and challenging scenario of FSSL, where data distribution is different not only across clients but also within a client between labeled and unlabeled data. To address this challenge, we propose a novel FSSL framework with dual regulators, FedDure. FedDure lifts the previous assumption with a coarse-grained regulator (C-reg) and a fine-grained regulator (F-reg): C-reg regularizes the updating of the local model by tracking the learning effect on labeled data distribution; F-reg learns an adaptive weighting scheme tailored for unlabeled instances in each client. We further formulate the client model training as bi-level optimization that adaptively optimizes the model in the client with two regulators. Theoretically, we show the convergence guarantee of the dual regulators. Empirically, we demonstrate that FedDure is superior to the existing methods across a wide range of settings, notably by more than 11 on CIFAR-10 and CINIC-10 datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Multi-Domain Synchronous Refinement Network for Unsupervised Cross-Domain Person Re-Identification. In 2021 IEEE International Conference on Multimedia and Expo (ICME), 1–6.
  2. Convergence and accuracy trade-offs in federated learning and meta-learning. In International Conference on Artificial Intelligence and Statistics, 2575–2583. PMLR.
  3. Debiased Self-Training for Semi-Supervised Learning. In Advances in Neural Information Processing Systems.
  4. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505.
  5. SemiFL: Communication efficient semi-supervised federated learning with unlabeled clients. arXiv preprint arXiv:2106.01432.
  6. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, 1126–1135. PMLR.
  7. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604.
  8. Ssfl: Tackling label deficiency in federated learning via personalized self-supervision. arXiv preprint arXiv:2110.02470.
  9. Federated semi-supervised learning with inter-client consistency & disjoint learning.
  10. A survey towards federated semi-supervised learning. arXiv preprint arXiv:2002.11545, 50.
  11. Towards utilizing unlabeled data in federated learning: A survey and prospective. arXiv preprint arXiv:2002.11545.
  12. Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence, 2(6): 305–311.
  13. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, 5132–5143. PMLR.
  14. Learning multiple layers of features from tiny images.
  15. Contrastive Regularization for Semi-Supervised Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3911–3920.
  16. Lee, D.-H.; et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, 896.
  17. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10713–10722.
  18. Groupformer: Group activity recognition with clustered spatial-temporal transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 13668–13677.
  19. Probing visual-audio representation for video highlight detection via hard-pairs guided contrastive learning. arXiv preprint arXiv:2206.10157.
  20. Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation. arXiv preprint arXiv:2206.10095.
  21. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3): 50–60.
  22. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2: 429–450.
  23. Privacy-preserving federated brain tumour segmentation. In International Workshop on Machine Learning in Medical Imaging, 133–141. Springer.
  24. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623.
  25. RSCFed: Random Sampling Consensus Federated Semi-supervised Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10154–10163.
  26. Semifed: Semi-supervised federated learning with consistency and pseudo-labeling. arXiv preprint arXiv:2108.09412.
  27. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055.
  28. Federated semi-supervised medical image classification via inter-client relation matching. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 325–335. Springer.
  29. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, 1273–1282. PMLR.
  30. Billion-scale federated learning on mobile clients: A submodel design with tunable privacy. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 1–14.
  31. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11557–11568.
  32. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33: 596–608.
  33. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30: 1195–1204.
  34. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440.
  35. Unsupervised Domain Adaptive Learning via Synthetic Data for Person Re-identification. arXiv preprint arXiv:2109.05542.
  36. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747.
  37. Unsupervised data augmentation for consistency training. Neurips, 33: 6256–6268.
  38. Fedcm: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874.
  39. Can Irrelevant Data Help Semi-Supervised Learning, Why and How? In The 20th ACM International Conference on Information and Knowledge Management.
  40. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2): 1–19.
  41. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34: 18408–18419.
  42. Improving semi-supervised federated learning by reducing the gradient diversity of models. In 2021 IEEE International Conference on Big Data (Big Data), 1214–1225. IEEE.
  43. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582.
  44. Divergence-aware Federated Self-Supervised Learning. In International Conference on Learning Representations.
  45. Pseudoseg: Designing pseudo labels for semantic segmentation. International Conference on Learning Representations.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now