Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

One-Shot Sequential Federated Learning for Non-IID Data by Enhancing Local Model Diversity (2404.12130v1)

Published 18 Apr 2024 in cs.LG, cs.CV, and cs.DC

Abstract: Traditional federated learning mainly focuses on parallel settings (PFL), which can suffer significant communication and computation costs. In contrast, one-shot and sequential federated learning (SFL) have emerged as innovative paradigms to alleviate these costs. However, the issue of non-IID (Independent and Identically Distributed) data persists as a significant challenge in one-shot and SFL settings, exacerbated by the restricted communication between clients. In this paper, we improve the one-shot sequential federated learning for non-IID data by proposing a local model diversity-enhancing strategy. Specifically, to leverage the potential of local model diversity for improving model performance, we introduce a local model pool for each client that comprises diverse models generated during local training, and propose two distance measurements to further enhance the model diversity and mitigate the effect of non-IID data. Consequently, our proposed framework can improve the global model performance while maintaining low communication costs. Extensive experiments demonstrate that our method exhibits superior performance to existing one-shot PFL methods and achieves better accuracy compared with state-of-the-art one-shot SFL methods on both label-skew and domain-shift tasks (e.g., 6%+ accuracy improvement on the CIFAR-10 dataset).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263, 2021.
  2. A survey of collaborative machine learning using 5g vehicular communications. IEEE Communications Surveys & Tutorials, 24(2):1280–1303, 2022.
  3. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges. IEEE Communications Surveys & Tutorials, 2023.
  4. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  5. Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems, 1:374–388, 2019.
  6. Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177–186. Springer, 2010.
  7. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405–22418, 2021.
  8. Distributed deep learning networks among institutions for medical imaging. Journal of the American Medical Informatics Association, 25(8):945–954, 2018.
  9. Metafed: Federated learning among federations with cyclic knowledge distillation for personalized healthcare. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  10. On the convergence of federated averaging with cyclic client participation. In International Conference on Machine Learning, pages 5677–5721. PMLR, 2023.
  11. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2022.
  12. Moming Duan. Towards open federated learning platforms: Survey and vision from technical and legal perspectives. arXiv preprint arXiv:2307.02140, 2023.
  13. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 32(1):59–71, 2020.
  14. Learning federated visual prompt in null space for mri reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  15. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=6Tm1mposlrM.
  16. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10112–10121, 2022.
  17. Privacy-preserving collaborative learning with automatic transformation search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 114–123, 2021.
  18. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pages 2066–2073. IEEE, 2012.
  19. One-shot federated learning. In NeurIPS 2018 Workshop on Machine Learning on the Phone and other Consumer Devices, 2018.
  20. Jointly learning from decentralized (federated) and centralized data to mitigate distribution shift. 2021.
  21. Decentralized learning works: An empirical comparison of gossip learning and federated learning. Journal of Parallel and Distributed Computing, 148:109–124, 2021.
  22. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
  23. Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10143–10153, 2022.
  24. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16312–16322. IEEE, 2023.
  25. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
  26. Towards a theoretical and practical understanding of one-shot federated learning with fisher information. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities, 2023.
  27. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  28. Federated learning from small datasets. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=hDDV1lsRV8.
  29. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020.
  30. Learning multiple layers of features from tiny images. 2009.
  31. Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  32. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017.
  33. Practical one-shot federated learning for cross-silo setting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, (IJCAI), pages 1484–1490. International Joint Conferences on Artificial Intelligence Organization, 8 2021a. doi: 10.24963/ijcai.2021/205. URL https://doi.org/10.24963/ijcai.2021/205.
  34. Adversarial collaborative learning on non-iid features. 2023.
  35. Learning to collaborate in decentralized learning of personalized models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9766–9775, 2022.
  36. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020.
  37. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021b.
  38. Convergence analysis of sequential federated learning on heterogeneous data. Advances in Neural Information Processing Systems, 36, 2024.
  39. On safeguarding privacy and security in the framework of federated learning. IEEE network, 34(4):242–248, 2020.
  40. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  41. Rethinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10061–10071, 2022.
  42. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821–10836, 2022.
  43. Braintorrent: A peer-to-peer environment for decentralized federated learning. arXiv preprint arXiv:1905.06731, 2019.
  44. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 10(1):12598, 2020.
  45. Improving the model consistency of decentralized federated learning. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  46. Overcoming forgetting in federated learning on non-iid data. arXiv preprint arXiv:1910.07796, 2019.
  47. One-shot federated learning without server-side training. Neural Networks, 164:203–215, 2023. doi: 10.1016/j.neunet.2023.04.035.
  48. Decentralized federated averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4289–4301, 2022.
  49. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394–21405, 2020.
  50. Collaborative machine learning: Schemes, robustness, and privacy. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  51. Data-free diversity-based ensemble selection for one-shot federated learning in machine learning model market. arXiv preprint arXiv:2302.11751, 2023.
  52. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965–23998. PMLR, 2022.
  53. Personalized federated learning under mixture of distributions. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 37860–37879, 2023.
  54. Exploring one-shot semi-supervised federated learning with a pre-trained diffusion model. arXiv preprint arXiv:2305.04063, 2023.
  55. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019.
  56. Decentralized federated learning: A survey and perspective. arXiv preprint arXiv:2306.01603, 2023.
  57. Speeding up heterogeneous federated learning with sequentially trained superclients. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 3376–3382. IEEE, 2022.
  58. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets