Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DFML: Decentralized Federated Mutual Learning (2402.01863v2)

Published 2 Feb 2024 in cs.LG, cs.AI, and cs.DC

Abstract: In the realm of real-world devices, centralized servers in Federated Learning (FL) present challenges including communication bottlenecks and susceptibility to a single point of failure. Additionally, contemporary devices inherently exhibit model and data heterogeneity. Existing work lacks a Decentralized FL (DFL) framework capable of accommodating such heterogeneity without imposing architectural restrictions or assuming the availability of public data. To address these issues, we propose a Decentralized Federated Mutual Learning (DFML) framework that is serverless, supports nonrestrictive heterogeneous models, and avoids reliance on public data. DFML effectively handles model and data heterogeneity through mutual learning, which distills knowledge between clients, and cyclically varying the amount of supervision and distillation signals. Extensive experimental results demonstrate consistent effectiveness of DFML in both convergence speed and global accuracy, outperforming prevalent baselines under various conditions. For example, with the CIFAR-100 dataset and 50 clients, DFML achieves a substantial increase of +17.20% and +19.95% in global accuracy under Independent and Identically Distributed (IID) and non-IID data shifts, respectively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. Advances in Neural Information Processing Systems, 35:29677–29690, 2022.
  2. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges. IEEE Communications Surveys & Tutorials, 2023.
  3. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025, 2021.
  4. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.07210, 2018.
  5. Bam! born-again multi-task networks for natural language understanding. arXiv preprint arXiv:1907.04829, 2019.
  6. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021.
  7. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020.
  8. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  9. Decentralized federated learning for nonintrusive load monitoring in smart energy communities. In 2022 30th Mediterranean Conference on Control and Automation (MED), pp.  312–317. IEEE, 2022.
  10. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  11. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  12. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34:12876–12889, 2021.
  13. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
  14. Learning multiple layers of features from tiny images. 2009.
  15. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  16. Vision transformer for small-size datasets. arXiv preprint arXiv:2112.13492, 2021.
  17. Re-weighted softmax cross-entropy to control forgetting in federated learning. arXiv preprint arXiv:2304.05260, 2023.
  18. Decentralized federated learning via mutual knowledge transfer. IEEE Internet of Things Journal, 9(2):1136–1147, 2021.
  19. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  20. Practical one-shot federated learning for cross-silo setting. arXiv preprint arXiv:2010.01017, 2020.
  21. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  22. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
  23. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  24. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
  25. Braintorrent: A peer-to-peer environment for decentralized federated learning. arXiv preprint arXiv:1905.06731, 2019.
  26. Federated learning with mutually cooperating devices: A consensus approach towards server-less model optimization. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  3937–3941. IEEE, 2020.
  27. Federated mutual learning. arXiv preprint arXiv:2006.16765, 2020.
  28. Smith, L. N. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp.  464–472. IEEE, 2017.
  29. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023.
  30. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3):1–44, 2023.
  31. Decentralized federated learning: A survey and perspective. arXiv preprint arXiv:2306.01603, 2023.
  32. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4320–4328, 2018.
  33. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.00650, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Yasser H. Khalil (2 papers)
  2. Amir H. Estiri (1 paper)
  3. Mahdi Beitollahi (6 papers)
  4. Nader Asadi (10 papers)
  5. Sobhan Hemati (15 papers)
  6. Xu Li (126 papers)
  7. Guojun Zhang (43 papers)
  8. Xi Chen (1036 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets