Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors (2405.11525v2)

Published 19 May 2024 in cs.LG and cs.AI

Abstract: Conventional Federated Learning (FL) involves collaborative training of a global model while maintaining user data privacy. One of its branches, decentralized FL, is a serverless network that allows clients to own and optimize different local models separately, which results in saving management and communication resources. Despite the promising advancements in decentralized FL, it may reduce model generalizability due to lacking a global model. In this scenario, managing data and model heterogeneity among clients becomes a crucial problem, which poses a unique challenge that must be overcome: How can every client's local model learn generalizable representation in a decentralized manner? To address this challenge, we propose a novel Decentralized FL technique by introducing Synthetic Anchors, dubbed as DeSA. Based on the theory of domain adaptation and Knowledge Distillation (KD), we theoretically and empirically show that synthesizing global anchors based on raw data distribution facilitates mutual knowledge transfer. We further design two effective regularization terms for local training: 1) REG loss that regularizes the distribution of the client's latent embedding with the anchors and 2) KD loss that enables clients to learn from others. Through extensive experiments on diverse client data distributions, we showcase the effectiveness of DeSA in enhancing both inter- and intra-domain accuracy of each client.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.  308–318, 2016.
  2. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804, 2019.
  3. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning, pp.  344–353. PMLR, 2019.
  4. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges. IEEE Communications Surveys & Tutorials, 2023.
  5. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
  6. A theory of learning from different domains. Machine learning, 79:151–175, 2010.
  7. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp.  1897–1914. IEEE, 2022a.
  8. No free lunch in” privacy for free: How does dataset condensation help privacy”. arXiv preprint arXiv:2209.14987, 2022b.
  9. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4750–4759, 2022.
  10. Fedhe: Heterogeneous models and communication-efficient federated learning. In 2021 17th International Conference on Mobility, Sensing and Networking (MSN), pp.  207–214. IEEE, 2021.
  11. Distributed deep learning networks among institutions for medical imaging. Journal of the American Medical Informatics Association, 25(8):945–954, 2018.
  12. Fair mixup: Fairness via interpolation. arXiv preprint arXiv:2103.06503, 2021.
  13. Learning from multiple sources. Journal of Machine Learning Research, 9(8), 2008.
  14. Scaling up dataset distillation to imagenet-1k with constant memory. In International Conference on Machine Learning, pp.  6565–6590. PMLR, 2023.
  15. Model-sharing games: Analyzing federated learning under voluntary participation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  5303–5311, 2021.
  16. Privacy for free: How does dataset condensation help privacy? arXiv preprint arXiv:2206.00240, 2022.
  17. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557–3568, 2020.
  18. Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation. In ICML, pp.  3274–3283, 2021.
  19. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp.  1180–1189. PMLR, 2015.
  20. A survey on heterogeneous federated learning. arXiv preprint arXiv:2210.04505, 2022.
  21. An efficient framework for clustered federated learning. IEEE Transactions on Information Theory, 68(12):8076–8091, 2022.
  22. Preserving privacy in federated learning with ensemble cross-domain knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  11891–11899, 2022.
  23. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  24. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723–773, 2012.
  25. Caltech-256 object category dataset. 2007.
  26. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
  27. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  28. Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10143–10153, 2022.
  29. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.  7865–7873, 2021a.
  30. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34:7232–7241, 2021b.
  31. Hull, J. J. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554, 1994.
  32. Federated distillation and augmentation under non-iid private data. NIPS Wksp. MLPCD, 2018.
  33. Iop-fl: Inside-outside personalization for federated medical image segmentation. IEEE Transactions on Medical Imaging, 2023.
  34. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp.  5132–5143. PMLR, 2020.
  35. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4401–4410, 2019.
  36. Learning multiple layers of features from tiny images. 2009.
  37. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  38. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pp.  12352–12364. PMLR, 2022.
  39. Decentralized federated learning via mutual knowledge transfer. IEEE Internet of Things Journal, 9(2):1136–1147, 2021a.
  40. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  41. Dataset distillation for medical dataset sharing. arXiv preprint arXiv:2209.14603, 2022.
  42. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10713–10722, 2021b.
  43. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020a.
  44. On the convergence of fedavg on non-iid data. International Conference on Learning Representations, 2020b.
  45. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  46. Decentralized stochastic variance reduced extragradient method. arXiv preprint arXiv:2202.00509, 2022.
  47. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34:15434–15447, 2021.
  48. Fedme: Federated learning via model exchange. In Proceedings of the 2022 SIAM international conference on data mining (SDM), pp.  459–467. SIAM, 2022.
  49. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  50. Reading digits in natural images with unsupervised feature learning. 2011.
  51. Ipls: A framework for decentralized federated learning. In 2021 IFIP Networking Conference (IFIP Networking), pp.  1–6. IEEE, 2021.
  52. Rostami, M. Lifelong domain adaptation via consolidated internal distribution. Advances in neural information processing systems, 34:11172–11183, 2021.
  53. Braintorrent: A peer-to-peer environment for decentralized federated learning. arXiv preprint arXiv:1905.06731, 2019.
  54. Adapting visual category models to new domains. In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp.  213–226. Springer, 2010.
  55. Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part I 4, pp.  92–104. Springer, 2019.
  56. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 10(1):12598, 2020.
  57. Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives. Frontiers of Information Technology & Electronic Engineering, 24(10):1390–1402, 2023.
  58. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp.  3–18. IEEE, 2017.
  59. Federated learning via decentralized dataset distillation in resource-constrained edge environments. In 2023 International Joint Conference on Neural Networks (IJCNN), pp.  1–10. IEEE, 2023.
  60. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  8432–8440, 2022.
  61. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. arXiv preprint arXiv:2206.02465, 2022.
  62. Fed-gloss-dp: Federated, global learning using synthetic sets with record level differential privacy. arXiv preprint arXiv:2302.01068, 2023.
  63. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020.
  64. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  12196–12205, 2022.
  65. Communication-efficient federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
  66. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  67. Feddm: Iterative distribution matching for communication-efficient federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16323–16332, 2023.
  68. Regularized mutual learning for personalized federated learning. In Asian Conference on Machine Learning, pp.  1521–1536. PMLR, 2021.
  69. Decentralized accelerated proximal gradient descent. Advances in Neural Information Processing Systems, 33:18308–18317, 2020.
  70. Fedfm: Anchor-based feature matching for data heterogeneity in federated learning. arXiv preprint arXiv:2210.07615, 2022.
  71. Peer-to-peer federated continual learning for naturalistic driving action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5249–5258, 2023a.
  72. Decentralized federated learning: A survey and perspective. arXiv preprint arXiv:2306.01603, 2023b.
  73. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022a.
  74. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10174–10183, 2022b.
  75. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4320–4328, 2018.
  76. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp.  12674–12685. PMLR, 2021.
  77. Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.07513, 2022.
  78. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.  6514–6523, 2023.
  79. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2020.
  80. Fedfa: Federated learning with feature anchors to align feature and classifier for heterogeneous data. arXiv preprint arXiv:2211.09299, 2022.
  81. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning, pp.  12878–12889. PMLR, 2021.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets