Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Gradient-Congruity Guided Federated Sparse Training (2405.01189v1)

Published 2 May 2024 in cs.LG and cs.AI

Abstract: Edge computing allows artificial intelligence and machine learning models to be deployed on edge devices, where they can learn from local data and collaborate to form a global model. Federated learning (FL) is a distributed machine learning technique that facilitates this process while preserving data privacy. However, FL also faces challenges such as high computational and communication costs regarding resource-constrained devices, and poor generalization performance due to the heterogeneity of data across edge clients and the presence of out-of-distribution data. In this paper, we propose the Gradient-Congruity Guided Federated Sparse Training (FedSGC), a novel method that integrates dynamic sparse training and gradient congruity inspection into federated learning framework to address these issues. Our method leverages the idea that the neurons, in which the associated gradients with conflicting directions with respect to the global model contain irrelevant or less generalized information for other clients, and could be pruned during the sparse training process. Conversely, the neurons where the associated gradients with consistent directions could be grown in a higher priority. In this way, FedSGC can greatly reduce the local computation and communication overheads while, at the same time, enhancing the generalization abilities of FL. We evaluate our method on challenging non-i.i.d settings and show that it achieves competitive accuracy with state-of-the-art FL methods across various scenarios while minimizing computation and communication costs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=B7v4QMR6Z9w.
  2. Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. Machine Learning, pp.  1–38, 2022.
  3. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  6080–6088, 2022.
  4. Communication-efficient federated learning. Proceedings of the National Academy of Sciences, 118(17):e2024789118, 2021.
  5. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.  7314–7322, 2023.
  6. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp.  2943–2952. PMLR, 2020.
  7. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp.  3259–3269. PMLR, 2020.
  8. google. Convolutional neural network (cnn), 1999. URL https://www.tensorflow.org/tutorials/images/cnn.
  9. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  10. SCAFFOLD: Stochastic controlled averaging for federated learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  5132–5143. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/karimireddy20a.html.
  11. Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets. arXiv preprint arXiv:2008.03371, 2020.
  12. Lotteryfl: Empower edge intelligence with personalized and communication-efficient federated learning. In 2021 IEEE/ACM Symposium on Edge Computing (SEC), pp.  68–79. IEEE, 2021a.
  13. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10713–10722, 2021b.
  14. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. arXiv preprint arXiv:2106.14568, 2021.
  15. Domain generalization via gradient surgery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  6630–6638, 2021.
  16. Communication-efficient learning of deep networks from decentralized data. In AISTAT, 2017.
  17. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018.
  18. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization, 2019. URL https://openreview.net/forum?id=S1xBioR5KX.
  19. Fedltn: Federated learning for sparse and personalized lottery ticket networks. In European Conference on Computer Vision, pp.  69–85. Springer, 2022.
  20. Fedsr: A simple and effective domain generalization method for federated learning. Advances in Neural Information Processing Systems, 35:38831–38843, 2022.
  21. Training adversarially robust sparse networks via bayesian connectivity sampling. In International Conference on Machine Learning, pp.  8314–8324. PMLR, 2021.
  22. Spacenet: Make free space for continual learning. Neurocomputing, 439:1–11, 2021a.
  23. Dynamic sparse training for deep reinforcement learning (poster). In Sparsity in Neural Networks: Advancing Understanding and Practice 2021, 2021b.
  24. Fedselect: Customized selection of parameters for fine-tuning during personalized federated learning. arXiv preprint arXiv:2306.13264, 2023.
  25. Privacy-preserving constrained domain generalization via gradient alignment. IEEE Transactions on Knowledge and Data Engineering, 2023.
  26. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkgsACVKPH.
  27. Communication-efficient federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
  28. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13(3):1–207, 2019.
  29. Federated learning with unbiased gradient aggregation and controllable meta updating. arXiv preprint arXiv:1910.08234, 2019.
  30. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836, 2020.
  31. Federated domain generalization with generalization adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3954–3963, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Chris Xing Tian (7 papers)
  2. Yibing Liu (12 papers)
  3. Haoliang Li (67 papers)
  4. Ray C. C. Cheung (9 papers)
  5. Shiqi Wang (162 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets