Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedSAC: Dynamic Submodel Allocation for Collaborative Fairness in Federated Learning (2405.18291v1)

Published 28 May 2024 in cs.LG, cs.AI, and cs.DC

Abstract: Collaborative fairness stands as an essential element in federated learning to encourage client participation by equitably distributing rewards based on individual contributions. Existing methods primarily focus on adjusting gradient allocations among clients to achieve collaborative fairness. However, they frequently overlook crucial factors such as maintaining consistency across local models and catering to the diverse requirements of high-contributing clients. This oversight inevitably decreases both fairness and model accuracy in practice. To address these issues, we propose FedSAC, a novel Federated learning framework with dynamic Submodel Allocation for Collaborative fairness, backed by a theoretical convergence guarantee. First, we present the concept of "bounded collaborative fairness (BCF)", which ensures fairness by tailoring rewards to individual clients based on their contributions. Second, to implement the BCF, we design a submodel allocation module with a theoretical guarantee of fairness. This module incentivizes high-contributing clients with high-performance submodels containing a diverse range of crucial neurons, thereby preserving consistency across local models. Third, we further develop a dynamic aggregation module to adaptively aggregate submodels, ensuring the equitable treatment of low-frequency neurons and consequently enhancing overall model accuracy. Extensive experiments conducted on three public benchmarks demonstrate that FedSAC outperforms all baseline methods in both fairness and model accuracy. We see this work as a significant step towards incentivizing broader client participation in federated learning. The source code is available at https://github.com/wangzihuixmu/FedSAC.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Anish Agarwal et al. 2019. A marketplace for data: An algorithmic solution. In Proceedings of the 2019 ACM Conference on Economics and Computation. 701–726.
  2. Chen Chen et al. 2022. Gear: a margin-based federated adversarial training approach. In International Workshop on Trustable, Verifiable, and Auditable Federated Learning in Conjunction with AAAI, Vol. 2022.
  3. Elastic Aggregation for Federated Optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12187–12197.
  4. FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. In NeurIPS 2022-Thirty-sixth Conference on Neural Information Processing Systems.
  5. FedDC: Federated Learning With Non-IID Data via Local Drift Decoupling and Correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10112–10121.
  6. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10112–10121.
  7. Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization. In In Proceedings of the International Conference on Learning Representations.
  8. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems 34 (2021), 12876–12889.
  9. Learn From Others and Be Yourself in Heterogeneous Federated Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10143–10153.
  10. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 7865–7873.
  11. Federated learning for cyber security: SOC collaboration for malicious URL detection. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS). IEEE, 1316–1321.
  12. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009. (2009).
  13. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
  14. Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems. 42–55.
  15. On the Effectiveness of Partial Variance Reduction in Federated Learning With Heterogeneous Data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3964–3973.
  16. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2 (2020), 429–450.
  17. Fair Resource Allocation in Federated Learning. In International Conference on Learning Representations.
  18. On the Convergence of FedAvg on Non-IID Data. In International Conference on Learning Representations.
  19. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems 33 (2020), 2351–2363.
  20. Jian-Hao Luo et al. 2017. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision. 5058–5066.
  21. How to democratise and protect AI: Fair and differentially private decentralised deep learning. IEEE Transactions on Dependable and Secure Computing (2020).
  22. Collaborative fairness in federated learning. In Federated Learning. Springer, 189–204.
  23. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics. PMLR, 1273–1282.
  24. Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8397–8406.
  25. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11264–11272.
  26. Pruning Convolutional Neural Networks for Resource Efficient Inference. In In Proceedings of the International Conference on Learning Representations.
  27. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 (2011).
  28. FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. Advances in Neural Information Processing Systems 35 (2022), 5315–5334.
  29. FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1954–1964.
  30. Adam Richardson et al. 2020. Budget-bounded incentives for federated learning. In Federated Learning. Springer, 176–188.
  31. Yuxin Shi et al. 2021. A survey of fairness-aware federated learning. arXiv preprint arXiv:2111.01872 (2021).
  32. Towards fairness-aware federated learning. IEEE Transactions on Neural Networks and Learning Systems (2023).
  33. Collaborative machine learning with incentive-aware model rewards. In International Conference on Machine Learning. PMLR, 8927–8936.
  34. Sebastian U Stich. 2018. Local SGD converges fast and communicates little. arXiv preprint arXiv:1805.09767 (2018).
  35. Sparsified SGD with memory. Advances in Neural Information Processing Systems 31 (2018).
  36. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems 34 (2021), 23309–23320.
  37. Federated learning on non-iid graphs via structural knowledge sharing. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 9953–9961.
  38. Incentivizing collaboration in machine learning via synthetic data rewards. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 9448–9456.
  39. Mutual information driven federated learning. IEEE Transactions on Parallel and Distributed Systems 32, 7 (2020), 1526–1538.
  40. A principled approach to data valuation for federated learning. Federated Learning: Privacy and Incentive (2020), 153–167.
  41. Theoretical Convergence Guaranteed Resource-Adaptive Federated Learning with Mixed Heterogeneity. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2444–2455.
  42. FedAVE: Adaptive data value evaluation framework for collaborative fairness in federated learning. Neurocomputing (2024), 127227.
  43. Xinyi Xu and Lingjuan Lyu. 2020. A reputation mechanism is all you need: Collaborative fairness and adversarial robustness in federated learning. arXiv preprint arXiv:2011.10464 (2020).
  44. Gradient driven rewards to guarantee fairness in collaborative machine learning. Advances in Neural Information Processing Systems 34 (2021), 16104–16117.
  45. Bias-Eliminating Augmentation Learning for Debiased Federated Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 20442–20452.
  46. Criticalfl: A critical learning periods augmented client selection framework for efficient federated learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2898–2907.
  47. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) 10, 2 (2019), 1–19.
  48. A fairness-aware incentive scheme for federated learning. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 393–399.
  49. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 5693–5700.
  50. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 9194–9203.
  51. TCT: Convexifying federated learning using bootstrapped neural tangent kernels. Advances in Neural Information Processing Systems 35 (2022), 30882–30897.
  52. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning. PMLR, 7252–7261.
  53. A survey of incentive mechanism design for federated learning. IEEE Transactions on Emerging Topics in Computing (2021).
  54. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 11237–11244.
  55. Incentive mechanism for horizontal federated learning based on reputation and reverse auction. In Proceedings of the Web Conference 2021. 947–956.
  56. Communication-efficient algorithms for statistical optimization. Advances in neural information processing systems 25 (2012).

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets