Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Fake It Till Make It: Federated Learning with Consensus-Oriented Generation (2312.05966v1)

Published 10 Dec 2023 in cs.LG and cs.CV

Abstract: In federated learning (FL), data heterogeneity is one key bottleneck that causes model divergence and limits performance. Addressing this, existing methods often regard data heterogeneity as an inherent property and propose to mitigate its adverse effects by correcting models. In this paper, we seek to break this inherent property by generating data to complement the original dataset to fundamentally mitigate heterogeneity level. As a novel attempt from the perspective of data, we propose federated learning with consensus-oriented generation (FedCOG). FedCOG consists of two key components at the client side: complementary data generation, which generates data extracted from the shared global model to complement the original dataset, and knowledge-distillation-based model training, which distills knowledge from global model to local model based on the generated data to mitigate over-fitting the original heterogeneous dataset. FedCOG has two critical advantages: 1) it can be a plug-and-play module to further improve the performance of most existing FL methods, and 2) it is naturally compatible with standard FL protocols such as Secure Aggregation since it makes no modification in communication process. Extensive experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2020.
  2. Practical secure aggregation for privacy-preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp.  1175–1191, 2017.
  3. Optimization methods for large-scale machine learning. Siam Review, 60(2):223–311, 2018.
  4. Fraug: Tackling federated learning with non-iid features via representation augmentation. arXiv preprint arXiv:2205.14900, 2022.
  5. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243, 2020.
  6. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pp.  2089–2099. PMLR, 2021.
  7. Fedavg with fine tuning: Local updates lead to representation learning. Advances in Neural Information Processing Systems, 35:10572–10586, 2022.
  8. Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine, 27(10):1735–1743, 2021.
  9. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2020.
  10. Data-free adversarial distillation. arXiv preprint arXiv:1912.11006, 2019.
  11. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp.  1322–1333, 2015.
  12. Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3310–3319, 2021.
  13. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.
  14. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  15. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34:12876–12889, 2021.
  16. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
  17. Fedexp: Speeding up federated averaging via extrapolation. arXiv preprint arXiv:2301.09604, 2023.
  18. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.
  19. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp.  5132–5143. PMLR, 2020.
  20. Learning multiple layers of features from tiny images. 2009.
  21. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951.
  22. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  23. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10713–10722, 2021a.
  24. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60, 2020a.
  25. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020b.
  26. Fair resource allocation in federated learning. In International Conference on Learning Representations, 2020c.
  27. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp.  6357–6368. PMLR, 2021b.
  28. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019.
  29. Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151, 1991.
  30. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  31. Federated learning for open banking. In Federated Learning: Privacy and Incentive, pp.  240–254. Springer, 2020.
  32. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5188–5196, 2015.
  33. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120:233–255, 2016.
  34. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  35. Inceptionism: Going deeper into neural networks. 2015.
  36. Zero-shot knowledge distillation in deep networks. In International Conference on Machine Learning, pp.  4743–4751. PMLR, 2019.
  37. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, pp.  18250–18280. PMLR, 2022.
  38. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=LkFG3lB13U5.
  39. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. arXiv preprint arXiv:2210.00226, 2022.
  40. Lightsecagg: a lightweight and versatile design for secure aggregation in federated learning. Proceedings of Machine Learning and Systems, 4:694–720, 2022.
  41. Flair: Federated learning annotated image repository. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  42. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In International Conference on Machine Learning, pp.  21111–21132. PMLR, 2022.
  43. Federated learning with matched averaging. In International Conference on Learning Representations, 2020a. URL https://openreview.net/forum?id=BkluqlSFDS.
  44. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020b.
  45. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021.
  46. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  47. Acceleration of federated learning with alleviated forgetting in local training, 2022.
  48. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 5(1):1–19, 2021.
  49. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019.
  50. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018.
  51. Personalized federated learning with inferred collaboration graphs. 2023a.
  52. Fedfm: Anchor-based feature matching for data heterogeneity in federated learning. IEEE Transactions on Signal Processing, 2023b.
  53. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  8715–8724, 2020.
  54. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pp.  12073–12086. PMLR, 2021.
  55. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.  5693–5700, 2019.
  56. Addressing heterogeneity in federated learning via distributional transformation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVIII, pp.  179–195. Springer, 2022.
  57. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pp.  7252–7261. PMLR, 2019.
  58. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414–21428, 2022a.
  59. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10174–10183, 2022b.
  60. Trading off privacy, utility and efficiency in federated learning. arXiv preprint arXiv:2209.00230, 2022c.
  61. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
  62. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
  63. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning, pp.  12878–12889. PMLR, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Rui Ye (42 papers)
  2. Yaxin Du (10 papers)
  3. Zhenyang Ni (7 papers)
  4. Siheng Chen (152 papers)
  5. Yanfeng Wang (211 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now