Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LEFL: Low Entropy Client Sampling in Federated Learning (2312.17430v2)

Published 29 Dec 2023 in cs.LG

Abstract: Federated learning (FL) is a machine learning paradigm where multiple clients collaborate to optimize a single global model using their private data. The global model is maintained by a central server that orchestrates the FL training process through a series of training rounds. In each round, the server samples clients from a client pool before sending them its latest global model parameters for further optimization. Naive sampling strategies implement random client sampling and fail to factor client data distributions for privacy reasons. Hence we propose LEFL, an alternative sampling strategy by performing a one-time clustering of clients based on their model's learned high-level features while respecting data privacy. This enables the server to perform stratified client sampling across clusters in every round. We show datasets of sampled clients selected with this approach yield a low relative entropy with respect to the global data distribution. Consequently, the FL training becomes less noisy and significantly improves the convergence of the global model by as much as 7.4% in some experiments. Furthermore, it also significantly reduces the communication rounds required to achieve a target accuracy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Optimizing decentralized learning with local heterogeneity using topology morphing and clustering. In 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 355–366. IEEE, 2023.
  2. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243, 2020.
  3. Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics, pages 10351–10375. PMLR, 2022.
  4. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921–2926. IEEE, 2017.
  5. Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications. In 2019 IEEE 37th international conference on computer design (ICCD), pages 246–254. IEEE, 2019.
  6. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. In International Conference on Machine Learning, pages 3407–3416. PMLR, 2021.
  7. Privacy-preserving heterogeneous federated transfer learning. In 2019 IEEE international conference on big data (Big Data), pages 2552–2559. IEEE, 2019.
  8. Active federated learning. arXiv preprint arXiv:1909.12641, 2019.
  9. A k-means clustering algorithm. JSTOR: Applied Statistics, 28(1):100–108, 1979.
  10. Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2020.
  11. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  12. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  13. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  14. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  15. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020.
  16. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMLR, 2019.
  17. Learning multiple layers of features from tiny images. 2009.
  18. S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 – 86, 1951.
  19. FedMD: Heterogenous Federated Learning via Model Distillation. 2019.
  20. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965–978. IEEE, 2022.
  21. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020.
  22. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.
  23. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM 2022-IEEE conference on computer communications, pages 1739–1748. IEEE, 2022.
  24. Cmfl: Mitigating communication overhead for federated learning. In 2019 IEEE 39th international conference on distributed computing systems (ICDCS), pages 954–964. IEEE, 2019.
  25. Communication-efficient learning of deep networks from decentralized data. In Proc. of International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
  26. Optimal importance sampling for federated learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3095–3099. IEEE, 2021.
  27. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 31(9):3400–3413, 2019.
  28. Federated Knowledge Distillation. Machine Learning and Wireless Communications, page 457, 2022.
  29. Model compression for communication efficient federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2021.
  30. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  31. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020.
  32. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  33. Ternary compression for communication-efficient federated learning. IEEE Transactions on Neural Networks and Learning Systems, 33(3):1162–1176, 2020.
  34. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO), pages 2174–2178. IEEE, 2021.
  35. Spatl: Salient parameter aggregation and transfer learning for heterogeneous clients in federated learning. arXiv preprint arXiv:2111.14345, 2021.
  36. Spatl: salient parameter aggregation and transfer learning for heterogeneous federated learning. In 2022 SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (SC), pages 495–508. IEEE Computer Society, 2022.
  37. Resource-aware heterogeneous federated learning using neural architecture search. arXiv preprint arXiv:2211.05716, 2022.
  38. Curse or redemption? how data heterogeneity affects the robustness of federated learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10807–10814, 2021.
  39. Fed-cbs: A heterogeneity-aware client sampling mechanism for federated learning via class-imbalance reduction. In International Conference on Machine Learning, pages 41354–41381. PMLR, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Waqwoya Abebe (6 papers)
  2. Pablo Munoz (10 papers)
  3. Ali Jannesari (56 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now