Harnessing Increased Client Participation with Cohort-Parallel Federated Learning (2405.15644v2)
Abstract: Federated learning (FL) is a machine learning approach where nodes collaboratively train a global model. As more nodes participate in a round of FL, the effectiveness of individual model updates by nodes also diminishes. In this study, we increase the effectiveness of client updates by dividing the network into smaller partitions, or cohorts. We introduce Cohort-Parallel Federated Learning (CPFL): a novel learning approach where each cohort independently trains a global model using FL, until convergence, and the produced models by each cohort are then unified using knowledge distillation. The insight behind CPFL is that smaller, isolated networks converge quicker than in a one-network setting where all nodes participate. Through exhaustive experiments involving realistic traces and non-IID data distributions on the CIFAR-10 and FEMNIST image classification tasks, we investigate the balance between the number of cohorts, model accuracy, training time, and compute resources. Compared to traditional FL, CPFL with four cohorts, non-IID data distribution, and CIFAR-10 yields a 1.9x reduction in train time and a 1.3x reduction in resource usage, with a minimal drop in test accuracy.
- REFL: Resource-Efficient Federated Learning.
- A Theory of Learning from Different Domains. Mach. Learn., 79(1–2): 151–175.
- Distributed distillation for on-device learning. Advances in Neural Information Processing Systems, 33: 22593–22604.
- Towards federated learning at scale: System design. Proceedings of machine learning and systems, 1: 374–388.
- Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In 2020 International Joint Conference on Neural Networks (IJCNN), 1–9.
- Leaf: A benchmark for federated settings. In 2nd Intl. Workshop on Federated Learning for Data Privacy and Confidentiality (FL-NeurIPS).
- On large-cohort training for federated learning. Advances in neural information processing systems, 34: 20461–20475.
- An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, 215–223. JMLR Workshop and Conference Proceedings.
- TEE-based decentralized recommender systems: The raw data sharing redemption. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 447–458. IEEE.
- Decentralized learning made easy with DecentralizePy. In Proceedings of the 3rd Workshop on Machine Learning and Systems, 34–41.
- Flexible clustered federated learning for client-level data distribution shift. IEEE Transactions on Parallel and Distributed Systems, 33(11): 2661–2674.
- An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33: 19586–19597.
- Preserving privacy in federated learning with ensemble cross-domain knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 11891–11899.
- Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 33: 14068–14080.
- Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop.
- The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, 4387–4398. PMLR.
- Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.
- Mobiperf: Mobile network measurement system. Technical Report. University of Michigan and Microsoft Research.
- Papaya: Practical, private, and scalable federated learning. Proceedings of Machine Learning and Systems, 4: 814–832.
- Ai benchmark: All about deep learning on smartphones in 2019. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 3617–3635. IEEE.
- Learning multiple layers of features from tiny images.
- Fedscale: Benchmarking model and system performance of federated learning at scale. In International Conference on Machine Learning, 11814–11827. PMLR.
- Oort: Efficient Federated Learning via Guided Participant Selection. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), 19–35. USENIX Association. ISBN 978-1-939133-22-9.
- Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems, 2.
- FedMD: Heterogenous Federated Learning via Model Distillation. NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality.
- Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2: 429–450.
- Ensemble Distillation for Robust Model Fusion in Federated Learning. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 2351–2363. Curran Associates, Inc.
- Auxo: Heterogeneity-Mitigating Federated Learning via Scalable Client Clustering. arXiv:2210.16656.
- Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, 1273–1282. PMLR.
- CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding. IEEE Transactions on Network Science and Engineering, 9(4): 2025–2038.
- Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8): 3710–3722.
- Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 21st international conference on pattern recognition (ICPR2012), 3288–3291. IEEE.
- Decentralized Learning with Multi-Headed Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8053–8063.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.