Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization (2203.05468v3)

Published 10 Mar 2022 in cs.LG

Abstract: Devices participating in federated learning (FL) typically have heterogeneous communication, computation, and memory resources. However, in synchronous FL, all devices need to finish training by the same deadline dictated by the server. Our results show that training a smaller subset of the neural network (NN) at constrained devices, i.e., dropping neurons/filters as proposed by state of the art, is inefficient, preventing these devices to make an effective contribution to the model. This causes unfairness w.r.t the achievable accuracies of constrained devices, especially in cases with a skewed distribution of class labels across devices. We present a novel FL technique, CoCoFL, which maintains the full NN structure on all devices. To adapt to the devices' heterogeneous resources, CoCoFL freezes and quantizes selected layers, reducing communication, computation, and memory requirements, whereas other layers are still trained in full precision, enabling to reach a high accuracy. Thereby, CoCoFL efficiently utilizes the available resources on devices and allows constrained devices to make a significant contribution to the FL system, increasing fairness among participants (accuracy parity) and significantly improving the final accuracy of the model.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. New directions in distributed deep learning: Bringing the network at forefront of iot design. In Design Automation Conference (DAC). IEEE, 2020.
  2. Leaf: A benchmark for federated settings. arXiv:1812.01097, 2019.
  3. Communication-efficient federated learning with adaptive parameter freezing. In Int. Conference on Distributed Computing Systems (ICDCS). IEEE, 2021.
  4. Emnist: an extension of mnist to handwritten letters. arXiv:1702.05373, 2017.
  5. Cinic-10 is not imagenet or cifar-10. arXiv:1810.03505, 2018.
  6. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Trans. Internet Things, 2(3), jul 2021. ISSN 2691-1914. doi: 10.1145/3450494.
  7. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations (ICLR), 2020.
  8. Residual knowledge distillation. arXiv:2002.09168, 2020.
  9. Andrea Goldsmith. Wireless Communications. Cambridge University Press, 2005. doi: 10.1017/CBO9780511841224.
  10. Layerout: Freezing layers in deep neural networks. SN Computer Science, 1(5), 2020.
  11. Yunhui Guo. A survey on methods and theories of quantized neural networks. arXiv:1808.04752, 2018.
  12. Deep learning with limited numerical precision. In International Conference on Machine Learning (ICML), pp. 1737–1746. PMLR, 2015.
  13. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  14. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. In Advances in Neural Information Processing Systems, volume 34. NeurIPS, 2021.
  15. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017.
  16. Measuring the effects of non-identical data distribution for federated visual classification. arXiv:1909.06335, 2019.
  17. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.  2261–2269, 2017. doi: 10.1109/CVPR.2017.243.
  18. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. pmlr, 2015.
  19. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2704–2713, 2018.
  20. Learning multiple layers of features from tiny images, 2009.
  21. Training quantized nets: A deeper understanding. In International Conference on Neural Information Processing Systems (NeurIPS), pp.  5813–5823, 2017.
  22. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, volume 2, pp. 429–450, 2020.
  23. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp.  142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
  24. Towards fair federated recommendation learning: Characterizing the inter-dependence of system and data heterogeneity. In Proceedings of the 16th ACM Conference on Recommender Systems, pp.  156–167, 2022.
  25. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp.  1273–1282, 2017.
  26. Mixed precision training. In International Conference on Learning Representations (ICLR), 2018.
  27. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeurIPS). NeurIPS, 2019.
  28. Distreal: Distributed resource-aware learning in heterogeneous systems. In AAAI Conference on Artificial Intelligence (AAAI). AAAI, 2022.
  29. Scaling language model size in cross-device federated learning. In Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022), pp.  6–20, 2022.
  30. Communication-Efficient Edge AI: Algorithms and Systems. IEEE Communications Surveys & Tutorials, 22(4):2167–2191, 2020.
  31. A survey of fairness-aware federated learning. arXiv:2111.01872, 2021.
  32. Compressing RNNs for IoT Devices by 15-38x using Kronecker Products. arXiv:1906.02876, 2019.
  33. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  34. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  35. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp.  3–19, 2018.
  36. Asynchronous federated optimization. arXiv:1903.03934, 2020.
  37. Helios: Heterogeneity-aware federated learning with dynamically balanced collaboration. In Design Automation Conference (DAC). IEEE, 2021.
  38. Partial variable training for efficient on-device federated learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  4348–4352. IEEE, 2022.
  39. Applied federated learning: Improving google keyboard query suggestions. arXiv:1812.02903, 2018.
  40. Energy efficient federated learning over wireless communication networks. IEEE Trans. on Wireless Communications (TWC), 20(3):1935–1949, 2020.
  41. Recent trends in deep learning based natural language processing. IEEE Computational Intelligence magazine (CIM), 13(3):55–75, 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Kilian Pfeiffer (9 papers)
  2. Martin Rapp (9 papers)
  3. Ramin Khalili (26 papers)
  4. Jörg Henkel (44 papers)
Citations (11)
View on Semantic Scholar