Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedMef: Towards Memory-efficient Federated Dynamic Pruning (2403.14737v1)

Published 21 Mar 2024 in cs.LG and cs.DC

Abstract: Federated learning (FL) promotes decentralized training while prioritizing data confidentiality. However, its application on resource-constrained devices is challenging due to the high demand for computation and memory resources to train deep learning models. Neural network pruning techniques, such as dynamic pruning, could enhance model efficiency, but directly adopting them in FL still poses substantial challenges, including post-pruning performance degradation, high activation memory usage, etc. To address these challenges, we propose FedMef, a novel and memory-efficient federated dynamic pruning framework. FedMef comprises two key components. First, we introduce the budget-aware extrusion that maintains pruning efficiency while preserving post-pruning performance by salvaging crucial information from parameters marked for pruning within a given budget. Second, we propose scaled activation pruning to effectively reduce activation memory footprints, which is particularly beneficial for deploying FL to memory-limited devices. Extensive experiments demonstrate the effectiveness of our proposed FedMef. In particular, it achieves a significant reduction of 28.5% in memory footprint compared to state-of-the-art methods while obtaining superior accuracy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375, 2018.
  2. Dual lottery ticket hypothesis. arXiv preprint arXiv:2203.04248, 2022.
  3. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6080–6088, 2022.
  4. Characterizing signal propagation to close the performance gap in unnormalized resnets. arXiv preprint arXiv:2101.08692, 2021.
  5. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning, pages 1803–1813. PMLR, 2021.
  6. Rex: Revisiting budgeted training with an improved schedule. Proceedings of Machine Learning and Systems, 4:64–76, 2022a.
  7. Dropit: Dropping intermediate tensors for memory-efficient dnn training. arXiv preprint arXiv:2202.13808, 2022b.
  8. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.
  9. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505, 2018.
  10. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019.
  11. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943–2952. PMLR, 2020.
  12. Optimal gradient checkpoint search for arbitrary computation graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11433–11442, 2021.
  13. Memory-efficient backpropagation through time. Advances in neural information processing systems, 29, 2016.
  14. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
  15. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  17. Fedtiny: Pruned federated learning towards specialized tiny models. arXiv preprint arXiv:2212.01977, 2022.
  18. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr, 2015.
  19. Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, 39(12):6600, 1989.
  20. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  21. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  22. A Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, University of Tront, 2009.
  23. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  24. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
  25. Federated learning for keyword spotting. In ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6341–6345. IEEE, 2019.
  26. Lotteryfl: Empower edge intelligence with personalized and communication-efficient federated learning. In 2021 IEEE/ACM Symposium on Edge Computing (SEC), pages 68–79. IEEE, 2021a.
  27. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.
  28. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021b.
  29. Adaptive network pruning for wireless federated learning. IEEE Wireless Communications Letters, 10(7):1572–1576, 2021.
  30. Gact: Activation compressed training for generic network architectures. In International Conference on Machine Learning, pages 14139–14152. PMLR, 2022.
  31. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  32. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1–12, 2018.
  33. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272, 2019a.
  34. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings, 2019b.
  35. Skeletonization: A technique for trimming the fat from a network via relevance assessment. Advances in neural information processing systems, 1, 1988.
  36. Fedprune: Towards inclusive federated learning. arXiv preprint arXiv:2110.14205, 2021.
  37. Federated evaluation and tuning for on-device personalization: System design & applications. arXiv preprint arXiv:2102.08503, 2021.
  38. Zerofl: Efficient on-device training for federated learning with local sparsity. arXiv preprint arXiv:2208.02507, 2022.
  39. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  40. Privacy preserving stochastic channel-based federated learning with neural network pruning. arXiv preprint arXiv:1910.02115, 2019.
  41. When to prune? a policy towards early structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12247–12256, 2022.
  42. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098–18109, 2020.
  43. Division: Memory efficient training via dual activation precision. arXiv preprint arXiv:2208.04187, 2023.
  44. Accelerating federated learning for iot in big data analytics with pruning, quantization and selective updating. IEEE Access, 9:38457–38466, 2021.
  45. Is normalization indispensable for multi-domain federated learning? arXiv preprint arXiv:2306.05879, 2023.
  46. Fedwon: Triumphing multi-domain federated learning without normalization. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
  47. Performance optimization of federated person re-identification via benchmark analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pages 955–963, 2020.
  48. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4912–4921, 2021.
  49. Divergence-aware federated self-supervised learning. In The Tenth International Conference on Learning Representations, ICLR. OpenReview.net, 2022.
  50. Mas: Towards resource-efficient federated multiple-task learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23414–23424, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Hong Huang (56 papers)
  2. Weiming Zhuang (21 papers)
  3. Chen Chen (753 papers)
  4. Lingjuan Lyu (131 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets