Papers
Topics
Authors
Recent
Search
2000 character limit reached

FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity

Published 15 Apr 2024 in cs.LG and cs.CR | (2404.09816v1)

Abstract: The interest in federated learning has surged in recent research due to its unique ability to train a global model using privacy-secured information held locally on each client. This paper pays particular attention to the issue of client-side model heterogeneity, a pervasive challenge in the practical implementation of FL that escalates its complexity. Assuming a scenario where each client possesses varied memory storage, processing capabilities and network bandwidth - a phenomenon referred to as system heterogeneity - there is a pressing need to customize a unique model for each client. In response to this, we present an effective and adaptable federated framework FedP3, representing Federated Personalized and Privacy-friendly network Pruning, tailored for model heterogeneity scenarios. Our proposed methodology can incorporate and adapt well-established techniques to its specific instances. We offer a theoretical interpretation of FedP3 and its locally differential-private variant, DP-FedP3, and theoretically validate their efficiencies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.  308–318, 2016.
  2. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. In Advances in Neural Information Processing Systems, 2022.
  3. Geo-indistinguishability: Differential privacy for location-based systems. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pp.  901–914, 2013.
  4. A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory, pp.  111–132. PMLR, 2020.
  5. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pp.  464–473. IEEE, 2014.
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  7. Broadening the scope of differential privacy using metrics. In Privacy Enhancing Technologies: 13th International Symposium, PETS 2013, Bloomington, IN, USA, July 10-12, 2013. Proceedings 13, pp.  82–102. Springer, 2013.
  8. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  9. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pp.  2921–2926. IEEE, 2017.
  10. Only tails matter: Average-case universality and robustness in the convex regime. In International Conference on Machine Learning, pp. 4474–4491. PMLR, 2022.
  11. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2021.
  12. Differentially private and communication efficient collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  7219–7227, 2021.
  13. Resist: Layer-wise decomposition of resnets for distributed training. In Uncertainty in Artificial Intelligence, pp.  610–620. PMLR, 2022.
  14. Efficient and light-weight federated learning via asynchronous distributed dropout. In International Conference on Artificial Intelligence and Statistics, pp.  6630–6660. PMLR, 2023.
  15. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp.  265–284. Springer, 2006.
  16. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  17. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp.  439–449, 2020.
  18. A survey on heterogeneous federated learning. arXiv preprint arXiv:2210.04505, 2022a.
  19. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  10112–10121, June 2022b.
  20. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013.
  21. Super-acceleration with cyclical step-sizes. In International Conference on Artificial Intelligence and Statistics, pp.  3028–3065. PMLR, 2022.
  22. Fednas: Federated deep learning via neural architecture search. 2021.
  23. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  24. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34:12876–12889, 2021.
  25. Fedtiny: Pruned federated learning towards specialized tiny models. arXiv preprint arXiv:2212.01977, 2022.
  26. Privacy-preserving learning via deep net pruning. arXiv preprint arXiv:2003.01876, 2020.
  27. Towards practical differentially private convex optimization. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 299–316. IEEE, 2019.
  28. Communication-efficient distributed dual coordinate ascent. Advances in neural information processing systems, 27, 2014.
  29. Federated learning in smart city sensing: Challenges and opportunities. Sensors, 20(21):6230, 2020.
  30. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022a.
  31. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022b.
  32. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  33. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132–5143. PMLR, 2020.
  34. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329, 2020.
  35. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016.
  36. Learning multiple layers of features from tiny images. 2009.
  37. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  38. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10713–10722, 2021.
  39. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020a.
  40. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020b.
  41. Simple and optimal stochastic gradient methods for nonsmooth nonconvex optimization. The Journal of Machine Learning Research, 23(1):10891–10951, 2022.
  42. Soteriafl: A unified framework for private federated learning with communication compression. Advances in Neural Information Processing Systems, 35:4285–4300, 2022.
  43. Adaptive channel sparsity for federated learning under system heterogeneity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  20432–20441, 2023.
  44. On the convergence of shallow neural network training with randomly masked neurons. Transactions on Machine Learning Research, 2022.
  45. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  46. Private non-convex federated learning without a trusted server. In International Conference on Artificial Intelligence and Statistics, pp.  5749–5786. PMLR, 2023.
  47. Adding vs. averaging in distributed primal-dual optimization. In International Conference on Machine Learning, pp. 1973–1982. PMLR, 2015.
  48. Variance reduced Proxskip: Algorithm, theory and application to federated learning. NeurIPS, 2022.
  49. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  50. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  8397–8406, June 2022.
  51. ProxSkip: Yes! Local gradient steps provably lead to communication acceleration! Finally! In 39th International Conference on Machine Learning (ICML 2022), 2022.
  52. Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
  53. Smoothness matrices beat smoothness constants: Better communication compression techniques for distributed optimization. Advances in Neural Information Processing Systems, 34:25688–25702, 2021.
  54. Towards a better theoretical understanding of independent subnetwork training. arXiv preprint arXiv:2306.16484, 2023.
  55. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
  56. Permutation compressors for provably faster distributed nonconvex optimization. arXiv preprint arXiv:2110.03300, 2021.
  57. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  8432–8440, 2022.
  58. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  59. Theoretically better and numerically faster distributed optimization with smoothness-aware quantization techniques. Advances in Neural Information Processing Systems, 35:9841–9852, 2022.
  60. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems, 30, 2017.
  61. Gist: Distributed training for large-scale graph convolutional networks. Journal of Applied and Computational Topology, pp.  1–53, 2023.
  62. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  63. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3):1–44, 2023.
  64. Explicit personalization and local training: Double communication acceleration in federated learning. arXiv preprint arXiv:2305.13170, 2023.
  65. Distributed learning of fully connected neural networks using independent subnet training. Proceedings of the VLDB Endowment, 15(8):1581–1590, 2022.
  66. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 818–833. Springer, 2014.
  67. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019.
  68. Fedcr: Personalized federated learning based on across-client common representation with conditional mutual information regularization. 2023.
  69. Private and communication-efficient edge learning: a sparse differential gaussian-masking distributed sgd approach. In Proceedings of the Twenty-First International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, pp.  261–270, 2020.
  70. Local differential privacy-based federated learning for internet of things. IEEE Internet of Things Journal, 8(11):8836–8853, 2020.
  71. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
  72. Quadratic models for understanding neural network dynamics. arXiv preprint arXiv:2205.11787, 2022.
  73. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
Citations (7)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up