Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

P4: Towards private, personalized, and Peer-to-Peer learning (2405.17697v2)

Published 27 May 2024 in cs.LG

Abstract: Personalized learning is a proposed approach to address the problem of data heterogeneity in collaborative machine learning. In a decentralized setting, the two main challenges of personalization are client clustering and data privacy. In this paper, we address these challenges by developing P4 (Personalized Private Peer-to-Peer) a method that ensures that each client receives a personalized model while maintaining differential privacy guarantee of each client's local dataset during and after the training. Our approach includes the design of a lightweight algorithm to identify similar clients and group them in a private, peer-to-peer (P2P) manner. Once grouped, we develop differentially-private knowledge distillation for clients to co-train with minimal impact on accuracy. We evaluate our proposed method on three benchmark datasets (FEMNIST or Federated EMNIST, CIFAR-10 and CIFAR-100) and two different neural network architectures (Linear and CNN-based networks) across a range of privacy parameters. The results demonstrate the potential of P4, as it outperforms the state-of-the-art of differential private P2P by up to 40 percent in terms of accuracy. We also show the practicality of P4 by implementing it on resource constrained devices, and validating that it has minimal overhead, e.g., about 7 seconds to run collaborative training between two clients.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Joakim Andén and Stéphane Mallat. 2014. Deep scattering spectrum. IEEE Transactions on Signal Processing 62, 16 (2014), 4114–4128.
  2. Privacy-preserving deep learning via additively homomorphic encryption. IEEE transactions on information forensics and security 13, 5 (2017), 1333–1345.
  3. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems 32 (2019).
  4. Differentially Private Decentralized Deep Learning with Consensus Algorithms. arXiv preprint arXiv:2306.13892 (2023).
  5. Personalized and private peer-to-peer machine learning. In International Conference on Artificial Intelligence and Statistics. PMLR, 473–481.
  6. Practical secure aggregation for privacy-preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 1175–1191.
  7. Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–9.
  8. Joan Bruna and Stéphane Mallat. 2013. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1872–1886.
  9. EMNIST: Extending MNIST to handwritten letters. In 2017 international joint conference on neural networks (IJCNN). IEEE, 2921–2926.
  10. Exploiting shared representations for personalized federated learning. In International conference on machine learning. PMLR, 2089–2099.
  11. A survey on application of machine learning for Internet of Things. International Journal of Machine Learning and Cybernetics 9 (2018), 1399–1417.
  12. Dispfl: Towards communication-efficient personalized federated learning via decentralized sparse training. arXiv preprint arXiv:2206.00187 (2022).
  13. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Transactions on Internet of Things 2, 3 (2021), 1–49.
  14. TEE-based decentralized recommender systems: The raw data sharing redemption. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 447–458.
  15. Flexible clustered federated learning for client-level data distribution shift. IEEE Transactions on Parallel and Distributed Systems 33, 11 (2021), 2661–2674.
  16. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 (2020).
  17. Inverting gradients-how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems 33 (2020), 16937–16947.
  18. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557 (2017).
  19. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems 33 (2020), 19586–19597.
  20. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019).
  21. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems 34 (2021), 7232–7241.
  22. Eunjeong Jeong and Marios Kountouris. 2023. Personalized Decentralized Federated Learning with Knowledge Distillation. arXiv preprint arXiv:2302.12156 (2023).
  23. FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning System. arXiv preprint arXiv:2303.10837 (2023).
  24. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning 14, 1–2 (2021), 1–210.
  25. Decentralized federated learning through proxy model sharing. Nature communications 14, 1 (2023), 2899.
  26. Learning multiple layers of features from tiny images. (2009).
  27. Peer-to-peer federated learning on graphs. arXiv preprint arXiv:1901.11173 (2019).
  28. Fully decentralized federated learning. In Third workshop on bayesian deep learning (NeurIPS), Vol. 2.
  29. Learning to collaborate in decentralized learning of personalized models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9766–9775.
  30. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning. PMLR, 6357–6368.
  31. Asynchronous federated learning with differential privacy for edge intelligence. arXiv preprint arXiv:1912.07902 (2019).
  32. Towards effective clustered federated learning: A peer-to-peer framework with adaptive neighbor matching. IEEE Transactions on Big Data (2022).
  33. Nike-based fast privacy-preserving highdimensional data aggregation for mobile devices. IEEE T Depend Secure (2018), 142–149.
  34. Christopher Manning and Hinrich Schutze. 1999. Foundations of statistical natural language processing. MIT press.
  35. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273–1282.
  36. Ilya Mironov. 2017. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF). IEEE, 263–275.
  37. PPFL: privacy-preserving federated learning with trusted execution environments. In Proceedings of the 19th annual international conference on mobile systems, applications, and services. 94–108.
  38. SoK: machine learning with confidential computing. arXiv preprint arXiv:2208.10134 (2022).
  39. Self-organizing democratized learning: Toward large-scale distributed learning systems. IEEE Transactions on Neural Networks and Learning Systems (2022).
  40. Differentially private federated learning on heterogeneous data. In International Conference on Artificial Intelligence and Statistics. PMLR, 10110–10145.
  41. Edouard Oyallon and Stéphane Mallat. 2015. Deep roto-translation scattering for object classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2865–2873.
  42. Braintorrent: A peer-to-peer environment for decentralized federated learning. arXiv preprint arXiv:1905.06731 (2019).
  43. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems 32, 8 (2020), 3710–3722.
  44. Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training. arXiv preprint arXiv:2305.15157 (2023).
  45. Improving the model consistency of decentralized federated learning. arXiv preprint arXiv:2302.04083 (2023).
  46. Decentralized federated averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 4 (2022), 4289–4301.
  47. Florian Tramer and Dan Boneh. 2020. Differentially private learning needs better features (or much more data). arXiv preprint arXiv:2011.11660 (2020).
  48. A hybrid approach to privacy-preserving federated learning. In Proceedings of the 12th ACM workshop on artificial intelligence and security. 1–11.
  49. LDP-Fed: Federated learning with local differential privacy. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking. 61–66.
  50. Accelerating decentralized federated learning in heterogeneous edge computing. IEEE Transactions on Mobile Computing (2022).
  51. Enhancing privacy preservation and trustworthiness for decentralized federated learning. Information Sciences 628 (2023), 449–468.
  52. Swarm learning for decentralized and confidential clinical machine learning. Nature 594, 7862 (2021), 265–270.
  53. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security 15 (2020), 3454–3469.
  54. Multi-center federated learning. arXiv preprint arXiv:2108.08647 (2021).
  55. Decentralized accelerated proximal gradient descent. Advances in Neural Information Processing Systems 33 (2020), 18308–18317.
  56. Fully decentralized joint learning of personalized models and collaboration graphs. In International Conference on Artificial Intelligence and Statistics. PMLR, 864–874.
  57. Private Node Selection in Personalized Decentralized Learning. arXiv preprint arXiv:2301.12755 (2023).
  58. Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.08565 (2020).
  59. A privacy-preserving and verifiable federated learning scheme. In ICC 2020-2020 IEEE International Conference on Communications (ICC). IEEE, 1–6.
  60. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4320–4328.
  61. PVD-FL: A privacy-preserving and verifiable decentralized federated learning framework. IEEE Transactions on Information Forensics and Security 17 (2022), 2059–2073.
  62. Helen: Maliciously secure coopetitive learning for linear models. In 2019 IEEE symposium on security and privacy (SP). IEEE, 724–738.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets