Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learning (2308.14606v4)

Published 28 Aug 2023 in cs.LG, cs.CR, and cs.DC

Abstract: This paper jointly considers privacy preservation and Byzantine-robustness in decentralized learning. In a decentralized network, honest-but-curious agents faithfully follow the prescribed algorithm, but expect to infer their neighbors' private data from messages received during the learning process, while dishonest-and-Byzantine agents disobey the prescribed algorithm, and deliberately disseminate wrong messages to their neighbors so as to bias the learning process. For this novel setting, we investigate a generic privacy-preserving and Byzantine-robust decentralized stochastic gradient descent (SGD) framework, in which Gaussian noise is injected to preserve privacy and robust aggregation rules are adopted to counteract Byzantine attacks. We analyze its learning error and privacy guarantee, discovering an essential tradeoff between privacy preservation and Byzantine-robustness in decentralized learning -- the learning error caused by defending against Byzantine attacks is exacerbated by the Gaussian noise added to preserve privacy. For a class of state-of-the-art robust aggregation rules, we give unified analysis of the "mixing abilities". Building upon this analysis, we reveal how the "mixing abilities" affect the tradeoff between privacy preservation and Byzantine-robustness. The theoretical results provide guidelines for achieving a favorable tradeoff with proper design of robust aggregation rules. Numerical experiments are conducted and corroborate our theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu, “Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent,” Advances in Neural Information Processing Systems, 2017.
  2. H. Ye, L. Liang, and G. Y. Li, “Decentralized federated learning with unreliable communications,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 3, pp. 487–500, 2022.
  3. J. Ding, G. Liang, J. Bi, and M. Pan, “Differentially private and communication efficient collaborative learning,” AAAI Conference on Artificial Intelligence, 2021.
  4. L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” Advances in Neural Information Processing Systems, 2019.
  5. Y. Wang and A. Nedić, “Tailoring gradient methods for differentially-private distributed optimization,” IEEE Transactions on Automatic Control, 2023.
  6. Z. Huang, R. Hu, Y. Guo, E. Chan-Tin, and Y. Gong, “DP-ADMM: ADMM-based distributed learning with differential privacy,” IEEE Transactions on Information Forensics and Security, vol. 15, pp. 1002–1012, 2019.
  7. Z. Huang and Y. Gong, “Differentially private ADMM for convex distributed learning: Improved accuracy via multi-step approximation,” arXiv preprint arXiv:2005.07890, 2020.
  8. X. Zhang, M. M. Khalili, and M. Liu, “Recycled ADMM: Improving the privacy and accuracy of distributed algorithms,” IEEE Transactions on Information Forensics and Security, vol. 15, pp. 1723–1734, 2020.
  9. C. Gratton, N. K. D. Venkategowda, R. Arablouei, and S. Werner, “Privacy-preserved distributed learning with zeroth-order optimization,” IEEE Transactions on Information Forensics and Security, vol. 17, pp. 265–279, 2022.
  10. L. Lamport, R. Shostak, and M. Pease, “The Byzantine generals problem,” ACM Transactions on Programming Languages and Systems, vol. 4, no. 3, pp. 382–401, 1982.
  11. Z. Yang and W. U. Bajwa, “Byrdie: Byzantine-resilient distributed coordinate descent for decentralized learning,” IEEE Transactions on Signal and Information Processing over Networks, vol. 5, no. 4, pp. 611–627, 2019.
  12. C. Fang, Z. Yang, and W. U. Bajwa, “Bridge: Byzantine-resilient decentralized gradient descent,” IEEE Transactions on Signal and Information Processing over Networks, vol. 8, pp. 610–626, 2022.
  13. J. Peng, W. Li, and Q. Ling, “Byzantine-robust decentralized stochastic optimization over static and time-varying networks,” Signal Processing, vol. 183, p. 108020, 2021.
  14. L. He, S. P. Karimireddy, and M. Jaggi, “Byzantine-robust decentralized learning via self-centered clipping,” arXiv preprint arXiv:2202.01545, 2022.
  15. Z. Wu, T. Chen, and Q. Ling, “Byzantine-resilient decentralized stochastic optimization with robust aggregation rules,” IEEE Transactions on Signal Processing, vol. 71, pp. 3179–3195, 2023.
  16. M. Hao, H. Li, G. Xu, H. Chen, and T. Zhang, “Efficient, private and robust federated learning,” Annual Computer Security Applications Conference, 2021.
  17. Y. Miao, Z. Liu, H. Li, K.-K. R. Choo, and R. H. Deng, “Privacy-preserving Byzantine-robust federated learning via blockchain systems,” IEEE Transactions on Information Forensics and Security, vol. 17, pp. 2848–2861, 2022.
  18. Z. Ma, J. Ma, Y. Miao, Y. Li, and R. H. Deng, “Shieldfl: Mitigating model poisoning attacks in privacy-preserving federated learning,” IEEE Transactions on Information Forensics and Security, vol. 17, pp. 1639–1654, 2022.
  19. X. Chen, H. Yu, X. Jia, and X. Yu, “Apfed: Anti-poisoning attacks in privacy-preserving heterogeneous federated learning,” IEEE Transactions on Information Forensics and Security, vol. 18, pp. 5749–5761, 2023.
  20. G. Hu, H. Li, W. Fan, and Y. Zhang, “Efficient Byzantine-robust and privacy-preserving federated learning on compressive domain,” IEEE Internet of Things Journal, 2023.
  21. W. Li, K. Fan, K. Yang, Y. Yang, and H. Li, “Pbfl: Privacy-preserving and Byzantine-robust federated learning empowered industry 4.0,” IEEE Internet of Things Journal, 2023.
  22. A. Choffrut, R. Guerraoui, R. Pinot, R. Sirdey, J. Stephan, and M. Zuber, “Practical homomorphic aggregation for Byzantine ML,” arXiv preprint arXiv:2309.05395, 2023.
  23. H. Zhu and Q. Ling, “Bridging differential privacy and Byzantine-robustness via model aggregation,” International Joint Conference on Artificial Intelligence, 2022.
  24. X. Ma, X. Sun, Y. Wu, Z. Liu, X. Chen, and C. Dong, “Differentially private Byzantine-robust federated learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 12, pp. 3690–3701, 2022.
  25. R. Guerraoui, N. Gupta, R. Pinot, S. Rouault, and J. Stephan, “Differential privacy and Byzantine resilience in SGD: Do they add up?” ACM Symposium on Principles of Distributed Computing, no. 11, pp. 391–401, 2021.
  26. Y. Allouah, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan, “On the privacy-robustness-utility trilemma in distributed learning,” International Conference on Machine Learning, 2023.
  27. J. Le, D. Zhang, X. Lei, L. Jiao, K. Zeng, and X. Liao, “Privacy-preserving federated learning with malicious clients and honest-but-curious servers,” IEEE Transactions on Information Forensics and Security, vol. 18, pp. 4329–4344, 2023.
  28. Z. Zhang and R. Hu, “Byzantine-robust federated learning with variance reduction and differential privacy,” arXiv preprint arXiv:2309.03437, 2023.
  29. Y. Wang and A. Nedić, “Decentralized gradient methods with time-varying uncoordinated stepsizes: Convergence analysis and privacy design,” arXiv preprint arXiv:2205.10934, 2022.
  30. Y. Wang and H. V. Poor, “Decentralized stochastic optimization with inherent privacy protection,” IEEE Transactions on Automatic Control, vol. 68, no. 4, pp. 2293–2308, 2023.
  31. H. Gao, Y. Wang, and A. Nedić, “Dynamics based privacy preservation in decentralized optimization,” Automatica, vol. 151, p. 110878, 2023.
  32. Y. Lou, L. Yu, S. Wang, and P. Yi, “Privacy preservation in distributed subgradient optimization algorithms,” IEEE Transactions on Cybernetics, vol. 48, no. 7, pp. 2154–2165, 2017.
  33. C. Zhang, M. Ahmad, and Y. Wang, “ADMM based privacy-preserving decentralized optimization,” IEEE Transactions on Information Forensics and Security, vol. 14, no. 3, pp. 565–580, 2018.
  34. J. Zhao, H. Zhu, F. Wang, R. Lu, Z. Liu, and H. Li, “PVD-FL: A privacy-preserving and verifiable decentralized federated learning framework,” IEEE Transactions on Information Forensics and Security, vol. 17, pp. 2059–2073, 2022.
  35. Y. Lu, Z. Yu, and N. Suri, “Privacy-preserving decentralized federated learning over time-varying communication graph,” ACM Transactions on Privacy and Security, vol. 26, no. 3, pp. 1–39, 2023.
  36. J. So, B. Guler, and S. Avestimehr, “A scalable approach for privacy-preserving collaborative machine learning,” Advances in Neural Information Processing Systems, 2020.
  37. D. Yin, Y. Chen, R. Kannan, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” International Conference on Machine Learning, 2018.
  38. Y. Chen, L. Su, and J. Xu, “Distributed statistical machine learning in adversarial settings: Byzantine gradient descent,” Proceedings of the ACM on Measurement and Analysis of Computing Systems, vol. 1, no. 2, pp. 1–25, 2017.
  39. P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” Advances in Neural Information Processing Systems, 2017.
  40. J. Yan, X. Li, Y. Mo, and C. Wen, “Resilient multi-dimensional consensus in adversarial environment,” Automatica, vol. 145, p. 110530, 2022.
  41. K. Kuwaranancharoen, L. Xin, and S. Sundaram, “Byzantine-resilient distributed optimization of multi-dimensional functions,” American Control Conference, pp. 4399–4404, 2020.
  42. S. Sundaram and B. Gharesifard, “Distributed optimization under adversarial nodes,” IEEE Transactions on Automatic Control, vol. 64, no. 3, pp. 1063–1076, 2019.
  43. L. Su and N. H. Vaidya, “Byzantine-resilient multiagent optimization,” IEEE Transactions on Automatic Control, vol. 66, no. 5, pp. 2227–2233, 2021.
  44. J. Xu, W. Zhang, and F. Wang, “A⁢(D⁢P)2⁢S⁢G⁢D𝐴superscript𝐷𝑃2𝑆𝐺𝐷A(DP)^{2}SGDitalic_A ( italic_D italic_P ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_S italic_G italic_D: Asynchronous decentralized parallel stochastic gradient descent with differential privacy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8036–8047, 2022.
  45. Y. Wang, B. Balle, and S. P. Kasiviswanathan, “Subsampled Rényi differential privacy and analytical moments accountant,” International Conference on Artificial Intelligence and Statistics, 2019.
  46. D. Cynthia, “Differential privacy,” International Colloquium on Automata, Languages, and Programming, 2006.
  47. I. Mironov, “Rényi differential privacy,” Computer Security Foundations Symposium, pp. 263–275, 2017.
  48. S. Farhadkhani, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan, “Byzantine machine learning made easy by resilient averaging of momentums,” International Conference on Machine Learning, 2022.
  49. Z. Wu, H. Shen, T. Chen, and Q. Ling, “Byzantine-resilient decentralized policy evaluation with linear function approximation,” IEEE Transactions on Signal Processing, vol. 69, pp. 3839–3853, 2021.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now