Papers
Topics
Authors
Recent
Search
2000 character limit reached

Differentially-Private Multi-Tier Federated Learning

Published 21 Jan 2024 in cs.LG, cs.CR, and cs.DC | (2401.11592v5)

Abstract: While federated learning (FL) eliminates the transmission of raw data over a network, it is still vulnerable to privacy breaches from the communicated model parameters. In this work, we propose Multi-Tier Federated Learning with Multi-Tier Differential Privacy (M2FDP), a DP-enhanced FL methodology for jointly optimizing privacy and performance in hierarchical networks. One of the key concepts of M2FDP is to extend the concept of HDP towards Multi-Tier Differential Privacy (MDP), while also adapting DP noise injection at different layers of an established FL hierarchy -- edge devices, edge servers, and cloud servers -- according to the trust models within particular subnetworks. We conduct a comprehensive analysis of the convergence behavior of M2FDP, revealing conditions on parameter tuning under which the training process converges sublinearly to a finite stationarity gap that depends on the network hierarchy, trust model, and target privacy level. Subsequent numerical evaluations demonstrate that M2FDP obtains substantial improvements in these metrics over baselines for different privacy budgets, and validate the impact of different system configurations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. B. Liu, M. Ding, S. Shaham, W. Rahayu, F. Farokhi, and Z. Lin, “When machine learning meets privacy: A survey and outlook,” ACM Comput. Surveys (CSUR), vol. 54, no. 2, 2021.
  2. M. Al-Rubaie and J. M. Chang, “Privacy-preserving machine learning: Threats and solutions,” IEEE Security Privacy, vol. 17, no. 2, pp. 49–58, 2019.
  3. E. De Cristofaro, “A critical overview of privacy in machine learning,” IEEE Security Privacy, vol. 19, no. 4, pp. 19–27, 2021.
  4. K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in Proc. ACM Conf. Comput. Commun. Security (SIGSAC), 2017, p. 1175–1191.
  5. Y. Yang, S. Huang, W. Huang, and X. Chang, “Privacy-preserving cost-sensitive learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 5, pp. 2105–2116, 2021.
  6. F. Farokhi, N. Wu, D. Smith, and M. A. Kaafar, “The cost of privacy in asynchronous differentially-private machine learning,” IEEE Trans. Inf. Forensics Security, vol. 16, pp. 2118–2129, 2021.
  7. Y.-W. Chu, S. Hosseinalipour, E. Tenorio, L. Cruz, K. A. Douglas, A. S. Lan, and C. G. Brinton, “Mitigating biases in student performance prediction via attention-based personalized federated learning,” Proc. ACM Int. Conf. Inf. Knowl. Manag., 2022.
  8. Z.-L. Chang, S. Hosseinalipour, M. Chiang, and C. G. Brinton, “Asynchronous multi-model federated learning over wireless networks: Theory, modeling, and optimization,” 2023.
  9. F. P.-C. Lin, S. Hosseinalipour, S. S. Azam, C. G. Brinton, and N. Michelusi, “Semi-decentralized federated learning with cooperative D2D local model aggregations,” IEEE J. Sel. Areas Commun., 2021.
  10. H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proc. Int. Conf. Artificial Intell. Stat. (AISTATS), 2017.
  11. F. Po-Chen Lin, S. Hosseinalipour, C. G. Brinton, and N. Michelusi, “Delay-aware hierarchical federated learning,” arXiv:2303.12414, 2023.
  12. S. Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan, “Adaptive federated learning in resource constrained edge computing systems,” IEEE J. Select. Areas Commun., vol. 37, no. 6, pp. 1205–1221, 2019.
  13. F. Haddadpour and M. Mahdavi, “On the convergence of local descent methods in federated learning,” arXiv:1910.14425, 2019.
  14. S. Hosseinalipour, S. Azam, C. Brinton, N. Michelusi, V. Aggarwal, D. Love, and H. Dai, “Multi-stage hybrid federated learning over large-scale D2D-enabled fog networks,” IEEE/ACM Trans. Netw., vol. 30, no. 4, pp. 1569–1584, 2022.
  15. W. Wei, L. Liu, M. L. Loper, K. H. Chow, M. E. Gursoy, S. Truex, and Y. Wu, “A framework for evaluating client privacy leakages in federated learning,” in Computer Security (ESORICS), 2020, pp. 545–566.
  16. B. Zhao, K. R. Mopuri, and H. Bilen, “idlg: Improved deep leakage from gradients,” arXiv:2001.02610, 2020.
  17. L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” in Proc. Adva. Neural Inf. Process. Syst., 2019.
  18. Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, “Beyond inferring class representatives: User-level privacy leakage from federated learning,” in Proc. IEEE Int. Conf. Comput. Comun. (INFOCOM), 2019, pp. 2512–2520.
  19. K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. S. Quek, and H. Vincent Poor, “Federated learning with differential privacy: Algorithms and performance analysis,” IEEE Trans. Inf. Forensics Security, vol. 15, pp. 3454–3469, 2020.
  20. X. Shen, Y. Liu, and Z. Zhang, “Performance-enhanced federated learning with differential privacy for internet of things,” IEEE Internet Things J., vol. 9, no. 23, pp. 24 079–24 094, 2022.
  21. Y. Zhao, J. Zhao, M. Yang, T. Wang, N. Wang, L. Lyu, D. Niyato, and K.-Y. Lam, “Local differential privacy-based federated learning for internet of things,” IEEE Internet Things J., vol. 8, no. 11, pp. 8836–8853, 2021.
  22. L. Shi, J. Shu, W. Zhang, and Y. Liu, “Hfl-dp: Hierarchical federated learning with differential privacy,” in Proc. IEEE Int. Glob. Commun. Conf. (GLOBECOM), 2021, pp. 1–7.
  23. V. Chandrasekaran, S. Banerjee, D. Perino, and N. Kourtellis, “Hierarchical federated learning with privacy,” arXiv:2206.05209, 2022.
  24. J. Wang, S. Guo, X. Xie, and H. Qi, “Protect privacy from gradient leakage attack in federated learning,” in IEEE Conf. Comput. Commun. (INFOCOM), 2022, pp. 580–589.
  25. P. Sun, X. Chen, G. Liao, and J. Huang, “A profit-maximizing model marketplace with differentially private federated learning,” in IEEE Conf. Comput. Commun. (INFOCOM), 2022, pp. 1439–1448.
  26. J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv:1610.05492, 2017.
  27. Z. Xiong, Z. Cai, D. Takabi, and W. Li, “Privacy threat and defense for federated learning with non-i.i.d. data in aiot,” IEEE Trans. Ind. Informat., vol. 18, no. 2, pp. 1310–1321, 2022.
  28. M. Naseri, J. Hayes, and E. D. Cristofaro, “Local and central differential privacy for robustness and privacy in federated learning,” arXiv:2009.03561, 2022.
  29. P. Liu, D. Willis, and S. Banerjee, “Paradrop: Enabling lightweight multi-tenancy at the network’s extreme edge,” in IEEE/ACM Symp. Edge Comput. (SEC), 2016, pp. 1–13.
  30. A. Wainakh, A. S. Guinea, T. Grube, and M. Mühlhäuser, “Enhancing privacy via hierarchical federated learning,” in Workshop IEEE European Symp. Security Privacy (EuroSPW), 2020, pp. 344–347.
  31. T. Zhou, “Hierarchical federated learning with gaussian differential privacy,” in Proc. Int. Conf. Advanced Inf. Sci. Syst., 2023.
  32. F. P.-C. Lin, C. G. Brinton, and N. Michelusi, “Federated learning with communication delay in edge networks,” in Proc. IEEE Int. Glob. Commun. Conf. (GLOBECOM), 2020, pp. 1–6.
  33. C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” vol. 9, no. 3-4, p. 211–407, 2014.
  34. L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in IEEE Symp. Security Privacy (SP), 2019, pp. 691–706.
  35. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proc. Conf. on Comput. Commun. Security (SIGSAC), vol. 24.   Association for Computing Machinery, 2016, p. 308–318.
  36. X. Zhang, X. Chen, M. Hong, S. Wu, and J. Yi, “Understanding clipping for federated learning: Convergence and client-level differential privacy,” in Proc. Machine Learn., vol. 162, 2022, pp. 26 048–26 067.
  37. S. J. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” in Proc. Int. Conf. Learn. Representations, 2021.
  38. L. Liu, J. Zhang, S. Song, and K. B. Letaief, “Client-edge-cloud hierarchical federated learning,” in Proc. IEEE Int. Conf. Commun. (ICC), 2020, pp. 1–6.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 1 like about this paper.

Sign Up for Free