Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Novel Privacy Enhancement Scheme with Dynamic Quantization for Federated Learning

Published 25 May 2024 in math.OC | (2405.16058v4)

Abstract: Federated learning (FL) has been widely regarded as a promising paradigm for privacy preservation of raw data in machine learning. Although, the data privacy in FL is locally protected to some extent, it is still a desideratum to enhance privacy and alleviate communication overhead caused by repetitively transmitting model parameters. Typically, these challenges are addressed separately, or jointly via a unified scheme that consists of noise-injected privacy mechanism and communication compression, which may lead to model corruption due to the introduced composite noise. In this work, we propose a novel model-splitting privacy-preserving FL (MSP-FL) scheme to achieve private FL with precise accuracy guarantee. Based upon MSP-FL, we further propose a model-splitting privacy-preserving FL with dynamic quantization (MSPDQ-FL) to mitigate the communication overhead, which incorporates a shrinking quantization interval to reduce the quantization error. We provide privacy and convergence analysis for both MSP-FL and MSPDQ-FL under non-i.i.d. dataset, partial clients participation and finite quantization level. Numerical results are presented to validate the superiority of the proposed schemes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Mag., vol. 37, no. 3, pp. 50–60, 2020.
  2. T. Gafni, N. Shlezinger, K. Cohen, Y. C. Eldar, and H. V. Poor, “Federated learning: A signal processing perspective,” IEEE Signal Processing Mag., vol. 39, no. 3, pp. 14–41, 2022.
  3. L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” in Proc. Adv. Neural Inf. Process. Syst., 2019, vol. 32.
  4. J. Zhao, H. Zhu, F. Wang, R. Lu, Z. Liu, and H. Li, “PVD-FL: A privacy-preserving and verifiable decentralized federated learning framework,” IEEE Trans. Inf. Forensics Secur., vol. 17, pp. 2059–2073, 2022.
  5. C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun, “SplitFed: When federated learning meets split learning,” in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 8, 2022, pp. 8485–8493.
  6. C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” Foundations and Trends® in Theoretical Computer Science, vol. 9, no. 3–4, pp. 211–407, 2014.
  7. S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and W. Wei, “LDP-Fed: Federated learning with local differential privacy,” in Proc. 3rd ACM Int. Workshop Edge Syst., Anal. Netw., 2020, pp. 61–66.
  8. K. Wei, J. Li, C. Ma, M. Ding, C. Chen, S. Jin, Z. Han, and H. V. Poor, “Low-latency federated learning over wireless channels with differential privacy,” IEEE J. Sel. Areas Commun., vol. 40, no. 1, pp. 290–307, 2022.
  9. K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. S. Quek, and H. V. Poor, “Federated learning with differential privacy: Algorithms and performance analysis,” IEEE Trans. Inf. Forensics Secur., vol. 15, pp. 3454–3469, 2020.
  10. X. Yuan, W. Ni, M. Ding, K. Wei, J. Li, and H. V. Poor, “Amplitude-varying perturbation for balancing privacy and utility in federated learning,” IEEE Trans. Inf. Forensics Secur., vol. 18, pp. 1884–1897, 2023.
  11. Y. Mo and R. M. Murray, “Privacy preserving average consensus,” IEEE Trans. Autom. Control, vol. 62, no. 2, pp. 753–765, 2017.
  12. Y. Wang, “Privacy-preserving average consensus via state decomposition,” IEEE Trans. Autom. Control, vol. 64, no. 11, pp. 4711–4716, 2019.
  13. F. Ye, Z. Cheng, X. Cao, and M. Chow, “A random-weight privacy-preserving algorithm with error compensation for microgrid distributed energy management,” IEEE Trans. Inf. Forensics Secur., vol. 16, pp. 4352–4362, 2021.
  14. N. Shlezinger, M. Chen, Y. C. Eldar, H. V. Poor, and S. Cui, “UVeQFed: Universal vector quantization for federated learning,” IEEE Trans. Signal Process., vol. 69, pp. 500–514, 2021.
  15. Y. Wang, L. Lin, and J. Chen, “Communication-efficient adaptive federated learning,” in Proc. ICML, vol. 162, 2022, pp. 22802–22838.
  16. S. Wang, M. Chen, C. G. Brinton, C. Yin, W. Saad, and S. Cui, “Performance optimization for variable bitwidth federated learning in wireless networks,” arXiv preprint arXiv:2209.10200, 2022.
  17. E. Rizk, S. Vlaski, and A. H. Sayed, “Federated learning under importance sampling,” IEEE Trans. Signal Process., vol. 70, pp. 5381–5396, 2022.
  18. X. He, X. Yi, Y. Zhao, K. H. Johansson, and V. Gupta, “Asymptotic analysis of federated learning under event-triggered communication,” IEEE Trans. Signal Process., vol. 71, pp. 2654–2667, 2023.
  19. A. Mahmoudi, J. H. da Cruz J’unior, H. S. Ghadikolaei, and C. Fischione, “A-LAQ: Adaptive lazily aggregated quantized gradient,” 2022 IEEE Globecom Workshops (GC Wkshps), pp. 1828–1833, 2022.
  20. W. Liu, L. Chen, Y. fei Chen, and W. Wang, “Communication-efficient design for quantized decentralized federated learning,” ArXiv, vol. abs/2303.08423, 2023.
  21. Y. Feng, T. Xiong, R. Wu, L. Lv, and L. Shi, “sqSGD: Locally private and communication efficient federated learning,” arXiv preprint arXiv:2206.10565, 2022.
  22. A. Girgis, D. Data, S. Diggavi, P. Kairouz, and A. Theertha Suresh, “Shuffled model of differential privacy in federated learning,” in Proc. Int. Conf. Artif. Int. Stat., vol. 130, 2021, pp. 2521–2529.
  23. Z. Li, H. Zhao, B. Li, and Y. Chi, “SoteriaFL: A unified framework for private federated learning with communication compression,” in Proc. Adv. Neural Inf. Process. Syst., vol. 35, 2022, pp. 4285–4300.
  24. N. Lang, E. Sofer, T. Shaked, and N. Shlezinger, “Joint privacy enhancement and quantization in federated learning,” IEEE Trans. Signal Process., vol. 71, pp. 295–310, 2023.
  25. X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” in Proc. ICLR, 2020.
  26. S. Zheng, C. Shen, and X. Chen, “Design and analysis of uplink and downlink communications for federated learning,” IEEE J. Sel. Areas Commun., vol. 39, no. 7, pp. 2150–2167, 2021.
  27. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proc. Int. Conf. Artif. Int. Stat., vol. 54, 2017, pp. 1273–1282.
  28. T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Proc. MLSys Conf., vol. 2, 2020, pp. 429–450.
  29. W. Fang, Z. Yu, Y. Jiang, Y. Shi, C. N. Jones, and Y. Zhou, “Communication-efficient stochastic zeroth-order optimization for federated learning,” IEEE Trans. Signal Process., vol. 70, pp. 5058–5073, 2022.
  30. X. Chen, L. Huang, K. Ding, S. Dey, and L. Shi, “Privacy-preserving push-sum average consensus via state decomposition,” IEEE Trans. Autom. Control, vol. 68, no. 12, pp. 7974–7981, 2023.
  31. T. T. Doan, S. T. Maguluri, and J. Romberg, “Fast convergence rates of distributed subgradient methods with adaptive quantization,” IEEE Trans. Autom. Control, vol. 66, no. 5, pp. 2191–2205, 2021.
  32. Y. Wang, Y. Xu, Q. Shi, and T. H. Chang, “Quantized federated learning under transmission delay and outage constraints” IEEE J. Sel. Areas Commun., vol. 40, no. 1, pp. 323–341, 2022.
  33. V. S. Mai and E. H. Abed, “Distributed optimization over directed graphs with row stochasticity and constraint regularity,” Automatica, vol. 102, pp. 94–104, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free