Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Differentially Private Distributed Nonconvex Stochastic Optimization with Quantized Communication (2403.18254v2)

Published 27 Mar 2024 in eess.SY and cs.SY

Abstract: This paper proposes a new distributed nonconvex stochastic optimization algorithm that can achieve privacy protection, communication efficiency and convergence simultaneously. Specifically, each node adds general privacy noises to its local state to avoid information leakage, and then quantizes its noise-perturbed state before transmitting to improve communication efficiency. By using a subsampling method controlled through the sample-size parameter, the proposed algorithm reduces cumulative differential privacy parameters $\epsilon$, $\delta$, and thus enhances the differential privacy level, which is significantly different from the existing works. By using a two-time-scale step-sizes method, the mean square convergence for nonconvex cost functions is given. Furthermore, when the global cost function satisfies the Polyak-{\L}ojasiewicz condition, the convergence rate and the oracle complexity of the proposed algorithm are given. In addition, the proposed algorithm achieves both the mean square convergence and finite cumulative differential privacy parameters $\epsilon$, $\delta$ over infinite iterations as the sample-size goes to infinity. A numerical example of the distributed training on the ``MNIST'' dataset is given to show the effectiveness of the algorithm.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. M. Zhu and S. Martínez, “On distributed convex optimization under inequality and equality constraints,” IEEE Trans. Autom. Control, vol. 57, no. 1, pp. 151–164, 2011.
  2. M. Zhu and S. Martínez, “An approximate dual subgradient algorithm for multi-agent non-convex optimization,” IEEE Trans. Autom. Control, vol. 58, no. 6, pp. 1534–1539, 2013.
  3. T. T. Doan, S. T. Maguluri, and J. Romberg, “Fast convergence rates of distributed subgradient methods with adaptive quantization,” IEEE Trans. Autom. Control, vol. 66, no. 5, pp. 2191–2205, 2021.
  4. T. T. Doan, S. T. Maguluri, and J. Romberg, “Convergence rates of distributed gradient methods under random quantization: a stochastic approximation approach,” IEEE Trans. Autom. Control, vol. 66, no. 10, pp. 4469–4484, 2021.
  5. R. Xin, U. A. Khan, and S. Kar, “A fast randomized incremental gradient method for decentralized nonconvex optimization,” IEEE Trans. Autom. Control, vol. 67, no. 10, pp. 5150–5165, 2022.
  6. Z. Jiang, A. Balu, C. Hegde, and S. Sarkar, “Collaborative deep learning in fixed topology networks,” in Proc. Adv. Neural Inf. Process. Syst., Long Beach, CA, USA, 2017, vol. 30, pp. 5904–5914.
  7. K. Lu, H. Wang, H. Zhang, and L. Wang, “Convergence in high probability of distributed stochastic gradient descent algorithms,” IEEE Trans. Autom. Control, doi: 10.1109/TAC.2023.3327319, 2024.
  8. A. Reisizadeh, H. Taheri, A. Mokhtari, H. Hassani, and R. Pedarsani, “Robust and communication-efficient collaborative learning,” in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2019, vol. 32, pp. 8388–8399.
  9. Z. Zhang, Y. Zhang, D. Guo, S. Zhao, and X. Zhu, “Communication-efficient federated continual learning for distributed learning system with non-iid data,” Sci. China Inf. Sci., vol. 66, no. 2, 2023, Art. no. 122102.
  10. K. Ge, Y. Zhang, Y. Fu, Z. Lai, X. Deng, and D. Li, “Accelerate distributed deep learning with cluster-aware sketch quantization,” Sci. China Inf. Sci., vol. 66, no. 6, 2023, Art. no. 162102.
  11. J. Lei, P. Yi, J. Chen, and Y. Hong, “Distributed variable sample-size stochastic optimization with fixed step-sizes,” IEEE Trans. Autom. Control, vol. 67, no. 10, pp. 5630–5637, 2022.
  12. J. F. Zhang, J. W. Tan, and J. Wang, “Privacy security in control systems,” Sci. China Inf. Sci., vol. 64, no. 7, 2021, Art. no. 176201.
  13. Y. Lu and M. Zhu, “Privacy preserving distributed optimization using homomorphic encryption,” Automatica, vol. 96, pp. 314–325, 2018.
  14. Y. L. Mo and R. M. Murray, “Privacy preserving average consensus,” IEEE Trans. Autom. Control, vol. 62, no. 2, pp. 753–765, 2017.
  15. Y. Lou, L. Yu, S. Wang, and P. Yi, “Privacy preservation in distributed subgradient optimization algorithms,” IEEE Trans. Cybern., vol. 48, no. 7, pp. 2154–2165, 2018.
  16. Y. Wang, “Privacy-preserving average consensus via state decomposition,” IEEE Trans. Autom. Control, vol. 64, no. 11, pp. 4711–4716, 2019.
  17. Y. Lu and M. Zhu, “On privacy preserving data release of linear dynamic networks,” Automatica, vol. 115, 2020, Art. no. 108839.
  18. J. Le Ny and G. J. Pappas, “Differentially private filtering,” IEEE Trans. Autom. Control, vol. 59, no. 2, pp. 341–354, 2014.
  19. C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” Found. Trends Theor. Comput. Sci., vol. 9, nos. 3–4, pp. 211–407, 2014.
  20. X. K. Liu, J. F. Zhang, and J. Wang, “Differentially private consensus algorithm for continuous-time heterogeneous multi-agent systems,” Automatica, vol. 122, 2020, Art. no. 109283.
  21. J. Wang, J. F. Zhang, and X. He, “Differentially private distributed algorithms for stochastic aggregative games,” Automatica, vol. 142, 2022, Art. no. 110440.
  22. X. Chen, C. Wang, Q. Yang, T. Hu, and C. Jiang, “Locally differentially private high-dimensional data synthesis,” Sci. China Inf. Sci., vol. 66, no. 1, 2023, Art. no. 112101.
  23. J. Wang, J. Ke, and J. F. Zhang, “Differentially private bipartite consensus over signed networks with time-varying noises,” IEEE Trans. Autom. Control, doi: 10.1109/TAC.2024.3351869, 2024.
  24. C. Li, P. Zhou, L. Xiong, Q. Wang, and T. Wang, “Differentially private distributed online learning,” IEEE Trans. Knowl. Data Eng., vol. 30, no. 8, pp. 1440–1453, 2018.
  25. Z. Huang, R. Hu, Y. Guo, E. Chan-Tin, and Y. Gong, “DP-ADMM: ADMM-based distributed learning with differential privacy,” IEEE Trans. Inf. Forensics Secur., vol. 15, pp. 1002–1012, 2020.
  26. J. Ding, G. Liang, J. Bi, and M. Pan, “Differentially private and communication efficient collaborative learning,“ in Proc. AAAI Conf. Artif. Intell., Palo Alto, CA, USA, 2021, vol. 35, no. 8, pp. 7219–7227.
  27. C. Gratton, N. K. D. Venkategowda, R. Arablouei, and S. Werner, “Privacy-preserved distributed learning with zeroth-order optimization,” IEEE Trans. Inf. Forensics Secur., vol. 17, pp. 265–279, 2022.
  28. C. Liu, K. H. Johansson, and Y. Shi, “Distributed empirical risk minimization with differential privacy,” Automatica, vol. 162, 2024, Art. no. 111514.
  29. J. Xu, W. Zhang, and F. Wang, “A (DP) 2 SGD: asynchronous decentralized parallel stochastic gradient descent with differential privacy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 8036–8047, 2022.
  30. Y. Wang and T. Başar, “Decentralized nonconvex optimization with guaranteed privacy and accuracy,” Automatica, vol. 150, 2023, Art. no. 110858.
  31. Y. Wang and T. Başar, “Quantization enabled privacy protection in decentralized stochastic optimization,” IEEE Trans. Autom. Control, vol. 68, no. 7, pp. 4038–4052, 2023.
  32. G. Yan, T. Li, K. Wu, and L. Song, “Killing two birds with one stone: quantization achieves privacy in distributed learning,” Digit. Signal Process., vol. 146, 2024, Art. no. 104353.
  33. A. Xie, X. Yi, X. Wang, M. Cao, and X. Ren, “Compressed differentially private distributed optimization with linear convergence,” IFAC-PapersOnLine, vol. 56, no. 2, pp. 8369–8374, 2023.
  34. H. Karimi, J. Nutini, and M. Schmidt, “Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition,” in Proc. Mach. Learn. Knowl. Discov. Databases Euro. Conf., Riva del Garda, Italy, 2016, pp. 795–811.
  35. T. C. Aysal, M. J. Coates, and M. G. Rabbat, “Distributed average consensus with dithered quantization,” IEEE Trans. Signal Process., vol. 56, no. 10, pp. 4905–4918, 2008.
  36. L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2019, vol. 32, pp. 14774–14784.
  37. S. Bubeck, “Convex optimization: algorithms and complexity,” Found. Trends Theor. Comput. Sci., vol. 8, nos. 3-4, pp. 231–357, 2015.
  38. V. A. Zorich, “Integration,” in Mathematical analysis I, Berlin, German: Springer-Verlag, 2015, ch. 6, sec. 2, pp. 349–360.
  39. R. A. Horn and C. R. Johnson, “Hermitian matrices, symmetric matrices, and congruences,” in Matrix analysis, Cambridge, U.K.: Cambridge University Press, 2012, ch. 4, sec. 2, pp. 234–239.
  40. Y. S. Chow and H. Teicher, “Integration in a probability space,”in Probability theory: independence, interchangeability, martingales, New York, NY, USA: Springer-Verlag, 2012, ch. 4, sec. 1, pp. 84–92.
  41. Y. LeCun, C. Cortes, and C. J. C. Burges, 1998, “The MNIST database of handwritten digits,” National Institute of Standards and Technology. [Online]. Available: http://yann.lecun.com/exdb/mnist/

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now