Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Privacy Power of Correlated Noise in Decentralized Learning (2405.01031v2)

Published 2 May 2024 in cs.LG, cs.CR, cs.DC, math.OC, and stat.ML

Abstract: Decentralized learning is appealing as it enables the scalable usage of large amounts of distributed data and resources (without resorting to any central entity), while promoting privacy since every user minimizes the direct exposure of their data. Yet, without additional precautions, curious users can still leverage models obtained from their peers to violate privacy. In this paper, we propose Decor, a variant of decentralized SGD with differential privacy (DP) guarantees. Essentially, in Decor, users securely exchange randomness seeds in one communication round to generate pairwise-canceling correlated Gaussian noises, which are injected to protect local models at every communication round. We theoretically and empirically show that, for arbitrary connected graphs, Decor matches the central DP optimal privacy-utility trade-off. We do so under SecLDP, our new relaxation of local DP, which protects all user communications against an external eavesdropper and curious users, assuming that every pair of connected users shares a secret, i.e., an information hidden to all others. The main theoretical challenge is to control the accumulation of non-canceling correlated noise due to network sparsity. We also propose a companion SecLDP privacy accountant for public use.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.  308–318, 2016.
  2. Information-theoretic lower bounds on the oracle complexity of convex optimization. Advances in Neural Information Processing Systems, 22, 2009.
  3. cpsgd: Communication-efficient and differentially-private distributed sgd. Advances in Neural Information Processing Systems, 31, 2018.
  4. The skellam mechanism for differentially private federated learning. Advances in Neural Information Processing Systems, 34:5052–5064, 2021.
  5. On the privacy-robustness-utility trilemma in distributed learning. In International Conference on Machine Learning, 2023.
  6. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199(1-2):165–214, 2023.
  7. Faster rates of convergence to stationary points in differentially private optimization. arXiv preprint arXiv:2206.00846, 2022.
  8. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pp.  464–473. IEEE, 2014.
  9. Personalized and private peer-to-peer machine learning. In International Conference on Artificial Intelligence and Statistics, pp.  473–481. PMLR, 2018.
  10. Practical secure aggregation for privacy-preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp.  1175–1191, 2017.
  11. Optimization methods for large-scale machine learning. Siam Review, 60(2):223–311, 2018.
  12. Randomized gossip algorithms. IEEE transactions on information theory, 52(6):2508–2530, 2006.
  13. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1–27, 2011.
  14. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011.
  15. Towards decentralized deep learning with differential privacy. In International Conference on Cloud Computing, pp.  130–145. Springer, 2019.
  16. Distributed differential privacy via shuffling. In Advances in Cryptology–EUROCRYPT 2019: 38th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Darmstadt, Germany, May 19–23, 2019, Proceedings, Part I 38, pp.  375–403. Springer, 2019.
  17. Privacy amplification by decentralization. In International Conference on Artificial Intelligence and Statistics, pp.  5334–5353. PMLR, 2022.
  18. Muffliato: Peer-to-peer privacy amplification for decentralized optimization and averaging. Advances in Neural Information Processing Systems, 35:15889–15902, 2022.
  19. De Abreu, N. M. M. Old and new results on algebraic connectivity of graphs. Linear algebra and its applications, 423(1):53–73, 2007.
  20. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pp.  429–438. IEEE, 2013.
  21. Minimax optimal procedures for locally private estimation. Journal of the American Statistical Association, 113(521):182–201, 2018.
  22. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  23. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, pp.  2468–2479. SIAM, 2019.
  24. Fiedler, M. Algebraic connectivity of graphs. Czechoslovak mathematical journal, 23(2):298–305, 1973.
  25. Rényi divergence measures for commonly used univariate continuous distributions. Information Sciences, 249:124–131, 2013.
  26. Dp-admm: Admm-based distributed learning with differential privacy. IEEE Transactions on Information Forensics and Security, 15:1002–1012, 2019.
  27. Distributed differentially private computation of functions with correlated noise. arXiv preprint arXiv:1904.10059, 2019.
  28. Distributed learning without distress: Privacy-preserving empirical risk minimization. Advances in Neural Information Processing Systems, 31, 2018.
  29. The distributed discrete gaussian mechanism for federated learning with secure aggregation. In International Conference on Machine Learning, pp.  5201–5212. PMLR, 2021a.
  30. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning, pp.  5213–5225. PMLR, 2021b.
  31. Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition. In Joint European conference on machine learning and knowledge discovery in databases, pp.  795–811. Springer, 2016.
  32. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp.  5132–5143. PMLR, 2020.
  33. What can we learn privately? SIAM Journal on Computing, 40(3):793–826, 2011.
  34. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pp.  5381–5393. PMLR, 2020.
  35. MNIST handwritten digit database. 2010.
  36. Convergence and privacy of decentralized nonconvex optimization with gradient clipping and communication compression. arXiv preprint arXiv:2305.09896, 2023.
  37. Privacy-preserving distributed average consensus based on additive secret sharing. In 2019 27th European Signal Processing Conference (EUSIPCO), pp.  1–5. IEEE, 2019.
  38. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
  39. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  40. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP), pp.  691–706. IEEE, 2019.
  41. Mironov, I. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp.  263–275. IEEE, 2017.
  42. Differentially private federated learning on heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pp.  10110–10145. PMLR, 2022.
  43. An accurate, scalable and verifiable protocol for federated differentially private averaging. Machine Learning, 111(11):4249–4293, 2022.
  44. Shamir, A. How to share a secret. Communications of the ACM, 22(11):612–613, 1979.
  45. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 10(1):1–12, 2020.
  46. Fast and faster convergence of sgd for over-parameterized models and an accelerated perceptron. In The 22nd international conference on artificial intelligence and statistics, pp.  1195–1204. PMLR, 2019.
  47. Vishnoi, N. K. Laplacian solvers and their algorithmic applications. Theoretical Computer Science, 8(1-2):1–141, 2012.
  48. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.  1226–1235. PMLR, 2019.
  49. Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems, 34:13975–13987, 2021.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now