Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Faster Algorithms for User-Level Private Stochastic Convex Optimization (2410.18391v1)

Published 24 Oct 2024 in cs.LG, cs.CR, and math.OC

Abstract: We study private stochastic convex optimization (SCO) under user-level differential privacy (DP) constraints. In this setting, there are $n$ users (e.g., cell phones), each possessing $m$ data items (e.g., text messages), and we need to protect the privacy of each user's entire collection of data items. Existing algorithms for user-level DP SCO are impractical in many large-scale machine learning scenarios because: (i) they make restrictive assumptions on the smoothness parameter of the loss function and require the number of users to grow polynomially with the dimension of the parameter space; or (ii) they are prohibitively slow, requiring at least $(mn){3/2}$ gradient computations for smooth losses and $(mn)3$ computations for non-smooth losses. To address these limitations, we provide novel user-level DP algorithms with state-of-the-art excess risk and runtime guarantees, without stringent assumptions. First, we develop a linear-time algorithm with state-of-the-art excess risk (for a non-trivial linear-time algorithm) under a mild smoothness assumption. Our second algorithm applies to arbitrary smooth losses and achieves optimal excess risk in $\approx (mn){9/8}$ gradient computations. Third, for non-smooth loss functions, we obtain optimal excess risk in $n{11/8} m{5/4}$ gradient computations. Moreover, our algorithms do not require the number of users to grow polynomially with the dimension.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Private stochastic convex optimization: Optimal rates in ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT geometry. In ICML, 2021.
  2. User-level differentially private stochastic convex optimization: Efficient algorithms with optimal rates. In International Conference on Artificial Intelligence and Statistics, pages 4240–4248. PMLR, 2024.
  3. Private stochastic convex optimization with heavy tails: Near-optimality from simple reductions. arXiv preprint arXiv:2406.02789, 2024.
  4. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  5. Private stochastic convex optimization with optimal rates. In Advances in Neural Information Processing Systems, volume 32, 2019.
  6. User-level private stochastic convex optimization with optimal rates. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1838–1851. PMLR, 23–29 Jul 2023.
  7. Extracting training data from large language models. In USENIX Security Symposium, volume 6, pages 2633–2650, 2021.
  8. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization, 22(2):674–701, 2012.
  9. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer, 2006.
  10. The Algorithmic Foundations of Differential Privacy, volume 9. Now Publishers, Inc., 2014.
  11. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 439–449, 2020.
  12. On user-level private convex optimization. In International Conference on Machine Learning, pages 11283–11299. PMLR, 2023.
  13. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework. SIAM Journal on Optimization, 22(4):1469–1492, 2012.
  14. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, ii: Shrinking procedures and optimal algorithms. SIAM Journal on Optimization, 23(4):2061–2089, 2013.
  15. Private heterogeneous federated learning without a trusted server revisited: Error-optimal and communication-efficient algorithms for convex losses. arXiv preprint arXiv:2407.09690, 2024.
  16. Train faster, generalize better: Stability of stochastic gradient descent. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1225–1234, New York, New York, USA, 20–22 Jun 2016. PMLR.
  17. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint arXiv:1902.03736, 2019.
  18. Private non-smooth erm and sco in subquadratic steps. Advances in Neural Information Processing Systems, 34:4053–4064, 2021.
  19. The composition theorem for differential privacy, 2015.
  20. Non-convex finite-sum optimization via scsg methods. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 2345–2355, 2017.
  21. Analyzing inference privacy risks through gradients in machine learning. arXiv preprint arXiv:2408.16913, 2024.
  22. Why does differential privacy with large epsilon defend against practical membership inference attacks? arXiv preprint arXiv:2402.09540, 2024.
  23. Private stochastic optimization with large worst-case lipschitz parameter, 2022.
  24. Learning with user-level privacy. Advances in Neural Information Processing Systems, 34:12466–12479, 2021.
  25. How to make the gradients small privately: Improved rates for differentially private non-convex optimization. In Forty-first International Conference on Machine Learning, 2024.
  26. Frank D McSherry. Privacy integrated queries: an extensible platform for privacy-preserving data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, pages 19–30, 2009.
  27. Learning differentially private recurrent language models. In International Conference on Learning Representations (ICLR), 2018.
  28. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017.
  29. Stochastic convex optimization. In COLT, volume 2, page 5, 2009.
  30. Friendlycore: Practical differentially private aggregation. In International Conference on Machine Learning, pages 21828–21863. PMLR, 2022.
  31. Jonathan Ullman. CS7880: rigorous approaches to data privacy, 2017.
  32. Advances in private training for production on-device language models. https://research.google/blog/advances-in-private-training-for-production-on-device-language-models/, 2024. Google Research Blog.
  33. On stochastic gradient and subgradient methods with adaptive steplength sequences. Automatica, 48(1):56–67, 2012.
  34. Bring your own algorithm for optimal differentially private stochastic minimax optimization. Advances in Neural Information Processing Systems, 35:35174–35187, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now