Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Mirror Descent Algorithms with Nearly Dimension-Independent Rates for Differentially-Private Stochastic Saddle-Point Problems (2403.02912v1)

Published 5 Mar 2024 in math.OC, cs.CR, and cs.LG

Abstract: We study the problem of differentially-private (DP) stochastic (convex-concave) saddle-points in the polyhedral setting. We propose $(\varepsilon, \delta)$-DP algorithms based on stochastic mirror descent that attain nearly dimension-independent convergence rates for the expected duality gap, a type of guarantee that was known before only for bilinear objectives. For convex-concave and first-order-smooth stochastic objectives, our algorithms attain a rate of $\sqrt{\log(d)/n} + (\log(d){3/2}/[n\varepsilon]){1/3}$, where $d$ is the dimension of the problem and $n$ the dataset size. Under an additional second-order-smoothness assumption, we improve the rate on the expected gap to $\sqrt{\log(d)/n} + (\log(d){3/2}/[n\varepsilon]){2/5}$. Under this additional assumption, we also show, by using bias-reduced gradient estimators, that the duality gap is bounded by $\log(d)/\sqrt{n} + \log(d)/[n\varepsilon]{1/2}$ with constant success probability. This result provides evidence of the near-optimality of the approach. Finally, we show that combining our methods with acceleration techniques from online learning leads to the first algorithm for DP Stochastic Convex Optimization in the polyhedral setting that is not based on Frank-Wolfe methods. For convex and first-order-smooth stochastic objectives, our algorithms attain an excess risk of $\sqrt{\log(d)/n} + \log(d){7/10}/[n\varepsilon]{2/5}$, and when additionally assuming second-order-smoothness, we improve the rate to $\sqrt{\log(d)/n} + \log(d)/\sqrt{n\varepsilon}$. Instrumental to all of these results are various extensions of the classical Maurey Sparsification Lemma, which may be of independent interest.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Distributed learning with sublinear communication. In K. Chaudhuri and R. Salakhutdinov, editors, ICML, volume 97 of Proceedings of Machine Learning Research, pages 40–50. PMLR, 2019. URL http://dblp.uni-trier.de/db/conf/icml/icml2019.html#AcharyaSFS19.
  2. Stochastic bias-reduced gradient methods. In Advances in Neural Information Processing Systems, 2021a.
  3. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning, pages 393–403. PMLR, 2021b.
  4. A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930–945, 1993.
  5. Private empirical risk minimization: Efficient algorithms and tight error bounds. In FOCS, pages 464–473. IEEE Computer Society, 2014.
  6. Private stochastic convex optimization with optimal rates. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/3bd8fdb090f1f5eb66a00c84dbc5ad51-Paper.pdf.
  7. Differentially private stochastic optimization: New results in convex and non-convex settings. Advances in Neural Information Processing Systems, 34, 2021a.
  8. Non-euclidean differentially private stochastic convex optimization. In M. Belkin and S. Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 474–499. PMLR, 15–19 Aug 2021b. URL https://proceedings.mlr.press/v134/bassily21a.html.
  9. Differentially private algorithms for the stochastic saddle point problem with optimal rates for the strong gap. In G. Neu and L. Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 2482–2508. PMLR, 12–15 Jul 2023. URL https://proceedings.mlr.press/v195/bassily23a.html.
  10. Unbiased monte carlo for optimization and functions of expectations via multi-level randomization. In Proceedings of the 2015 Winter Simulation Conference, Huntington Beach, CA, USA, December 6-9, 2015, pages 3656–3667. IEEE/ACM, 2015. doi: 10.1109/WSC.2015.7408524. URL https://doi.org/10.1109/WSC.2015.7408524.
  11. Learning energy networks with generalized fenchel-young losses. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=CmD5z_2DVuM.
  12. D. Boob and C. Guzmán. Optimal algorithms for differentially private stochastic monotone variational inequalities and saddle-point problems. Mathematical Programming, pages 1–43, 2023.
  13. Fingerprinting codes and the price of approximate differential privacy. In Proceedings of the Forty-Sixth Annual ACM Symposium on Theory of Computing, STOC ’14, page 1–10, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450327107. doi: 10.1145/2591796.2591877. URL https://doi.org/10.1145/2591796.2591877.
  14. C. Carathéodory. Über den variabilitätsbereich der fourier’schen konstanten von positiven harmonischen funktionen. Rendiconti del Circolo Matematico di Palermo (1884-1940), 32:193–217, 1911. URL https://api.semanticscholar.org/CorpusID:120032616.
  15. Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, July 2006. ISBN 0471241954.
  16. A. Cutkosky. Anytime online-to-batch, optimism and acceleration. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1446–1454. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/cutkosky19a.html.
  17. Optimal affine-invariant smooth minimization algorithms. SIAM Journal on Optimization, 28(3):2384–2405, 2018.
  18. Our data, ourselves: Privacy via distributed noise generation. In Annual international conference on the theory and applications of cryptographic techniques, pages 486–503. Springer, 2006a.
  19. Calibrating noise to sensitivity in private data analysis. In TCC, volume 3876 of Lecture Notes in Computer Science, pages 265–284. Springer, 2006b.
  20. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  21. Dual query: Practical private query release for high dimensional data. In International Conference on Machine Learning, pages 1170–1178. PMLR, 2014.
  22. Differentially private optimization with sparse gradients. Unpublished manuscript, 2024.
  23. Frank-Wolfe Algorithms for Saddle Point Problems. In A. Singh and J. Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 362–371. PMLR, 20–22 Apr 2017. URL https://proceedings.mlr.press/v54/gidel17a.html.
  24. Private streaming SCO in ℓpsubscriptℓ𝑝\ell_{p}roman_ℓ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT geometry with applications in high dimensional online decision making. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8249–8279. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/han22d.html.
  25. M. Hardt and G. N. Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In in FOCS, pages 61–70, 2010.
  26. A simple and practical algorithm for differentially private data release. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2, NIPS’12, page 2339–2347, Red Hook, NY, USA, 2012. Curran Associates Inc.
  27. Differential privacy for the analyst via private equilibrium computation. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pages 341–350, 2013.
  28. Differentially private fair learning. In Proceedings of the 36 th International Conference on Machine Learning, 2019.
  29. P. Jain and A. G. Thakurta. (near) dimension independent risk bounds for differentially private learning. In International Conference on Machine Learning, pages 476–484. PMLR, 2014.
  30. A. Juditsky and A. S. Nemirovski. Large deviations of vector-valued martingales in 2-smooth normed spaces, 2008.
  31. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems, 1(1):17 – 58, 2011. doi: 10.1214/10-SSY011. URL https://doi.org/10.1214/10-SSY011.
  32. Stability and generalization of differentially private minimax problems. arXiv preprint arXiv:2204.04858, 2022.
  33. Fair resource allocation in federated learning. CoRR, abs/1905.10497, 2019. URL http://arxiv.org/abs/1905.10497.
  34. Stochastic differentially private and fair learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=3nM5uhPlfv6.
  35. P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: performance guarantees and tractable reformulations. Mathematical Programming, 171(1-2):115–166, 2018.
  36. Robust stochastic approximation approach to stochastic programming. SIAM J. Optimization, 19(4):1574–1609, 2009. URL http://dblp.uni-trier.de/db/journals/siamjo/siamjo19.html#NemirovskiJLS09.
  37. F. Orabona. A modern introduction to online learning. CoRR, abs/1912.13213, 2019. URL http://arxiv.org/abs/1912.13213.
  38. G. Pisier. Remarques sur un résultat non publié de B. Maurey. Séminaire Analyse fonctionnelle (dit ”Maurey-Schwartz”), pages 1–12, 1980-1981. URL http://eudml.org/doc/109255.
  39. G. Pisier. Martingales in Banach Spaces. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2016. ISBN 9781107137240.
  40. R. Rockafellar. Monotone operators associated with saddle-functions and minimax problems. nonlinear functional analysis, vol. i. In Proceedings of Symposia in Pure Mathematical American Mathematical Society, 1970.
  41. Private empirical risk minimization beyond the worst case: The effect of the constraint set geometry. arXiv preprint arXiv:1411.5417, 2014.
  42. Nearly-optimal private lasso. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2, pages 3025–3033, 2015.
  43. R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Number 47 in Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. ISBN 978-1-108-41519-4.
  44. B. Waggoner. Lp testing and learning of discrete distributions. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science, ITCS ’15, page 347–356, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450333337. doi: 10.1145/2688073.2688095. URL https://doi.org/10.1145/2688073.2688095.
  45. A. Wald. On cumulative sums of random variables. The Annals of Mathematical Statistics, 15(3):283–296, 1944.
  46. Fully-adaptive composition in differential privacy. In International Conference on Machine Learning, pages 36990–37007. PMLR, 2023.
  47. Differentially private sgda for minimax problems. In J. Cussens and K. Zhang, editors, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pages 2192–2202. PMLR, 01–05 Aug 2022.
  48. Bring your own algorithm for optimal differentially private stochastic minimax optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022a. URL http://papers.nips.cc/paper_files/paper/2022/hash/e46fc33e80e9fa2febcdb058fba4beca-Abstract-Conference.html.
  49. Differentially private online-to-batch for smooth losses. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022b. URL https://openreview.net/forum?id=fLOU5jXlJZV.
  50. T. Zhang. Covering number bounds of certain regularized linear function classes. J. Mach. Learn. Res., 2:527–550, 2002. URL http://jmlr.org/papers/v2/zhang02b.html.
  51. X. Zhou and R. Bassily. Differentially private worst-group risk minimization. unpublished manuscript, 2024.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now