Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Oracle-Efficient Differentially Private Learning with Public Data (2402.09483v1)

Published 13 Feb 2024 in stat.ML, cs.CR, and cs.LG

Abstract: Due to statistical lower bounds on the learnability of many function classes under privacy constraints, there has been recent interest in leveraging public data to improve the performance of private learning algorithms. In this model, algorithms must always guarantee differential privacy with respect to the private samples while also ensuring learning guarantees when the private data distribution is sufficiently close to that of the public data. Previous work has demonstrated that when sufficient public, unlabelled data is available, private learning can be made statistically tractable, but the resulting algorithms have all been computationally inefficient. In this work, we present the first computationally efficient, algorithms to provably leverage public data to learn privately whenever a function class is learnable non-privately, where our notion of computational efficiency is with respect to the number of calls to an optimization oracle for the function class. In addition to this general result, we provide specialized algorithms with improved sample complexities in the special cases when the function class is convex or when the task is binary classification.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Online learning via the differential privacy lens. Advances in Neural Information Processing Systems, 32, 2019.
  2. Private PAC learning implies finite littlestone dimension. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019, pages 852–860. ACM, 2019. doi: 10.1145/3313276.3316312. URL https://doi.org/10.1145/3313276.3316312.
  3. Public data-assisted mirror descent for private model training. In International Conference on Machine Learning, pages 517–535. PMLR, 2022.
  4. Fat-shattering and the learnability of real-valued functions. In Proceedings of the seventh annual conference on Computational learning theory, pages 299–310, 1994.
  5. Limits of private learning with access to public data. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 10342–10352, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/9a6a1aaafe73c572b7374828b03a1881-Abstract.html.
  6. Private query release assisted by public data. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 695–703. PMLR, 2020a. URL http://proceedings.mlr.press/v119/bassily20a.html.
  7. Learning from mixtures of private and public populations. Advances in Neural Information Processing Systems, 33:2947–2957, 2020b.
  8. Private domain adaptation from a public source. arXiv preprint arXiv:2208.06135, 2022.
  9. Principled approaches for private adaptation from a public source. In International Conference on Artificial Intelligence and Statistics, pages 8405–8432. PMLR, 2023.
  10. Learning privately with labeled and unlabeled examples. In Proceedings of the twenty-sixth annual ACM-SIAM symposium on Discrete algorithms, pages 461–477. SIAM, 2014.
  11. Private distribution learning with public data: The view from sample compression. arXiv preprint arXiv:2308.06239, 2023.
  12. Private estimation with public data. Advances in Neural Information Processing Systems, 35:18653–18666, 2022.
  13. The sample complexity of approximate rejection sampling with applications to smoothed online learning. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 228–273. PMLR, 12–15 Jul 2023. URL https://proceedings.mlr.press/v195/block23a.html.
  14. Efficient and near-optimal smoothed online learning for generalized linear functions. Advances in Neural Information Processing Systems, 35:7477–7489, 2022.
  15. Smoothed online learning is as easy as statistical learning. In Conference on Learning Theory, pages 1716–1786. PMLR, 2022.
  16. Oracle-efficient smoothed online learning for piecewise continuous decision making. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 1618–1665. PMLR, 12–15 Jul 2023a. URL https://proceedings.mlr.press/v195/block23b.html.
  17. Smoothed online learning for prediction in piecewise affine systems. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2023b. URL https://openreview.net/pdf?id=Izt7rDD7jN.
  18. Training a 3-node neural network is np-complete. Advances in neural information processing systems, 1, 1988.
  19. Olivier Bousquet. Concentration inequalities and empirical processes theory applied to the analysis of learning algorithms. PhD thesis, École Polytechnique: Department of Applied Mathematics Paris, France, 2002.
  20. Differentially private release and learning of threshold functions. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 634–649. IEEE, 2015.
  21. An equivalence between private classification and online prediction. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS), pages 389–402. IEEE, 2020.
  22. Prediction, learning, and games. Cambridge university press, 2006.
  23. Repeated bilateral trade against a smoothed adversary. In The Thirty Sixth Annual Conference on Learning Theory, pages 1095–1130. PMLR, 2023.
  24. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011.
  25. Comparison and anti-concentration bounds for maxima of gaussian random vectors. Probability Theory and Related Fields, 162:47–70, 2015.
  26. Oracle-efficient online learning and auction design. Journal of the ACM (JACM), 67(5):1–57, 2020.
  27. Richard Mansfield Dudley. The speed of mean glivenko-cantelli convergence. The Annals of Mathematical Statistics, 40(1):40–50, 1969.
  28. Smoothed analysis of online non-parametric auctions. In Proceedings of the 24th ACM Conference on Economics and Computation, pages 540–560, 2023.
  29. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings, volume 3876 of Lecture Notes in Computer Science, pages 265–284. Springer, 2006. doi: 10.1007/11681878_14. URL https://doi.org/10.1007/11681878_14.
  30. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  31. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pages 3199–3210. PMLR, 2020.
  32. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487, 2021.
  33. Dual query: Practical private query release for high dimensional data. In International Conference on Machine Learning, pages 1170–1178. PMLR, 2014.
  34. Sample-efficient proper pac learning with approximate differential privacy. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 183–196, 2021.
  35. Mixed differential privacy in computer vision. In CVPR 2022, 2022. URL https://www.amazon.science/publications/mixed-differential-privacy-in-computer-vision.
  36. Exact identification of read-once formulas using fixed points of amplification functions. SIAM Journal on Computing, 22(4):705–726, 1993.
  37. Geoffrey J Gordon. Regret bounds for prediction problems. In Proceedings of the twelfth annual conference on Computational learning theory, pages 29–40, 1999.
  38. Smoothed analysis of online and differentially private learning. Advances in Neural Information Processing Systems, 33:9203–9215, 2020.
  39. Oracle-efficient online learning for smoothed adversaries. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 4072–4084. Curran Associates, Inc., 2022a. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/1a04df6a405210aab4986994b873db9b-Paper-Conference.pdf.
  40. Smoothed analysis with adaptive adversaries. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 942–953. IEEE, 2022b.
  41. The computational power of optimization in online learning. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 128–141, 2016.
  42. (Nearly) dimension independent private erm with adagrad rates via publicly estimated subspaces. In Conference on Learning Theory, pages 2717–2746. PMLR, 2021.
  43. Efficient algorithms for online decision problems. Journal of Computer and System Sciences, 71(3):291–307, 2005.
  44. Cube root asymptotics. The Annals of Statistics, pages 191–219, 1990.
  45. Simple online learning with consistency oracle. arXiv preprint arXiv:2308.08055, 2023.
  46. Jean-François Le Gall. Brownian motion, martingales, and stochastic calculus. Springer, 2016.
  47. Leveraging public data for practical private query release. In International Conference on Machine Learning, pages 6968–6977. PMLR, 2021a.
  48. Iterative methods for private synthetic data: Unifying framework and new methods. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 690–702, 2021b. URL https://proceedings.neurips.cc/paper/2021/hash/0678c572b0d5597d2d4a6b5bd135754c-Abstract.html.
  49. Optimal differentially private learning with public data. arXiv preprint arXiv:2306.15056, 2023.
  50. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2007), October 20-23, 2007, Providence, RI, USA, Proceedings, pages 94–103. IEEE Computer Society, 2007. doi: 10.1109/FOCS.2007.41. URL https://doi.org/10.1109/FOCS.2007.41.
  51. Entropy and the combinatorial dimension. Inventiones mathematicae, 152(1):37–55, 2003.
  52. Efficient model-free exploration in low-rank mdps. arXiv preprint arXiv:2307.03997, 2023a.
  53. Representation learning with multi-step inverse kinematics: An efficient and optimal approach to rich-observation rl. arXiv preprint arXiv:2304.05889, 2023b.
  54. Oracle efficient private non-convex optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 7243–7252. PMLR, 2020. URL http://proceedings.mlr.press/v119/neel20a.html.
  55. How to use heuristics for differential privacy. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pages 72–93. IEEE, 2019.
  56. The geometry of differential privacy: the sparse and approximate cases. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pages 351–360, 2013.
  57. Scalable private learning with PATE. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=rkZB1XbRZ.
  58. Information Theory: From Coding to Learning. Cambridge University Press, 2022+.
  59. Online learning: Stochastic, constrained, and smoothed adversaries. Advances in neural information processing systems, 24, 2011.
  60. Online learning via sequential complexities. J. Mach. Learn. Res., 16(1):155–186, 2015.
  61. Empirical entropy, minimax regret and minimax risk. Bernoulli, pages 789–824, 2017.
  62. Ralph Tyrell Rockafellar. Convex Analysis:(PMS-28). Princeton university press, 2015.
  63. Combinatorics of random processes and sections of convex bodies. Annals of Mathematics, pages 603–648, 2006.
  64. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM), 51(3):385–463, 2004.
  65. Smoothness, low noise and fast rates. Advances in neural information processing systems, 23, 2010.
  66. Efficient algorithms for adversarial contextual learning. In International Conference on Machine Learning, pages 2159–2168. PMLR, 2016.
  67. Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.
  68. Ramon Van Handel. Probability in high dimension. Lecture Notes (Princeton University), 2014.
  69. New oracle-efficient algorithms for private synthetic data release. In International Conference on Machine Learning, pages 9765–9774. PMLR, 2020.
  70. Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
  71. Differentially private fine-tuning of language models. In International Conference on Learning Representations (ICLR), 2022.
  72. Bypassing the ambient dimension: Private SGD with gradient subspace identification. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=7dpmlkBuJFC.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets