Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Private PAC Learning May be Harder than Online Learning (2402.11119v1)

Published 16 Feb 2024 in cs.LG, cs.CR, and cs.DS

Abstract: We continue the study of the computational complexity of differentially private PAC learning and how it is situated within the foundations of machine learning. A recent line of work uncovered a qualitative equivalence between the private PAC model and Littlestone's mistake-bounded model of online learning, in particular, showing that any concept class of Littlestone dimension $d$ can be privately PAC learned using $\mathrm{poly}(d)$ samples. This raises the natural question of whether there might be a generic conversion from online learners to private PAC learners that also preserves computational efficiency. We give a negative answer to this question under reasonable cryptographic assumptions (roughly, those from which it is possible to build indistinguishability obfuscation for all circuits). We exhibit a concept class that admits an online learner running in polynomial time with a polynomial mistake bound, but for which there is no computationally-efficient differentially private PAC learner. Our construction and analysis strengthens and generalizes that of Bun and Zhandry (TCC 2016-A), who established such a separation between private and non-private PAC learner.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Gentle measurement of quantum states and differential privacy. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019, pages 322–333. ACM, 2019. doi: 10.1145/3313276.3316378. URL https://doi.org/10.1145/3313276.3316378.
  2. The price of differential privacy for online learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 32–40. PMLR, 2017. URL http://proceedings.mlr.press/v70/agarwal17a.html.
  3. Private PAC learning implies finite Littlestone dimension. In Proceedings of the 51st Annual ACM Symposium on the Theory of Computing, STOC ’19, New York, NY, USA, 2019. ACM.
  4. Private and online learnability are equivalent. J. ACM, 69(4):28:1–28:34, 2022. doi: 10.1145/3526074. URL https://doi.org/10.1145/3526074.
  5. Dana Angluin. Queries and concept learning. Machine Learning, 2(4):319–342, 1988.
  6. Private online prediction from experts: Separations and faster rates. In Gergely Neu and Lorenzo Rosasco, editors, The Thirty Sixth Annual Conference on Learning Theory, COLT 2023, 12-15 July 2023, Bangalore, India, volume 195 of Proceedings of Machine Learning Research, pages 674–699. PMLR, 2023. URL https://proceedings.mlr.press/v195/asi23a.html.
  7. Privacy amplification by subsampling: Tight analyses via couplings and divergences, 2018.
  8. Derandomization in cryptography. SIAM Journal on Computing, 37(2):380–400, 2007. doi: 10.1137/050641958.
  9. Bounds on the sample complexity for private learning and private data release. Machine Learning, 94(3):401–437, 2014.
  10. Private learning and sanitization: Pure vs. approximate differential privacy. Theory of Computing, 12(1):1–61, 2016.
  11. Tighter bounds on multi-party coin flipping via augmented weak martingales and differentially private sampling. In Mikkel Thorup, editor, 59th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2018, Paris, France, October 7-9, 2018, pages 838–849. IEEE Computer Society, 2018. doi: 10.1109/FOCS.2018.00084. URL https://doi.org/10.1109/FOCS.2018.00084.
  12. Characterizing the sample complexity of pure private learners. Journal of Machine Learning Research, 20(146):1–33, 2019. URL http://jmlr.org/papers/v20/18-269.html.
  13. A note on perfect correctness by derandomization. J. Cryptol., 35(3):18, 2022. doi: 10.1007/s00145-022-09428-0. URL https://doi.org/10.1007/s00145-022-09428-0.
  14. Practical privacy: The SuLQ framework. In Proceedings of the 24th ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS ’05, pages 128–138, New York, NY, USA, 2005. ACM.
  15. Multi-input functional encryption in the private-key setting: Stronger security from weaker assumptions. J. Cryptol., 31(2):434–520, 2018. doi: 10.1007/s00145-017-9261-0. URL https://doi.org/10.1007/s00145-017-9261-0.
  16. Mark Bun. A computational separation between private learning and online learning. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ee715daa76f1b51d80343f45547be570-Abstract.html.
  17. Order-revealing encryption and the hardness of private learning. In Eyal Kushilevitz and Tal Malkin, editors, Theory of Cryptography - 13th International Conference, TCC 2016-A, Tel Aviv, Israel, January 10-13, 2016, Proceedings, Part I, volume 9562 of Lecture Notes in Computer Science, pages 176–206. Springer, 2016.
  18. Differentially private release and learning of threshold functions. In Proceedings of the 56th Annual IEEE Symposium on Foundations of Computer Science, FOCS ’15, pages 634–649, Washington, DC, USA, 2015. IEEE Computer Society.
  19. Stability is stable: Connections between replicability, privacy, and adaptive generalization. In Barna Saha and Rocco A. Servedio, editors, Proceedings of the 55th Annual ACM Symposium on Theory of Computing, STOC 2023, Orlando, FL, USA, June 20-23, 2023, pages 520–527. ACM, 2023. doi: 10.1145/3564246.3585246. URL https://doi.org/10.1145/3564246.3585246.
  20. Reducing the leakage in practical order-revealing encryption. IACR Cryptol. ePrint Arch., page 661, 2016. URL http://eprint.iacr.org/2016/661.
  21. Practical order-revealing encryption with limited leakage. In Thomas Peyrin, editor, Fast Software Encryption - 23rd International Conference, FSE 2016, Bochum, Germany, March 20-23, 2016, Revised Selected Papers, volume 9783 of Lecture Notes in Computer Science, pages 474–493. Springer, 2016. doi: 10.1007/978-3-662-52993-5_24. URL https://doi.org/10.1007/978-3-662-52993-5_24.
  22. Differential privacy and robust statistics. In Proceedings of the 41st Annual ACM Symposium on the Theory of Computing, STOC ’09, pages 371–380, New York, NY, USA, 2009. ACM.
  23. Our data, ourselves: Privacy via distributed noise generation. In Proceedings of the 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, EUROCRYPT ’06, pages 486–503, Berlin, Heidelberg, 2006a. Springer.
  24. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Conference on Theory of Cryptography, TCC ’06, pages 265–284, Berlin, Heidelberg, 2006b. Springer.
  25. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248):636–638, 2015.
  26. Sample complexity bounds on differentially private learning via communication complexity. SIAM Journal on Computing, 44(6):1740–1764, 2015.
  27. Optimal mistake bound learning is hard. Inf. Comput., 144(1):66–82, 1998. doi: 10.1006/inco.1998.2709. URL https://doi.org/10.1006/inco.1998.2709.
  28. Sample-efficient proper PAC learning with approximate differential privacy. In Samir Khuller and Virginia Vassilevska Williams, editors, STOC ’21: 53rd Annual ACM SIGACT Symposium on Theory of Computing, Virtual Event, Italy, June 21-25, 2021, pages 183–196. ACM, 2021. doi: 10.1145/3406325.3451028. URL https://doi.org/10.1145/3406325.3451028.
  29. Private learning implies online learning: An efficient reduction. NeurIPS, 2019.
  30. New techniques for noninteractive zero-knowledge. J. ACM, 59(3), jun 2012. ISSN 0004-5411. doi: 10.1145/2220357.2220358. URL https://doi.org/10.1145/2220357.2220358.
  31. Preventing false discovery in interactive data analysis is hard. In Proceedings of the 55th Annual IEEE Symposium on Foundations of Computer Science, FOCS ’14, pages 454–463, Washington, DC, USA, 2014. IEEE Computer Society.
  32. Indistinguishability obfuscation from well-founded assumptions. In Samir Khuller and Virginia Vassilevska Williams, editors, STOC ’21: 53rd Annual ACM SIGACT Symposium on Theory of Computing, Virtual Event, Italy, June 21-25, 2021, pages 60–73. ACM, 2021. doi: 10.1145/3406325.3451093. URL https://doi.org/10.1145/3406325.3451093.
  33. Privately learning thresholds: Closing the exponential gap. CoRR, abs/1911.10137, 2019. URL http://arxiv.org/abs/1911.10137.
  34. What can we learn privately? SIAM Journal on Computing, 40(3):793–826, 2011.
  35. Michael J. Kearns. Efficient noise-tolerant learning from statistical queries. J. ACM, 45(6):983–1006, 1998. doi: 10.1145/293347.293351. URL https://doi.org/10.1145/293347.293351.
  36. On the learnability of boolean formulae. In Alfred V. Aho, editor, Proceedings of the 19th Annual ACM Symposium on Theory of Computing, 1987, New York, New York, USA, pages 285–295. ACM, 1987. doi: 10.1145/28395.28426. URL https://doi.org/10.1145/28395.28426.
  37. Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2(4):285–318, 1987.
  38. Nick Littlestone. From on-line to batch learning. In Ronald L. Rivest, David Haussler, and Manfred K. Warmuth, editors, Proceedings of the Second Annual Workshop on Computational Learning Theory, COLT 1989, Santa Cruz, CA, USA, July 31 - August 2, 1989, pages 269–284. Morgan Kaufmann, 1989. URL http://dl.acm.org/citation.cfm?id=93365.
  39. Pasin Manurangsi. Improved inapproximability of VC dimension and littlestone’s dimension via (unbalanced) biclique. In Yael Tauman Kalai, editor, 14th Innovations in Theoretical Computer Science Conference, ITCS 2023, January 10-13, 2023, MIT, Cambridge, Massachusetts, USA, volume 251 of LIPIcs, pages 85:1–85:18. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2023. doi: 10.4230/LIPIcs.ITCS.2023.85. URL https://doi.org/10.4230/LIPIcs.ITCS.2023.85.
  40. Inapproximability of VC dimension and littlestone’s dimension. CoRR, abs/1705.09517, 2017. URL http://arxiv.org/abs/1705.09517.
  41. Mechanism design via differential privacy. In Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science, FOCS ’07, pages 94–103, Washington, DC, USA, 2007. IEEE Computer Society.
  42. Approximately optimal mechanism design via differential privacy. In Shafi Goldwasser, editor, Innovations in Theoretical Computer Science 2012, Cambridge, MA, USA, January 8-10, 2012, pages 203–213. ACM, 2012. doi: 10.1145/2090236.2090254. URL https://doi.org/10.1145/2090236.2090254.
  43. On the sample complexity of privately learning axis-aligned rectangles. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 28286–28297, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ee0e95249268b86ff2053bef214bfeda-Abstract.html.
  44. Marcus Schaefer. Deciding the Vapnik-Červonenkis dimension is Σ3psubscriptsuperscriptΣp3\Sigma^{\text{p}}_{3}roman_Σ start_POSTSUPERSCRIPT p end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT-complete. J. Comput. Syst. Sci., 58(1):177–182, 1999. doi: 10.1006/jcss.1998.1602. URL https://doi.org/10.1006/jcss.1998.1602.
  45. Leslie G. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now