Papers
Topics
Authors
Recent
Search
2000 character limit reached

Cryptographic Hardness of Score Estimation

Published 4 Apr 2024 in cs.LG, cs.CC, cs.CR, math.ST, stat.ML, and stat.TH | (2404.03272v1)

Abstract: We show that $L2$-accurate score estimation, in the absence of strong assumptions on the data distribution, is computationally hard even when sample complexity is polynomial in the relevant problem parameters. Our reduction builds on the result of Chen et al. (ICLR 2023), who showed that the problem of generating samples from an unknown data distribution reduces to $L2$-accurate score estimation. Our hard-to-estimate distributions are the "Gaussian pancakes" distributions, originally due to Diakonikolas et al. (FOCS 2017), which have been shown to be computationally indistinguishable from the standard Gaussian under widely believed hardness assumptions from lattice-based cryptography (Bruna et al., STOC 2021; Gupte et al., FOCS 2022).

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (86)
  1. “An improved constant in Banaszczyk’s transference theorem” In arXiv preprint arXiv:1907.09020, 2019
  2. Wojciech Banaszczyk “New bounds in some transference theorems in the geometry of numbers” In Mathematische Annalen 296 Springer-Verlag, 1993, pp. 625–635
  3. “Reducibility and statistical-computational gaps from secret leakage” In Conference on Learning Theory, 2020, pp. 648–847 PMLR
  4. “Nearly d-linear convergence bounds for diffusion models via stochastic localization” In The Twelfth International Conference on Learning Representations, 2024
  5. Dominique Bakry, Ivan Gentil and Michel Ledoux “Analysis and geometry of Markov diffusion operators” Springer, 2014
  6. “In Search of Non-Gaussian Components of a High-Dimensional Distribution.” In Journal of Machine Learning Research 7.2, 2006
  7. Adam Block, Youssef Mroueh and Alexander Rakhlin “Generative modeling with denoising auto-encoders and Langevin sampling” In arXiv preprint arXiv:2002.00107, 2020
  8. “Public-Key Encryption from Continuous LWE” In Cryptology ePrint Archive, 2022
  9. Afonso S Bandeira, Amelia Perry and Alexander S Wein “Notes on computational-to-statistical gaps: predictions using statistical physics” In Portugaliae Mathematica 75.2, 2018, pp. 159–186
  10. “Computational lower bounds for sparse PCA” In arXiv preprint arXiv:1304.0828, 2013
  11. “Continuous LWE” In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, 2021
  12. Ben Brubaker “In Neural Networks, Unbreakable Locks Can Hide Invisible Doors” In Quanta magazine, 2023 URL: https://www.quantamagazine.org/cryptographers-show-how-to-hide-invisible-backdoors-in-ai-20230302/
  13. “Sum-of-squares proofs and the quest toward optimal algorithms” In arXiv preprint arXiv:1404.5236, 2014
  14. Clément L Canonne “A short note on an inequality between KL and TV” In arXiv preprint arXiv:2202.07198, 2022
  15. Yi-Hsiu Chen, Kai-Min Chung and Jyun-Jie Liao “On the complexity of simulating auxiliary input” In Annual International Conference on the Theory and Applications of Cryptographic Techniques, 2018, pp. 371–390 Springer
  16. “Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions” In International Conference on Learning Representations (ICLR), 2023
  17. Miranda Christ, Sam Gunn and Or Zamir “Undetectable watermarks for language models” In arXiv preprint arXiv:2306.09194, 2023
  18. Sinho Chewi “Log-concave sampling”, 2024 URL: https://chewisinho.github.io/main.pdf
  19. “Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data” In International Conference on Machine Learning, 2023, pp. 4672–4712 PMLR
  20. Hongrui Chen, Holden Lee and Jianfeng Lu “Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions” In International Conference on Machine Learning, 2023, pp. 4735–4763 PMLR
  21. Valentin De Bortoli “Convergence of denoising diffusion models under the manifold hypothesis” In arXiv preprint arXiv:2208.05314, 2022
  22. Scott Decatur, Oded Goldreich and Dana Ron “Computational sample complexity” In Proceedings of the tenth annual conference on Computational learning theory, 1997, pp. 130–142
  23. “Statistical query lower bounds for tensor pca” In Journal of Machine Learning Research 22.83, 2021, pp. 1–51
  24. “Non-gaussian component analysis via lattice basis reduction” In Conference on Learning Theory, 2022, pp. 4535–4547 PMLR
  25. Ilias Diakonikolas and Daniel M. Kane “Algorithmic High-Dimensional Robust Statistics” Cambridge university press Cambridge, 2023
  26. “Algorithms and sq lower bounds for pac learning one-hidden-layer relu networks” In Conference on Learning Theory, 2020, pp. 1514–1539 PMLR
  27. Ilias Diakonikolas, Daniel Kane and Lisheng Ren “Near-optimal cryptographic hardness of agnostically learning halfspaces and relu regression under gaussian marginals” In International Conference on Machine Learning, 2023, pp. 7922–7938 PMLR
  28. “Outcome indistinguishability” In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, 2021, pp. 1095–1108
  29. “SQ Lower Bounds for Non-Gaussian Component Analysis with Weaker Assumptions” In Advances in Neural Information Processing Systems 36, 2024
  30. Ilias Diakonikolas, Daniel M Kane and Alistair Stewart “Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures” In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), 2017, pp. 73–84 IEEE
  31. Ilias Diakonikolas, Daniel Kane and Nikos Zarifis “Near-optimal sq lower bounds for agnostically learning halfspaces and relus under gaussian marginals” In Advances in Neural Information Processing Systems 33, 2020, pp. 13586–13596
  32. Amit Daniely, Nati Linial and Shai Shalev-Shwartz “From average case complexity to improper learning complexity” In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, 2014, pp. 441–448
  33. “Diffusion schrödinger bridge with applications to score-based generative modeling” In Advances in Neural Information Processing Systems 34, 2021, pp. 17695–17709
  34. “Statistical algorithms and a lower bound for detecting planted cliques” In J. ACM 64.2 New York, NY, USA: Association for Computing Machinery, 2017
  35. Jerome H Friedman and John W Tukey “A projection pursuit algorithm for exploratory data analysis” In IEEE Transactions on computers 100.9 IEEE, 1974, pp. 881–890
  36. David Gamarnik “The overlap gap property: A topological barrier to optimizing over random structures” In Proceedings of the National Academy of Sciences 118.41 National Acad Sciences, 2021, pp. e2108492118
  37. “Superpolynomial lower bounds for learning one-layer neural networks using gradient descent” In International Conference on Machine Learning, 2020, pp. 3587–3596 PMLR
  38. “Planting undetectable backdoors in machine learning models” In arXiv preprint arXiv:2204.06974, 2022
  39. Aparna Gupte, Neekon Vafa and Vinod Vaikuntanathan “Continuous LWE is as Hard as LWE & Applications to Learning Gaussian Mixtures” In arXiv preprint arXiv:2204.02550, 2022
  40. Jonathan Ho, Ajay Jain and Pieter Abbeel “Denoising diffusion probabilistic models” In Advances in neural information processing systems 33, 2020, pp. 6840–6851
  41. “Multicalibration: Calibration for the (computationally-identifiable) masses” In International Conference on Machine Learning, 2018, pp. 1939–1948 PMLR
  42. Peter J Huber “Projection pursuit” In The annals of Statistics JSTOR, 1985, pp. 435–475
  43. Aapo Hyvärinen “Estimation of non-normalized statistical models by score matching.” In Journal of Machine Learning Research 6.4, 2005
  44. “Deep residual learning for image recognition” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
  45. “How to fake auxiliary input” In Theory of Cryptography Conference, 2014, pp. 566–590 Springer
  46. Michael Kearns “Efficient noise-tolerant learning from statistical queries” In Journal of the ACM (JACM) 45.6 ACM New York, NY, USA, 1998, pp. 983–1006
  47. Dmitriy Kunisky “Spectral Barriers in Certification Problems”, 2022
  48. Dmitriy Kunisky, Alexander S Wein and Afonso S Bandeira “Notes on computational hardness of hypothesis testing: Predictions using the low-degree likelihood ratio” In Mathematical Analysis, its Applications and Computation: ISAAC 2019, Aveiro, Portugal, July 29–August 2 Springer, 2022, pp. 1–50
  49. Jean B Lasserre “Global optimization with polynomials and the problem of moments” In SIAM Journal on optimization 11.3 SIAM, 2001, pp. 796–817
  50. Holden Lee, Jianfeng Lu and Yixin Tan “Convergence for score-based generative modeling with polynomial complexity” In Advances in Neural Information Processing Systems 35, 2022, pp. 22870–22882
  51. Holden Lee, Jianfeng Lu and Yixin Tan “Convergence of score-based generative modeling for general data distributions” In International Conference on Algorithmic Learning Theory, 2023, pp. 946–985 PMLR
  52. “Better key sizes (and attacks) for LWE-based encryption” In Topics in Cryptology–CT-RSA 2011: The Cryptographers’ Track at the RSA Conference 2011, San Francisco, CA, USA, February 14-18, 2011. Proceedings, 2011, pp. 319–339 Springer
  53. “Let us build bridges: Understanding and extending diffusion generative models” In arXiv preprint arXiv:2208.14699, 2022
  54. “Lattice-based cryptography” In Post-quantum cryptography Springer, 2009, pp. 147–191
  55. “Computational barriers in minimax submatrix detection” In The Annals of Statistics 43.3 Institute of Mathematical Statistics, 2015
  56. “Deep networks as denoising algorithms: Sample-efficient learning of diffusion models in high-dimensional graphical models” In arXiv preprint arXiv:2309.11420, 2023
  57. NIST “Post-Quantum Cryptography” URL: https://csrc.nist.gov/Projects/Post-Quantum-Cryptography
  58. “Estimation of Wasserstein distances in the Spiked Transport Model” In Bernoulli 28.4 Bernoulli Society for Mathematical StatisticsProbability, 2022, pp. 2663–2688
  59. Pablo A Parrilo “Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization” California Institute of Technology, 2000
  60. Chris Peikert “A decade of lattice cryptography” In Foundations and trends® in theoretical computer science 10.4 Now Publishers, Inc., 2016, pp. 283–424
  61. Jakiw Pidstrigach “Score-based generative models detect manifolds” In Advances in Neural Information Processing Systems 35, 2022, pp. 35852–35865
  62. “High-resolution image synthesis with latent diffusion models” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695
  63. “Hierarchical text-conditional image generation with clip latents” In arXiv preprint arXiv:2204.06125, 2022
  64. Oded Regev “On lattices, learning with errors, random linear codes, and cryptography” In Journal of the ACM (JACM) 56.6 ACM New York, NY, USA, 2009, pp. 1–40
  65. “A statistical model for tensor PCA” In Advances in neural information processing systems 27, 2014
  66. Steven Roman “The Umbral Calculus” In Pure and Applied Mathematics Academic Press, 1984
  67. Kulin Shah, Sitan Chen and Adam Klivans “Learning mixtures of gaussians using the ddpm objective” In Advances in Neural Information Processing Systems 36, 2024
  68. “Photorealistic text-to-image diffusion models with deep language understanding” In Advances in neural information processing systems 35, 2022, pp. 36479–36494
  69. “Generative modeling by estimating gradients of the data distribution” In Advances in neural information processing systems 32, 2019
  70. Yang Song and Diederik P Kingma “How to train your energy-based models” In arXiv preprint arXiv:2101.03288, 2021
  71. “Applied stochastic differential equations” Cambridge University Press, 2019
  72. Shai Shalev-Shwartz, Ohad Shamir and Eran Tromer “Using more data to speed-up training time” In Artificial Intelligence and Statistics, 2012, pp. 1019–1027 PMLR
  73. Noah Stephens-Davidowitz “On the Gaussian Measure Over Lattices.”, 2017
  74. “Deep unsupervised learning using nonequilibrium thermodynamics” In International conference on machine learning, 2015, pp. 2256–2265 PMLR
  75. Min Jae Song, Ilias Zadik and Joan Bruna “On the Cryptographic Hardness of Learning Single Periodic Neurons” In Advances in Neural Processing Systems (NeurIPS), 2021
  76. Stefan Tiegel “Hardness of agnostically learning halfspaces from worst-case lattice problems” In The Thirty Sixth Annual Conference on Learning Theory, 2023, pp. 3029–3064 PMLR
  77. Luca Trevisan, Madhur Tulsiani and Salil Vadhan “Regularity, boosting, and efficiently simulating every high-entropy distribution” In 2009 24th Annual IEEE Conference on Computational Complexity, 2009, pp. 126–136 IEEE
  78. Roman Vershynin “High-dimensional probability: An introduction with applications in data science” Cambridge university press, 2018
  79. Pascal Vincent “A connection between score matching and denoising autoencoders” In Neural computation 23.7 MIT Press, 2011, pp. 1661–1674
  80. Martin J Wainwright “Constrained forms of statistical minimax: Computation, communication and privacy” In Proceedings of the International Congress of Mathematicians, 2014, pp. 13–21
  81. Andre Wibisono, Yihong Wu and Kaylee Yingxi Yang “Optimal score estimation via empirical Bayes smoothing” In arXiv preprint arXiv:2402.07747, 2024
  82. “Statistical problems with planted structures: Information theoretical and computational limits” In Information-Theoretic Methods in Data Science 383 Cambridge University Press, 2021, pp. 13
  83. Kaylee Yingxi Yang and Andre Wibisono “Convergence of the Inexact Langevin Algorithm and Score-based Generative Models in KL Divergence” In arXiv preprint arXiv:2211.01512, 2022
  84. “Statistical physics of inference: Thresholds and algorithms” In Advances in Physics 65.5 Taylor & Francis, 2016, pp. 453–552
  85. “Lattice-based methods surpass sum-of-squares in clustering” In Conference on Learning Theory, 2022, pp. 1247–1248 PMLR
  86. Yuchen Zhang, Martin J Wainwright and Michael I Jordan “Lower bounds on the performance of polynomial-time algorithms for sparse linear regression” In Conference on Learning Theory, 2014, pp. 921–948 PMLR

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free