Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SQ Lower Bounds for Non-Gaussian Component Analysis with Weaker Assumptions (2403.04744v1)

Published 7 Mar 2024 in cs.LG, cs.DS, math.ST, stat.ML, and stat.TH

Abstract: We study the complexity of Non-Gaussian Component Analysis (NGCA) in the Statistical Query (SQ) model. Prior work developed a general methodology to prove SQ lower bounds for this task that have been applicable to a wide range of contexts. In particular, it was known that for any univariate distribution $A$ satisfying certain conditions, distinguishing between a standard multivariate Gaussian and a distribution that behaves like $A$ in a random hidden direction and like a standard Gaussian in the orthogonal complement, is SQ-hard. The required conditions were that (1) $A$ matches many low-order moments with the standard univariate Gaussian, and (2) the chi-squared norm of $A$ with respect to the standard Gaussian is finite. While the moment-matching condition is necessary for hardness, the chi-squared condition was only required for technical reasons. In this work, we establish that the latter condition is indeed not necessary. In particular, we prove near-optimal SQ lower bounds for NGCA under the moment-matching condition only. Our result naturally generalizes to the setting of a hidden subspace. Leveraging our general SQ lower bound, we obtain near-optimal SQ lower bounds for a range of concrete estimation tasks where existing techniques provide sub-optimal or even vacuous guarantees.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (30)
  1. In search of non-gaussian components of a high-dimensional distribution. Journal of Machine Learning Research, 7(9):247–282, 2006.
  2. Continuous LWE. In 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 694–707. ACM, 2021.
  3. Learning (very) simple generative models is hard. In NeurIPS, 2022.
  4. J. Ding and Y. Hua. SQ lower bounds for random sparse planted vector problem. In International Conference on Algorithmic Learning Theory, volume 201 of Proceedings of Machine Learning Research, pages 558–596. PMLR, 2023.
  5. I. Diakonikolas and D. Kane. Near-optimal statistical query hardness of learning halfspaces with massart noise. In Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 4258–4282. PMLR, 2022. Full version available at https://arxiv.org/abs/2012.09720.
  6. I. Diakonikolas and D. Kane. Non-gaussian component analysis via lattice basis reduction. In Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 4535–4547. PMLR, 2022.
  7. I. Diakonikolas and D. M. Kane. Algorithmic High-Dimensional Robust Statistics. Cambridge university press, 2023. Full version available at https://sites.google.com/view/ars-book.
  8. Learning general halfspaces with general massart noise under the gaussian distribution. In STOC ’22: 54th Annual ACM SIGACT Symposium on Theory of Computing, pages 874–885, 2022. Full version available at https://arxiv.org/abs/2108.08767.
  9. Algorithms and SQ lower bounds for PAC learning one-hidden-layer relu networks. In Conference on Learning Theory, COLT 2020, volume 125 of Proceedings of Machine Learning Research, pages 1514–1539. PMLR, 2020.
  10. Statistical query lower bounds for list-decodable linear regression. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pages 3191–3204, 2021.
  11. The optimality of polynomial regression for agnostic learning under gaussian marginals in the SQ model. In Conference on Learning Theory, COLT 2021, volume 134 of Proceedings of Machine Learning Research, pages 1552–1584. PMLR, 2021.
  12. SQ lower bounds for learning mixtures of separated and bounded covariance gaussians. In The Thirty Sixth Annual Conference on Learning Theory, COLT 2023, volume 195 of Proceedings of Machine Learning Research, pages 2319–2349. PMLR, 2023.
  13. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, pages 73–84, 2017. Full version at http://arxiv.org/abs/1611.03473.
  14. List-decodable robust mean estimation and learning mixtures of spherical gaussians. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, pages 1047–1060, 2018. Full version available at https://arxiv.org/abs/1711.07211.
  15. Efficient algorithms and lower bounds for robust linear regression. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2019, pages 2745–2754, 2019.
  16. SQ lower bounds for learning mixtures of linear classifiers. CoRR, abs/2310.11876, 2023.
  17. Near-optimal SQ lower bounds for agnostically learning halfspaces and relus under gaussian marginals. CoRR, abs/2006.16200, 2020.
  18. V. Feldman. Statistical query learning. In Encyclopedia of Algorithms, pages 2090–2095. Springer New York, 2016.
  19. Statistical algorithms and a lower bound for detecting planted cliques. J. ACM, 64(2):8:1–8:37, 2017.
  20. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2017, pages 1265–1277. SIAM, 2017.
  21. Statistical-query lower bounds via functional gradients. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
  22. N. Goyal and A. Shetty. Non-gaussian component analysis using entropy methods. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, pages 840–851. ACM, 2019.
  23. M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM, 45(6):983–1006, 1998.
  24. I. Krasikov. New bounds on the Hermite polynomials. arXiv preprint math/0401310, 2004.
  25. M. Meister and G. Valiant. A data prism: Semi-verified learning in the small-alpha regime. In Conference On Learning Theory, COLT 2018, volume 75 of Proceedings of Machine Learning Research, pages 1530–1546. PMLR, 2018.
  26. On the complexity of learning neural networks. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NeurIPS 2017, pages 5514–5522, 2017.
  27. On the cryptographic hardness of learning single periodic neurons. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pages 29602–29615, 2021.
  28. Y. S. Tan and R. Vershynin. Polynomial time and sample complexity for non-gaussian component analysis: Spectral methods. In Conference On Learning Theory, COLT 2018, volume 75 of Proceedings of Machine Learning Research, pages 498–534. PMLR, 2018.
  29. L. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.
  30. Lattice-based methods surpass sum-of-squares in clustering. In Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 1247–1248. PMLR, 2022.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now