Statistical Query Lower Bounds for Robust Estimation of High-dimensional Gaussians and Gaussian Mixtures (1611.03473v2)

Abstract: We describe a general technique that yields the first {\em Statistical Query lower bounds} for a range of fundamental high-dimensional learning problems involving Gaussian distributions. Our main results are for the problems of (1) learning Gaussian mixture models (GMMs), and (2) robust (agnostic) learning of a single unknown Gaussian distribution. For each of these problems, we show a {\em super-polynomial gap} between the (information-theoretic) sample complexity and the computational complexity of {\em any} Statistical Query algorithm for the problem. Our SQ lower bound for Problem (1) is qualitatively matched by known learning algorithms for GMMs. Our lower bound for Problem (2) implies that the accuracy of the robust learning algorithm in~\cite{DiakonikolasKKLMS16} is essentially best possible among all polynomial-time SQ algorithms. Our SQ lower bounds are attained via a unified moment-matching technique that is useful in other contexts and may be of broader interest. Our technique yields nearly-tight lower bounds for a number of related unsupervised estimation problems. Specifically, for the problems of (3) robust covariance estimation in spectral norm, and (4) robust sparse mean estimation, we establish a quadratic {\em statistical--computational tradeoff} for SQ algorithms, matching known upper bounds. Finally, our technique can be used to obtain tight sample complexity lower bounds for high-dimensional {\em testing} problems. Specifically, for the classical problem of robustly {\em testing} an unknown mean (known covariance) Gaussian, our technique implies an information-theoretic sample lower bound that scales {\em linearly} in the dimension. Our sample lower bound matches the sample complexity of the corresponding robust {\em learning} problem and separates the sample complexity of robust testing from standard (non-robust) testing.

• The paper introduces super-polynomial lower bounds for SQ algorithms in high-dimensional Gaussian and Gaussian mixture estimation.
• It employs moment-matching techniques to reveal the computational complexity gap between sample and computational limits.
• The study highlights a computational separation between the agnostic and Huber’s contamination models, motivating new algorithmic approaches.

Statistical Query Lower Bounds for Robust Estimation of High-Dimensional Gaussians and Gaussian Mixtures

The paper presents a comprehensive paper on the computational limitations of Statistical Query (SQ) algorithms for robust estimation tasks in high-dimensional Gaussian settings. The authors provide significant insights into the inherent complexity of learning problems involving Gaussian distributions, specifically focusing on Gaussian Mixture Models (GMMs) and robust learning of single Gaussians. The results establish a discrepancy between the sample complexity required for information-theoretically optimal solutions and the computational complexity when constrained to SQ algorithms.

Main Contributions

This work introduces several key results by utilizing a generic lower bound construction based on moment-matching techniques. Specifically, the authors derive super-polynomial lower bounds on the complexity of any SQ algorithm for learning high-dimensional GMMs and robustly learning an unknown Gaussian distribution.

Gaussian Mixture Models

The paper proves an SQ lower bound of $n^{\Omega(k)}$ for learning GMMs, where $n$ is the dimensionality and $k$ is the number of mixture components. This bound applies even when the mixture components are nearly non-overlapping, highlighting the exponential complexity of such learning tasks using SQ algorithms. The paper establishes that current algorithms, which operate under strong separability assumptions or exhibit runtimes of $n^{\Omega(k)}$, are operating near the limits of what is feasible under the SQ model.

Robust Learning of High-Dimensional Gaussians

For the problem of robustly learning an unknown mean Gaussian, the authors present a super-polynomial SQ lower bound in terms of the total variation distance. The paper concludes that achieving the optimal error, $O(\epsilon)$, in the agnostic model comes at a computational cost that no polynomial-time SQ algorithm can match. Furthermore, they present a computational separation between the standard agnostic model and Huber’s contamination model, where a polynomial-time algorithm achieving $O(\epsilon)$ error exists.

Moreover, the paper extends these findings to the robustness estimation of a zero-mean Gaussian with an unknown covariance matrix, providing similar computational lower bounds for SQ algorithms.

Implications and Future Directions

The authors' results emphasize an intrinsic computational gap in robust high-dimensional Gaussian learning, challenging the efficiency of polynomial-time SQ algorithms in approaching the information-theoretic limits. These findings suggest that alternate models beyond SQ or complexity relaxations may be necessary to design effective algorithms for these tasks.

Conclusion

This paper's robust theoretical framework and rigorous lower bounds contribute significantly to our understanding of the complexity of learning within high-dimensional Gaussian settings. The results advocate for a careful consideration of computational limitations when designing learning algorithms, inviting future exploration in non-SQ domains to overcome these entrenched computational barriers. The implications extend broadly within the contexts of machine learning and statistics, especially in settings requiring robust, reliable inference under adversarial conditions. These insights ought to drive the development of new algorithmic strategies that can effectively balance statistical efficiency with computational feasibility.