Fast Algorithms for Robust PCA via Gradient Descent (1605.07784v2)

Abstract: We consider the problem of Robust PCA in the fully and partially observed settings. Without corruptions, this is the well-known matrix completion problem. From a statistical standpoint this problem has been recently well-studied, and conditions on when recovery is possible (how many observations do we need, how many corruptions can we tolerate) via polynomial-time algorithms is by now understood. This paper presents and analyzes a non-convex optimization approach that greatly reduces the computational complexity of the above problems, compared to the best available algorithms. In particular, in the fully observed case, with $r$ denoting rank and $d$ dimension, we reduce the complexity from $\mathcal{O}(r^2d^2\log(1/\varepsilon))$ to $\mathcal{O}(rd^2\log(1/\varepsilon))$ -- a big savings when the rank is big. For the partially observed case, we show the complexity of our algorithm is no more than $\mathcal{O}(r^4d \log d \log(1/\varepsilon))$. Not only is this the best-known run-time for a provable algorithm under partial observation, but in the setting where $r$ is small compared to $d$, it also allows for near-linear-in-$d$ run-time that can be exploited in the fully-observed case as well, by simply running our algorithm on a subset of the observations.

• The paper introduces a projected gradient descent method in a factorized space to iteratively recover low-rank and sparse components.
• It achieves substantial runtime improvements by reducing complexity from r²d²log(1/ε) to rd²log(1/ε) for fully observed data and extends to partial observations.
• Empirical validations on synthetic and real datasets, including video frame decomposition, demonstrate its practical robustness and efficiency.

Fast Algorithms for Robust PCA via Gradient Descent

The paper "Fast Algorithms for Robust PCA via Gradient Descent" focuses on developing efficient algorithms for solving the robust principal component analysis (RPCA) problem. RPCA extends the conventional PCA by addressing scenarios involving data matrices that are both incomplete and corrupted with outliers. Traditional PCA approaches, typically employed through singular value decomposition (SVD), are inadequate when faced with such dirty data, primarily due to their sensitivity to outliers and inability to handle missing entries.

The authors propose a novel approach using non-convex optimization that significantly reduces computational complexity compared to existing methodologies. The paper introduces algorithms that apply gradient descent on a factorized space, enabling both robustness and improved computational efficiency. In particular, this work provides a solution with significant runtime reductions from the previous best-known results for fully observed datasets and sets a new benchmark for partially observed datasets.

Key Contributions

1. Algorithmic Innovation: The primary innovation is a projected gradient descent method conducted in the factorized space of low-rank matrices. In both the fully and partially observed settings, the algorithm iteratively refines estimates of the low-rank and sparse components. The technique leverages a novel sparse estimator that targets corruptions dispersed across the matrix, harnessing the matrix’s incoherence properties.
2. Runtime Improvements: For fully observed data, the proposed algorithm achieves a complexity of $(rd^2\log(1/\varepsilon))$, where $r$ is the rank, $d$ is the matrix dimension, and $\varepsilon$ is the error tolerance. This marks an improvement from $(r^2d^2\log(1/\varepsilon))$, representing a substantial reduction in computational cost, especially when the rank is large.
3. Robustness to Missing Data: The extension to the partially observed setting generalizes matrix completion and robust PCA. Here, the algorithms operate under sub-sampling, obtaining near-linear-in-dimension runtime by using only a subset of observations. The sample complexity for achieving robust PCA is established at $(\mu^2r^2d\log d)$, matching matrix completion bounds for positive semidefinite matrices.
4. Theoretical Guarantees: The paper provides rigorous theoretical analysis demonstrating linear convergence rates under specific initialization conditions. It further elucidates error bounds for both fully and partially observed cases, indicating the robustness of the proposed methods against a fraction of corruptions.
5. Empirical Validation: Several experiments on synthetic and real datasets, including foreground-background separation in video data, substantiate the efficacy of the proposed algorithms. The results illustrate not only the computational advantages but also the practical utility in complex real-world scenarios.

Implications and Future Directions

The implications of this work are notable both theoretically and practically:

• Theoretical Significance: The research advances the understanding of non-convex optimization for matrix factorization problems and bridges the gap between efficacy and efficiency in robust data analysis settings. The introduction of a gradient-based method underlines potential pathways for further explorations into other non-convex formulations.
• Practical Applications: The proposed methodology is well-suited for large-scale data problems in various fields such as computer vision, bioinformatics, and finance, where data is frequently subject to corruption and incomplete entries. The demonstrated effectiveness in video frame decomposition showcases its applicability in dynamic and large-scale environments.
• Future Research: Potential future developments could explore further improvements in sample complexity and runtime, particularly focusing on datasets with more structural complexities or less favorable incoherence properties. Additionally, extending these techniques to other low-rank approximation problems while maintaining computational efficiency remains a promising avenue.

In conclusion, this paper presents robust and computationally efficient solutions for robust PCA problems and sets a foundation for future work in handling corrupted and incomplete data matrices via non-convex approaches.