Robust Low-Rank + Sparse Tensor Decomposition

• The paper introduces a nonconvex alternating algorithm using a regularized tensor power method to separate a tensor into low-rank and block-sparse parts.
• It provides rigorous recovery guarantees under incoherence and block-sparsity conditions, outperforming traditional matrix RPCA methods.
• Empirical results in video analysis and latent variable estimation demonstrate improved accuracy and speed compared to existing approaches.

Regularized robust low-rank + sparse tensor decomposition refers to a class of methods for separating a high-dimensional tensor into a sum of a low-rank tensor and a sparse tensor, under explicit regularization schemes. These techniques generalize robust principal component analysis (RPCA) to higher-order arrays (tensors), exploiting structure unique to multiway data for improved recovery performance under structured noise, especially block-sparse or gross corruptions. Regularized formulations leverage precise algebraic penalties, alternated updates, and convergence analyses to ensure practical tractability and statistical guarantees across diverse applications in computer vision, data mining, signal processing, and machine learning.

1. Model Formulation and Algorithmic Framework

The task is to decompose an observed tensor $T \in \mathbb{R}^{n \times n \times n}$ (or more generally, an order-$d$ tensor) as $T = L^* + S^*$, where $L^*$ is low-rank and $S^*$ is sparse. In the setting considered in (Anandkumar et al., 2015), $L^*$ is assumed to admit a canonical polyadic (CP) decomposition,

$$L^* = \sum_{i=1}^r \sigma_i^* u_i \otimes u_i \otimes u_i$$

with orthonormal $u_i \in \mathbb{R}^n$ and positive scalars $\sigma_i^* > 0$. The sparse component $S^*$ is block-sparse; outlier entries are not merely scattered but organized in contiguous blocks or across entire slices/fibers.

The core algorithm (“RTD”) alternates two principal steps:

• Low-rank update: Solve a regularized eigenvalue problem to extract the leading CP components of $T - S^{(t)}$ using a gradient ascent variant of the tensor power method. This uses the regularized objective,

$f(v) = T(v, v, v) - \frac{3}{4}\lambda \|v\|_4^4$

with update

$$v \leftarrow v + \frac{1}{4\lambda(1 + \lambda/\sqrt{n})}\left[T(I, v, v) - \lambda\|v\|^2 v\right].$$

• Sparse update: Apply hard thresholding to the residual. For a current low-rank estimate $L^{(t+1)}$, update $S$ by

$S^{(t+1)} = \mathrm{HT}_\zeta(T - L^{(t+1)}),$

where $\zeta$ is adaptively tuned based on the estimated spectrum.

This two-step process is performed sequentially for each component, “peeling off” rank-1 terms until all $r$ CP components are extracted. The overall procedure is nonconvex but enjoys theory-backed convergence guarantees under suitable regularity and sparsity conditions.

2. Theoretical Guarantees and Regularization Properties

A distinguished feature of this approach is that it admits rigorous global convergence analysis, based on two key conditions:

• Incoherence of the low-rank tensor: Each factor $u_i$ satisfies a uniform bound $\|u_i\|_\infty \leq \mu/\sqrt{n}$.
• Block-sparsity of the noise: The sparse tensor $S^*$ is structured, with strong limits on the number of nonzeros per fiber and overlap parameter $\eta$.

The analysis proceeds via careful characterization of the alternating scheme:

• The low-rank step achieves linear convergence within a neighborhood of each true eigenvector. Specifically, Lemma 3.1 states that the regularized objective $f(v)$ is locally strongly concave and smooth, ensuring that initialized within a spectral ball, the iterates rapidly contract to the eigenpair.
• The hard thresholding step preserves support recovery, ensuring that false positives in estimating the sparse support do not propagate across iterations.

Given the incoherence and block-sparsity assumptions, Theorem 3.1 and related results guarantee (with explicit error rates) that the recovered factors $(\hat{u}_i, \hat{\lambda}_i)$ satisfy

$$\|\hat{u}_i - u_i\|_\infty \leq \frac{\delta}{\mu^2 r \sqrt{n} \sigma^*_{\min}},\qquad |\hat{\lambda}_i - \sigma_i^*| \leq \delta$$

with recovery bounds on $L$ and $S$ in the Frobenius and sup norms, respectively.

3. Comparison to Matrix-Based Robust PCA Methods

A critical contribution of (Anandkumar et al., 2015) is the formal comparison of tensor versus matrix robust PCA in the presence of structured perturbations:

• Flattening approaches (unfolding the tensor into matrices) and per-slice matrix RPCA approaches disregard the multilinear structure, thereby underutilizing algebraic regularities.
• When the sparse noise is block-structured, tensor methods (using CP decomposition) provably accommodate far more corruption than is possible for any matrix-based approach. For example, in the rank-1 case, tensor methods tolerate $O(n^{17/12})$ nonzeros per fiber, compared to $O(n)$ per row/column for matrix methods.
• With small normalized block overlap $\eta$, the allowable block size for tensor methods is asymptotically larger—this structural advantage is unattainable with per-mode decompositions.

Empirically, for both synthetic and real data, tensor methods were "2–3 times more accurate" and "8–14 times faster" in recovery than strong matrix RPCA baselines, especially as block size and rank increase.

4. Algorithmic Details and Implementation Considerations

The algorithm’s practical tractability is due to several methodological refinements:

• Gradient ascent variant of the tensor power method with regularization enables robust extraction of CP components even when the input is heavily perturbed by sparse block noise.
• Hard thresholding updates for the sparse part are aggressive, and the threshold schedule ensures decay of false positives. The threshold is updated adaptively as $$\zeta_{t+1} = \beta\left[\sigma_{l+1}(T-S^{(t+1)}) + (1/2)^t \sigma_{l}(T-S^{(t+1)})\right]$$, with $\beta$ set based on the incoherence parameter $\mu$ and rank $r$.
• Stagewise estimation: By sequentially deflating the tensor and recomputing the top CP component, the method avoids the need to solve a prohibitively large eigenproblem in a single shot.
• Initialization: Local convergence is guaranteed within a spectral ball around the true eigenvector; this is achieved by SVD-based or randomized initialization.
• Stopping criterion: The recovery error is checked against a prescribed tolerance $\delta$ in both low-rank and sparse parts, ensuring that excess computation is avoided once the desired accuracy is reached.

5. Practical Applications and Empirical Validation

Robust regularized tensor decomposition is especially well-suited to scenarios involving block or group sparse corruption:

• Foreground detection in video: Each video is represented as a third-order tensor (two spatial and one temporal mode). Foreground objects correspond to structured sparse perturbations, while the relatively low-rank background is separated via CP decomposition. On the Curtain video dataset, the proposed method yields visually superior foreground masks and is about 10% faster than competing matrix-based RPCA.
• Latent variable model estimation: The method applies to robustly estimate moments and structure in latent variable models (e.g., mixtures of product distributions, Bayesian networks) when moment tensors are contaminated with gross errors.
• Extensive simulation studies confirm the theoretical findings: as the block size, overlap, or rank increases, tensor methods continue to achieve accurate separation, whereas matrix methods degrade earlier.

6. Limitations, Extensions, and Theoretical Implications

Key theoretical insights from this work include:

• The recovery guarantee holds for nonconvex CP-decomposition-based methods under block-sparse corruption, given incoherence and bounded overlap.
• The approach does not address settings with generic, unstructured sparse noise as efficiently; its power is maximized for block or group-sparse patterns.
• The global convergence result is significant, as CP-rank regularized optimization is generally NP-hard; leveraging alternating minimization and careful regularization, recovery is possible for a larger class of structured perturbations.

Natural extensions involve generalizing to higher-order tensors, integrating different forms of regularization (e.g., enforcing additional smoothness or group structure), and combining tensor methods with probabilistic graphical models for high-dimensional robust inference.

---

In sum, regularized robust low-rank + sparse tensor decomposition as advanced in (Anandkumar et al., 2015) formalizes a principled approach to separating structured low-rank and block-sparse components from tensors, with a concrete nonconvex alternating framework, algorithmic refinements, and precise recovery guarantees. Its empirical and theoretical advantages over matrix-based robust PCA approaches are most pronounced when the latent corruption is block-structured or exhibits global multilinear dependencies. The method is widely applicable to video analysis, robust latent variable estimation, and any context where high-dimensional tensor data are corrupted by nontrivial sparse structure.