Mixed Low-Rank & Sparse Minimization

• Mixed low-rank and sparse minimization is a framework that recovers matrices exhibiting global low rank and local sparse patterns from incomplete or noisy data.
• It integrates convex surrogates like the nuclear norm and group norms, along with nonconvex penalties, to ensure stable recovery under RIP conditions and optimal sample complexity.
• Efficient algorithmic approaches such as proximal splitting, IRLS, and two-stage convex procedures are employed, enhancing applications in compressed sensing, multichannel signal processing, and high-dimensional statistics.

Mixed low-rank and sparse minimization is the class of mathematical and algorithmic frameworks designed to recover or describe matrices (or tensors) that exhibit both low-rank and (group) sparse structures, frequently under incomplete or noisy observations. These models arise in compressed sensing, machine learning, multichannel signal processing, and high-dimensional statistics, where data are simultaneously structured globally (low rank) and locally (sparse in some rows, columns, or entries). The core principle of mixed minimization is to integrate convex or nonconvex surrogates for rank and sparsity into a unified recovery or estimation program, typically leveraging tools such as the nuclear norm, group (e.g., $\ell_{2,1}$) norms, and advanced iterative algorithms for optimization.

1. Mathematical Foundations and Model Formulation

Mixed low-rank and sparse minimization seeks to solve inverse problems where the unknown $X_0 \in \mathbb{R}^{n_1 \times n_2}$ is both low-rank and (joint-)sparse. The canonical observation model is: $$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$ where $\mathcal{A}$ is a known linear measurement operator and $z$ is noise (Golbabaee et al., 2012).

To exploit both structures, one solves the convex program: $$\min_{X}\; \|X\|_* + \lambda\,\|X\|_{2,1}\qquad\text{s.t. } \|\mathcal{A}(X)-y\|_2 \leq \varepsilon,$$ where $\|X\|_*$ is the nuclear norm (sum of singular values, convex envelope of rank) and $\|X\|_{2,1}$ is the sum of the $\ell_2$-norms of the rows (joint/row sparsity surrogate).

Similar constructions arise in alternate application contexts. Low-rank representation (LRR) models regularize with nuclear and group norms under affine constraints (Lu et al., 2014). When the measurement process is nested or structured, specialized two-stage convex algorithms are deployed (e.g., nuclear norm estimation followed by $\ell_{2,1}$ refinement) (Bahmani et al., 2015).

2. Key Theoretical Guarantees: Restricted Isometry and Sample Complexity

Central to the recovery guarantees in mixed low-rank and sparse minimization is an adaptation of the Restricted Isometry Property (RIP). A linear operator $X_0 \in \mathbb{R}^{n_1 \times n_2}$0 is said to satisfy the $X_0 \in \mathbb{R}^{n_1 \times n_2}$1-RIP with constant $X_0 \in \mathbb{R}^{n_1 \times n_2}$2 if

$X_0 \in \mathbb{R}^{n_1 \times n_2}$3

for all $X_0 \in \mathbb{R}^{n_1 \times n_2}$4 of rank $X_0 \in \mathbb{R}^{n_1 \times n_2}$5 and with at most $X_0 \in \mathbb{R}^{n_1 \times n_2}$6 nonzero rows (Golbabaee et al., 2012).

Main results show:

• Stable recovery: If $X_0 \in \mathbb{R}^{n_1 \times n_2}$7 satisfies $X_0 \in \mathbb{R}^{n_1 \times n_2}$8-RIP with $X_0 \in \mathbb{R}^{n_1 \times n_2}$9,

$$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$0

where $$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$1 is the best rank-$$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$2, $$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$3-row sparse approximation.

• Sample complexity: For subgaussian $$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$4, RIP holds with high probability when

$$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$5

which is near-optimal and interpolates between pure sparsity and pure low-rank regimes.

For nested measurement scenarios, the minimax optimal recovery rate under Gaussian or subgaussian operators is also established up to polylogarithmic factors, and cannot be improved by any estimator (Bahmani et al., 2015).

3. Algorithmic Approaches

The implementation of mixed low-rank and sparse minimization draws on several algorithmic frameworks:

Proximal Splitting and Parallel Proximal Algorithms:

The convex optimization problem is recast as the minimization of the sum of separable convex functions, allowing for block-wise updates: $$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$6 Blockwise (PPXA) schemes iterate updates using proximity operators:

• Row-wise $$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$7 shrinkage for joint sparsity,
• Singular value soft-thresholding for the nuclear norm,
• Euclidean ball projection for fidelity constraints.

This yields convergence at rate $$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$8 in objective residuals per iteration.

Iteratively Reweighted Least Squares (IRLS):

For quasi-norms (Schatten-$$y = \mathcal{A}(X_0) + z,\quad \|z\|_2 \leq \varepsilon,$$9 and $\mathcal{A}$0), IRLS implements a majorization-minimization scheme with smoothed surrogates: $\mathcal{A}$1 Alternating solution of Sylvester-type linear equations and reweighting enables efficient optimization, with convergence to stationary points and global optima when $\mathcal{A}$2 (Lu et al., 2014).

Two-Stage Convex Procedures:

With nested measurements, one first solves a nuclear norm minimization to estimate a compressed target, then applies $\mathcal{A}$3 minimization for sparsity (Bahmani et al., 2015). Each stage uses standard convex solvers (proximal or thresholding-based).

4. Nonconvex and Discrete Formulations

Nonconvex extensions and discrete formulations are developed for settings where convex surrogates are not optimal:

• Nonconvex penalties such as SCAD, MCP, capped-$\mathcal{A}$4, or direct $\mathcal{A}$5 approaches remove the estimator bias inherent in convex relaxations. Algorithms such as alternating proximal gradient descent and ADMM with dual momentum achieve improved estimation accuracy, maintain convergence, and often yield global minimizers in practical regimes (Sagan et al., 2021, Wu et al., 2019, Brbić et al., 2018).
• Discrete optimization approaches explicitly model the rank and sparsity constraints using indicator variables and projectors, leading to alternating minimization heuristics and branch-and-bound strategies with tight semidefinite programming relaxations capable of certifying near-optimality (Bertsimas et al., 2021).

5. Representative Applications

Mixed low-rank and sparse minimization has broad applications:

• Compressed Sensing and Multichannel Acquisition: Recovery of high-dimensional signals with joint-sparse and low-dimensional structure, such as in sensor networks and hyperspectral imaging (Golbabaee et al., 2012).
• Multivariate Regression and PCA: Sparse PCA and sparse reduced-rank regression settings, where simultaneous dimension reduction and variable selection are required (Ma et al., 2014, Lu et al., 2014).
• Blind Deconvolution and Phase Retrieval: Reconstruction of signals from convolutional/masked observations, exploiting the structure in lifted matrix form (Bahmani et al., 2015).
• Subspace Clustering and Image/Signal Denoising: Clustering high-dimensional data by exploiting global subspace structure and local outlier sparsity; robust background-foreground decomposition in video surveillance (Lu et al., 2014, Brbić et al., 2018).
• Distributed Estimation in Networks: Efficient algorithms for network-scale anomaly detection, robust imputation, and delay tomography using distributed ADMM schemes that respect privacy and scalability constraints (Mardani et al., 2012).

6. Empirical Performance and Phase Transitions

Empirical investigations confirm sharp phase transitions in the recovery of mixed structured matrices. For random instances (fixed rank $\mathcal{A}$6 and row sparsity $\mathcal{A}$7), there is a well-defined sampling threshold $\mathcal{A}$8 below which recovery fails, and above which recovery is nearly exact (normalized error $\mathcal{A}$9). These observed transitions closely align with the theoretically predicted sample complexities (Golbabaee et al., 2012, Bahmani et al., 2015).

Comparisons to pure sparse or pure low-rank recovery confirm that mixed-structure exploitation dramatically reduces the necessary number of measurements, especially when both $z$0 and $z$1 are small relative to ambient dimensions. The algorithms are robust to noise and perform consistently under various measurement ensembles.

7. Extensions and Open Directions

• The above frameworks extend naturally to tensor data (using nuclear and sparsity norms on tensor unfoldings), multilevel group sparsity, and structure in overcomplete dictionaries (Cohen, 2021).
• Nonconvex models admit generalization to robust PCA, LRR, and more, achieving faster convergence and reduced estimation bias without precise a priori knowledge of model parameters (Sagan et al., 2021, Wu et al., 2019).
• Exact and near-exact support and rank estimation are enabled via adaptive parameter selection and thresholding strategies, with theoretical justifications under probabilistic models.

Remaining challenges include the design of measurement operators with optimal RIP for complex hierarchies, scaling algorithms to massive data, and optimal selection of regularization parameters in fully unsupervised scenarios. Recent work also explores identifiability, minimax optimality, and the integration with nonlinear and non-Gaussian models.

---

References:

• (Golbabaee et al., 2012) Compressed Sensing of Simultaneous Low-Rank and Joint-Sparse Matrices
• (Lu et al., 2014) Smoothed Low Rank and Sparse Matrix Recovery by IRLS Minimization
• (Bahmani et al., 2015) Near-Optimal Estimation of Simultaneously Sparse and Low-Rank Matrices
• (Sagan et al., 2021) Provable Low Rank Plus Sparse Matrix Separation Via Nonconvex Regularizers
• (Bertsimas et al., 2021) Sparse Plus Low Rank Matrix Decomposition: A Discrete Optimization Approach
• (Ma et al., 2014) Adaptive Estimation in Two-way Sparse Reduced-rank Regression
• (Mardani et al., 2012) In-network Sparsity-regularized Rank Minimization: Algorithms and Applications
• (Brbić et al., 2018) $z$2-Motivated Low-Rank Sparse Subspace Clustering