Sparse-plus-Low-Rank Decomposition

• Sparse-plus-low-rank decomposition is a framework that expresses data as the sum of a low-rank matrix and a sparse matrix, facilitating dimensionality reduction and anomaly detection.
• It employs convex relaxations, nonconvex optimization, and Bayesian methods to accurately recover underlying structures even in high-dimensional settings.
• Applications span covariance estimation, graphical modeling, LLM compression, and signal processing, highlighting its broad impact on robust data analysis.

Sparse-plus-low-rank decomposition refers to the class of mathematical and computational strategies for expressing a matrix (or related object) as the sum of a component with low rank and a component that is sparse. This structure arises naturally in diverse fields such as statistical learning, signal processing, time series analysis, graphical modeling, LLM compression, and scientific computing, forming the foundation for robust dimensionality reduction, anomaly detection, model compression, and separation of structured signals from outliers or correlated noise.

1. Mathematical Formulations and Theoretical Properties

Let $D \in \mathbb{R}^{m \times n}$ be a data matrix. The generic sparse-plus-low-rank (S+LR) model posits $D = L + S$, where $L$ is a low-rank matrix and $S$ is sparse (either in the standard basis or some transformed domain). Formally, archetypal formulations include:

• Principal Component Pursuit (convex surrogate for rank/Sparsity):

$$\min_{L, S} \ \|L\|_* + \lambda \|S\|_1 \quad \text{subject to} \quad D = L + S$$

where $\|L\|_*$ is the nuclear norm and $\|S\|_1$ is the elementwise $\ell_1$-norm (Rahmani et al., 2015, Baete et al., 2018, Leibovich et al., 2019).

• Exact non-convex constraints:

$$\min_{L, S} \ \operatorname{rank}(L) + \lambda \|S\|_0 \quad \text{subject to} \quad D = L + S$$

Non-convex relaxations using fraction functions or hard constraints on sparsity and rank have been employed to improve fidelity beyond convex surrogates (Cui et al., 2018, Bertsimas et al., 2021).

For covariance estimation, the S+LR idea is generalized to $\Sigma = L + S$, with $D = L + S$0 positive semidefinite and low rank, and $D = L + S$1 sparse/positive-definite, relevant for factor models and graphical models (1310.4195, 1901.10613, Baes et al., 2019).

2. Bayesian and Robust Statistical Approaches

Bayesian formulations extend the S+LR framework by placing hierarchical priors on both $D = L + S$2 and $D = L + S$3. A representative model (1310.4195):

• $D = L + S$4, with binary factor-selection vector $D = L + S$5, allowing the number of active factors (rank) to be inferred.
• $D = L + S$6 draws from a Bayesian lasso prior or graphical structure prior, supporting high-dimensional covariance modeling with uncertainty quantification and automatic model selection.

Robust identification for graphical models introduces uncertainty sets for the sample covariance $D = L + S$7, enforcing e.g., Kullback-Leibler divergence constraints: $D = L + S$8 (1901.10613).

Simulation and theoretical studies show Bayesian S+LR models recover both rank and sparsity patterns with high accuracy when the sample size is sufficient, and robust approaches maintain consistent recovery in the presence of perturbations or estimation noise in the empirical covariance.

3. Optimization Algorithms and Computational Strategies

A spectrum of algorithms address S+LR decomposition:

• Convex Relaxations: Proximal/alternating minimization, augmented Lagrangian or ADMM methods for nuclear- and $D = L + S$9-norm surrogates (Rahmani et al., 2015, 1310.4195). These are supported by strong theoretical guarantees on recovery under incoherence and random support assumptions.
• Nonconvex and Discrete Approach: Alternating minimization with hard constraints on rank and support, utilizing hard-thresholding or proximal maps for non-convex fraction penalties (Bertsimas et al., 2021, Cui et al., 2018). Algorithmic steps often admit closed-form (or nearly closed-form) solutions:
  • SVD-based updates for the low-rank component
  • Entrywise thresholding or combinatorial optimization for sparsity
• Bayesian MCMC: Closed-form Gibbs updates for factor loadings and indicator variables, Metropolis-Hastings for elements of $L$0 where conditional distributions do not admit direct sampling (1310.4195).
• Scalable Subspace Methods: For large-scale data, two-stage subspace-pursuit algorithms use sampled column/row sketches, dramatically reducing computational and memory burdens (Rahmani et al., 2015).
• Specialized Algorithms:
  • Recursive sparse LU factoring with hierarchical low-rank approximations for PDE matrices (Xuanru et al., 2024).
  • Structured sparse dictionaries for signal detection (e.g., exoplanet detection) (Vary et al., 2023, Gonzalez et al., 2016).

4. Applications across Domains

LLM Compression

Recent methods for LLMs decompose each transformer's weight matrix $L$1 as $L$2 or $L$3, with semi-structured sparsity (N:M patterns) and low-rank corrections. State-of-the-art frameworks include:

• HASSLE-free: Alternating exact minimizations of layer-wise Frobenius error in the activation domain, blending hard sparsity and hard rank constraints, providing significantly improved perplexity and zero-shot evaluation scores over previous relaxations (Makni et al., 2 Feb 2025).
• 3BASiL: A 3-block ADMM with provable convergence, followed by transformer-matching refinement, yielding substantially reduced test perplexity gap and faster compression runtimes (Makni et al., 2 Mar 2026).

A comparison table summarizes LLM S+LR benchmarks:

Method	Pattern+Rank	WT2 Perplexity	Runtime (A100, LLaMA 8B)
Hf-ALPS	2:4+64	13.79	15.7 hr
3BASiL-TM	2:4+64	11.79	7.0 hr
HASSLE-free	2:4+64	12.66	—
Dense	—	7.81	Baseline

Both 3BASiL-TM and HASSLE-free achieve 30–40% reductions in perplexity gaps and up to 2.5× speed-ups compared to previous S+LR methods (Makni et al., 2 Feb 2025, Makni et al., 2 Mar 2026).

Graphical Models and Covariance Estimation

S+LR structure is central in contemporary models for high-dimensional covariance estimation and graphical model inference, enabling consistent simultaneous estimation of sparse graphical structure (conditional independence) and low-rank latent variable effects. The Bayesian and robust optimization advances significantly improve model selection and interpretability for high-dimensional data (1310.4195, 1901.10613).

Dynamic Systems and Time Series

In autoregressive and graphical time-series modeling, S+LR decompositions allow separation of manifest sparse interactions and latent (smooth or dynamic) dependencies, leading to convex or non-convex identification programs for both the AR and latent components (Liégeois et al., 2015, You et al., 2023). For dynamical latent-variable graphical AR models, trace and nuclear norm relaxations combined with block-Toeplitz lifting make joint recovery computationally tractable (You et al., 2023).

Imaging and Signal Processing

In robust statistical analysis (e.g., diffusion MRI ODFs, SAR), S+LR enables decomposition of structured signals from outliers or anomalous activity, leading to increased statistical power in group analyses or sharper detection (e.g., moving target detection in SAR, exoplanet imaging) (Baete et al., 2018, Leibovich et al., 2019, Vary et al., 2023, Gonzalez et al., 2016).

In compressed sensing MRI and video, S+LR and its deep unrolled variants (e.g., L+S-Net) outperform traditional models by learning layerwise singular value thresholding and proximal mappings, supporting very high acceleration factors (Huang et al., 2020, Ting et al., 2024).

5. Algorithmic Extensions and Innovations

Recent developments expand the S+LR paradigm to more sophisticated structural priors:

• Manifold and Dictionary Structure: Structured dictionaries capture physically meaningful transformations (e.g., planet trajectories in direct imaging) (Vary et al., 2023).
• Graphical Priors: Hyper-inverse Wishart and graphical lasso priors model structured conditional independences in the sparse component (1310.4195).
• Neural Network Parametrization: Representing low-rank factors as neural network outputs enables scalable, parameter-efficient, and convergent low-rank + sparse decompositions, with convergence rates polynomial in problem dimensions (Baes et al., 2019).
• Smoothness Augmentation: Additional temporal/spatial smoothness regularization in the low-rank part improves separation and reconstruction fidelity in dynamic settings (Ting et al., 2024).

6. Comparative Performance and Limitations

Simulation and empirical results across settings consistently show that S+LR decompositions:

• Achieve high accuracy in recovery of both low-rank and sparse parts, outperforming classic PCA or pure sparsity/low-rank-only models.
• For Bayesian and robust S+LR, rank and sparsity recovery rates exceed 90% in many practical scenarios, with superior estimation error compared to competing methods (e.g., LOREC, sample covariance) (1310.4195).
• Accelerate large-scale problems via adaptive sketching, randomized SVD, or parameter-efficient deep unrolling while maintaining theoretical guarantees or certifiable optimum gaps (Rahmani et al., 2015, Huang et al., 2020, Baete et al., 2018).

However, nonconvex/discrete approaches may lack full global convergence guarantees, can be sensitive to initialization and regularization parameters, and for small instances suffer from computational bottlenecks in semidefinite relaxations (Cui et al., 2018, Bertsimas et al., 2021). In Bayesian settings, reliable model selection typically requires sample sizes commensurate with problem dimensionality.

7. Directions for Future Research

Open research frontiers include:

• Extending S+LR models to hierarchical, tensor, or multiscale decompositions in multi-way data.
• Tightening guarantees and scalability for nonconvex and neural-parametric methods.
• Developing broader robust and uncertainty-aware S+LR inference under high-noise or nonideal sampling.
• Elucidating precise theoretical limits of S+LR identification in online, adaptive, or distributed environments.

Sparse-plus-low-rank decomposition thus remains a foundational tool for interpretable, scalable, and robust modeling in modern high-dimensional data analysis and computational science.