Sparse Regression Algorithms

• Sparse regression algorithms are computational methods that estimate regression coefficients with few nonzeros, enhancing interpretability in high-dimensional data.
• They span a range of approaches including convex relaxations like Lasso, greedy selections such as OMP, nonconvex penalties, and exact combinatorial methods.
• Recent advances improve support recovery, reduce false discoveries, and address challenges like correlated designs and structured sparsity in applications from genomics to finance.

A sparse regression algorithm is any computational method for estimating regression coefficients subject to a cardinality (sparsity) constraint, i.e., seeking parameter vectors (or matrices) with few nonzeros. These algorithms are fundamental in high-dimensional statistics and signal processing, where model interpretability, noise reduction, and the challenge of limited sample regimes demand methods that provide accurate parameter and support recovery using parsimonious representations. Sparse regression algorithms encompass a wide spectrum, including convex relaxations (e.g., Lasso), greedy selection (e.g., OMP), nonconvex penalties, combinatorial optimization, group-based sparsity models, robust regression, and recent hybrid and learning-based approaches. Many modern frameworks address complexities such as correlations in the design, structured sparsity, robust estimation, or multi-task settings.

1. Classifications and Basic Principles

Sparse regression algorithms can be systematically categorized based on their underlying optimization approach and the structure they exploit:

Algorithm class	Typical regularizer/constraint	Key mechanism
Convex relaxations	ℓ₁-norm (Lasso), ℓ₁,∞-norm, etc.	Convex optimization
Greedy/Sequential selection	Implicit ℓ₀ (support sets)	Stepwise inclusion/removal
Combinatorial/discrete	Explicit ℓ₀ (cardinality)	Integer/binary programming
Nonconvex penalty methods	MCP, SCAD, truncated ℓ₁, etc.	Nonconvex optimization
Group/structured sparsity	ℓ₁,∞, ℓ₁,₂, other mixed norms	Structured penalty/design
Robust or hybrid approaches	Extended loss and/or penalties	Block coordinate descent, alternating, etc.

Traditional methods include Lasso (ℓ₁ regularization), group Lasso, and their quantile regression and multi-task analogues (e.g., structured penalties for joint support recovery (Nassiri et al., 2013)). Greedy methods range from OMP/OLS (Hashemi et al., 2016) to specialized forward-backward procedures for group-sparse estimation (Jalali et al., 2012). Exact support recovery can, in principle, be achieved by combinatorial optimization (Bertsimas et al., 2017), but such methods are traditionally intractable for large p.

To capture real-world sparsity patterns, advanced frameworks may include outlier detection (Katayama et al., 2015, Liu et al., 2018), handling missing data (Ganti et al., 2015), adaptation to design correlations (Ghorbani et al., 2015, Kelner et al., 2023), or flexible structural constraints (entropy-based, grouping, etc.) (Srivastava et al., 2023).

2. Mathematical Formulations

The canonical sparse regression problem seeks

$$\min_{w \in \mathbb{R}^p} \frac{1}{2n}\|y - Xw\|^2_2 \quad \text{subject to} \quad \|w\|_0 \leq k$$

which is nonconvex and combinatorial due to the ℓ₀-norm. Standard variants include:

• Constraint or penalization ($\lambda\|w\|_0$ or mixed norms for group sparsity (Nassiri et al., 2013)).
• Multi-task regression: parameterized as a $p \times r$ matrix β, with both singleton (column-specific) and row (shared feature) sparsity (Jalali et al., 2012).
• Structured regression: replacing ℓ₁ with group penalties, e.g., ℓ₁,∞ promotes entire group selection (Nassiri et al., 2013).
• Combinatorial reformulation via support selection variables s ∈ {0,1}^p, as in the binary convex programming approach (Bertsimas et al., 2017):

$$\min_{s \in \{0,1\}^p,\, \sum s_j \le k}\; c(s) = \frac{1}{2} Y^T (I_n + \gamma\sum_j s_j K_j )^{-1} Y$$

with K_j = X_j X_j^T.

Nonconvex approaches replace ℓ₁ with SCAD, MCP, or truncated ℓ₁; robust models augment the regression with an explicit outlier vector (Katayama et al., 2015), and multi-task extensions impose constraints over matrix rows.

Recent algorithmic frameworks leverage alternative geometric or probabilistic representations: e.g., entropy-based relaxations using column stochastic binary matrices for support selection (Srivastava et al., 2023), or approximate nearest neighbor data structures for search (Price et al., 2022).

3. Algorithmic Methodologies

Greedy and Forward-Backward Procedures

A class of algorithms iteratively expand and contract the support. The forward-backward greedy algorithm for multi-task regression (Jalali et al., 2012) selects at each step either a singleton (β_{ij}) or a whole feature (entire row of β), based on maximal loss reduction reward, and performs backward "pruning" if an included object no longer contributes sufficiently. The selection between singleton and row is balanced via a group-sparsity weight w.

Convex and Nonconvex Relaxations

Convex relaxations such as the Lasso and group Lasso replace the intractable ℓ₀ setup with ℓ₁ (or group) penalties. These methods are efficient, but support recovery is contingent on strong assumptions (restricted eigenvalue, incoherence); they also introduce solution bias.

Nonconvex penalties (MCP, SCAD, truncated ℓ₁) and sequential convex relaxation (Bi et al., 2 Nov 2024) can improve support recovery and reduce estimation bias, often relying on local search, adaptive updating of the penalty sets, or homotopy in the penalization function.

Exact and Cutting-Plane Algorithms

Exact approaches reformulate the problem into mixed-integer or binary convex programs (Bertsimas et al., 2017, Bertsimas et al., 2019), solved via outer approximation or cutting-plane algorithms. These methods, once restricted to small p, have been demonstrated to scale to n, p ~10^5 via kernel-based dual formulations and lazy constraint addition; they permit observation of phase transitions in statistical and computational complexity.

Adaptive and Hybrid Algorithms

Advanced methods adapt the feature representation or regularization dynamically, especially in the presence of high correlation or sparse dependencies among the covariates (Ghorbani et al., 2015, Kelner et al., 2023). Feature adaptation frameworks first detect problematic coordinates (using spectral projection or iterative peeling) and augment the dictionary (basis) or adjust regularization, thereby ensuring optimal sample complexity and robust recovery.

Structured and Flexible Constraint Modeling

Entropy-based frameworks (Srivastava et al., 2023) decompose the sparse vector into a binary selection matrix and coefficient vector, assign a probabilistic model over feasible supports, and perform constrained optimization via homotopy to the nonconvex cost function, allowing explicit incorporation of practical feature constraints (correlation, grouping).

4. Theoretical Guarantees and Null Space Properties

Sparse regression algorithms can be analyzed via various properties of the measurement or design matrix:

• Restricted Eigenvalue and Null Space Properties: Guarantees for convex ℓ₁, group, and structured approaches usually require the Restricted Isometry Property, Restricted Eigenvalue, or Null Space Property. For sequential convex relaxation (Bi et al., 2 Nov 2024), the robust Restricted Null Space Property (rRNSP) and its sequential version (rSRNSP) are sufficient for support recovery—even under weaker conditions than needed for the Lasso.
• Phase Transitions: In exact optimization (Bertsimas et al., 2017), "statistical" phase transitions manifest as thresholds in n: above which support recovery approaches 100%. "Computational" transitions are also observed, with solution time dropping sharply once these sample thresholds are reached.
• Sample Complexity: Greedy and adaptive methods (e.g., (Jalali et al., 2012, Kelner et al., 2023)) empirically require fewer samples, especially with shared supports or structured design. Nonconvex and sequential approaches claim weaker requirements than the Lasso for correct support recovery.

5. Empirical Performance, Computational Complexity, and Practical Applications

Method	Support Recovery	False Discovery	Computation
Lasso/Elastic Net	Moderate	Often high	Fast
Group/Structured Lasso	Context dependent	Moderate	Fast/Moderate
Greedy forward-backward	High (group/singleton mix)	Low if conditions hold	Fast/Moderate
Nonconvex/Sequential	Superior under weaker NSPs	Low	Moderate
Exact/Cutting-plane	Perfect at high n	Nearly zero	Modest (even large p)
Hybrid/adaptive	High, even ill-conditioned	Low	Polynomial
Entropy/constraint model	As above, highly flexible	User-controlled	Moderate

Empirical studies (Bertsimas et al., 2019) show that integer and Boolean-relaxation-based optimizers outperform Lasso and even nonconvex methods, particularly as feature correlation increases or signal-to-noise ratio improves. Adaptive feature methods maintain high recovery error and low false positives regardless of design conditioning for constant sparsity (Kelner et al., 2023).

Practical applications include:

• Multi-task learning (multi-class classification, joint regression across conditions) (Jalali et al., 2012);
• Genomics, imaging, finance (highly correlated features, missing data, group structure) (Ganti et al., 2015, Ghorbani et al., 2015, Bertsimas et al., 2019);
• Robust regression under adversarial corruption or gross outliers (Katayama et al., 2015, Liu et al., 2018);
• Cases with partial or conditional structure (e.g., conditional DNF constraints) (Juba, 2016).

Flexible modeling, as in the entropy-based or Bayesian (empirical Bayes ECM) algorithms (McLain et al., 2022, Srivastava et al., 2023), allows the explicit addition of application-driven constraints and prior structure.

6. Current Limitations, Contrasts, and Future Perspectives

• Provable Optimality vs. Practical Scalability: While exact methods increasingly scale due to algorithmic innovation (Bertsimas et al., 2017), combinatorial and nonconvex approaches still incur significant complexity at high sparsity levels and/or in highly correlated regimes. Adaptive and sequential methods offer polynomial improvements, but may be limited by preprocessing (e.g., spectral decomposition, clustering) and the combinatorics of structured dependencies.
• Assumptions and Robustness: Standard convex methods require strong design assumptions (mutual incoherence, RIP). Algorithms exploiting weaker null space properties or alternate boosting/sequential mechanisms provide recovery guarantees even if Lasso fails (Bi et al., 2 Nov 2024). Robust and hybrid methods relax classical noise/outlier assumptions while still furnishing performance bounds.
• Flexible Constraints: The entropy-based and partitioned-Bayes ECM frameworks (Srivastava et al., 2023, McLain et al., 2022) offer user-controlled constraint imposition, supporting complex modeling needs. Such flexibility comes with increased design and computational burden.
• Open Problems: Scaling flexible/sequential/structured methods to ultra-high p and generalizing precise statistical guarantees, especially under minimal assumptions or heavy-tailed/corrupted designs, is ongoing. Bridging statistical-computational gaps—i.e., matching information-theoretic limits with tractable algorithms—remains a central theme.

7. Summary Table of Representative Methods

Method	Key Feature	Reference
Forward-backward greedy	Singleton/row support tradeoff	(Jalali et al., 2012)
Structured quantile regression	ℓ₁,∞-norm path, group structure	(Nassiri et al., 2013)
SWAP (variable swapping)	Support refinement, correlated X	(Vats et al., 2013)
Adaptive feature/peeling	Robust to ill-conditioning	(Kelner et al., 2023)
Exact binary convex	Cutting-plane/dual reformulation	(Bertsimas et al., 2017)
Entropy-based/constraint model	Explicit selection constraints, flexible	(Srivastava et al., 2023)

All sparse regression algorithms balance computational tractability, recovery guarantees, and practical modeling fidelity. The field continues to develop methods that combine efficient optimization, robust theoretical underpinning (especially under weakened requirements), and the flexibility to model complex, high-dimensional data structures.