Sparse Linear Regression

• Sparse linear regression is the estimation of a sparse parameter vector in a linear model, crucial for high-dimensional data analysis.
• Graph-based Square-Root Estimation (GSRE) employs a square-root loss and overlapping group penalties to interpolate between Lasso, group Lasso, and ridge-type methods.
• Efficient ADMM algorithms and strong theoretical guarantees, including finite-sample error bounds and asymptotic normality, support its application in statistics, machine learning, and signal processing.

Sparse linear regression is the statistical problem of estimating a parameter vector $\beta^*$ in the linear model $y = X\beta^* + \varepsilon$, under the assumption that $\beta^*$ is sparse (i.e., most of its entries are zero). This formulation is central in high-dimensional statistics, machine learning, signal processing, and computational biology, where the number of variables $p$ may greatly exceed the number of observations $n$. Sparse linear regression is closely connected to concepts in optimization, computation, and graphical modeling, and the theoretical and algorithmic aspects are subject to active research.

1. General Framework and Graph-based Square-Root Estimation

In the contemporary mathematical formulation, given $X\in\mathbb{R}^{n\times p}$ and $y\in\mathbb{R}^n$, the goal is to recover $\beta^*\in\mathbb{R}^p$ satisfying $y = X\beta^* + \varepsilon$, under sparsity constraints. The "Graph-based Square-Root Estimation" (GSRE) model provides a flexible framework for incorporating prior structural information among predictors and addressing the high-dimensional regime.

The GSRE estimator solves the optimization problem:

$$\min_{\beta\in\mathbb{R}^p} \;\frac{1}{\sqrt{n}}\|y-X\beta\|_2 + \frac{\lambda}{n}\|\beta\|_{G,\tau}$$

where:

• The square-root loss $y = X\beta^* + \varepsilon$0 renders the regularization parameter $y = X\beta^* + \varepsilon$1 pivotal, i.e., independent of the unknown noise standard deviation.
• The graph-based norm $y = X\beta^* + \varepsilon$2 depends on an undirected "predictor graph" $y = X\beta^* + \varepsilon$3 over variables, with local neighborhoods $y = X\beta^* + \varepsilon$4 and positive weights $y = X\beta^* + \varepsilon$5. It is defined as

$y = X\beta^* + \varepsilon$6

This norm encodes overlapping group sparsity regularization adapted to the graphical structure of predictors.

Depending on $y = X\beta^* + \varepsilon$7, GSRE recovers a range of classic estimators:

• For $y = X\beta^* + \varepsilon$8 with no edges, $y = X\beta^* + \varepsilon$9; GSRE becomes the classic square-root Lasso.
• For $\beta^*$0 as disjoint unions of complete graphs, GSRE reduces to the group square-root Lasso with group-specific $\beta^*$1-penalties.
• For complete $\beta^*$2, the estimator induces an $\beta^*$3 (ridge-type) penalty.

2. Theoretical Guarantees: Error Bounds, Asymptotics, and Model Selection

GSRE admits comprehensive theoretical properties concerning both estimation and variable selection.

Finite-Sample Error Bounds:

Suppose $\beta^*$4 is a restricted eigenvalue or compatibility constant for $\beta^*$5. Under compatibility conditions and mild overlap assumptions on the graph, GSRE enjoys:

$\beta^*$6

for true sparsity $\beta^*$7, noise level $\beta^*$8, and constants $\beta^*$9 related to graph overlap.

Asymptotic Normality:

For fixed $p$0 and suitable $p$1, GSRE is asymptotically efficient on the active set:

$p$2

with $p$3 in probability.

Model-Selection Consistency:

If $p$4, the method selects the correct support with probability tending to one under an irrepresentable-type condition, $p$5, and sufficient signal strength ($p$6 large relative to noise).

3. Algorithmic Strategies: Efficient Computation via ADMM

The GSRE estimator is computed by introducing auxiliary variables and solving an augmented Lagrangian via the Alternating Direction Method of Multipliers (ADMM). The algorithm iterates through:

• Quadratic minimization for the $p$7-variable (efficient via Sherman–Morrison–Woodbury when $p$8).
• Proximal updates for $p$9 given square-root loss and for $n$0 given the graph-based norm.
• Customized proximal maps for group-wise projections in computing $n$1.
• Dual variable updates with step-size $n$2, where standard 2-block ADMM convergence theory applies.

This yields a scalable procedure for high-dimensional structured-sparsity problems.

4. Special Cases and Relation to Classical Methods

The GSRE framework encompasses, as special cases:

Graph Structure	Penalty Term	Resulting Estimator
No edges (diag.)	$n$3	Classic Square-root Lasso
Disjoint complete	$n$4	Group Square-root Lasso
Complete graph	$n$5	Square-root Ridge-type Penalty

Thus, GSRE interpolates naturally between element-wise, group, and ridge-type regularization within a unified square-root-loss framework.

5. Empirical Performance and Robustness

Extensive synthetic and real-data experiments evidence the empirical advantages of GSRE over traditional methods:

• In the $n$6 regime ($n$7), GSRE achieves:
  • Lowest average $n$8-error (e.g., $n$9 versus $X\in\mathbb{R}^{n\times p}$0--$X\in\mathbb{R}^{n\times p}$1 for Lasso/Alasso/SRL)
  • Best Relative Prediction Error (RPE)
  • Nearly zero false negative rate; false positive rate below 3%
• Under non-Gaussian and heavy-tailed noise (Student $X\in\mathbb{R}^{n\times p}$2, Laplace, Uniform), GSRE outperforms alternatives owing to the robustness induced by the square-root loss.
• On high-dimensional real datasets (e.g., bodyfat2, miRNA–cancer survival), 10-fold CV demonstrates that GSRE's median test-MSE is about half or less compared to Lasso, Elastic-Net, square-root Lasso, or least-squares graph-penalty. For instance, on the bodyfat2 dataset, GSRE's MSE median is $X\in\mathbb{R}^{n\times p}$3 versus $X\in\mathbb{R}^{n\times p}$4 for competing approaches.

6. Significance and Practical Impact

The GSRE model addresses key challenges in sparse linear regression:

• Unknown Noise Level: Pivotal regularization via square-root loss eliminates the need to know or estimate the noise variance, simplifying parameter selection and enhancing robustness.
• Graphical Structure Utilization: Node-wise overlapping group penalties encode arbitrary predictor relationships—collinearity, clusters, group structure—leading to improved estimation and feature selection.
• Scalability and Adaptability: Efficient ADMM-based solvers, together with the flexibility in structural prior specification, facilitate application to high-dimensional and complex-structure regression problems.

These properties, together with finite-sample near-oracle error rates, asymptotic normality, and strong empirical results, underline GSRE as a general and effective methodology for modern sparse linear regression tasks in high-dimensional statistics and machine learning (Li et al., 2024).