Penalized Least Squares (PLS)

• Penalized Least Squares is a framework that augments classical least squares with penalties (e.g., lasso, ridge, SCAD) to enforce sparsity, smoothness, or robustness.
• It employs efficient optimization algorithms like coordinate descent, ADMM, and semismooth Newton methods to handle high-dimensional and nonconvex problems.
• Theoretical guarantees such as oracle properties, error bounds, and minimax rates validate its performance in variable selection and nonparametric estimation.

Penalized Least Squares (PLS) constitutes a foundational methodology for statistical estimation and model selection in high-dimensional regression, nonparametric inference, inverse problems in imaging, and function estimation. PLS augments classic least squares with a penalty functional designed to enforce sparsity, smoothness, shape-restriction, or robustness properties, and has undergone extensive theoretical and algorithmic development across a range of applications, from variable selection in genomics and chemometrics to signal denoising and baseline correction in spectroscopy.

1. General Formulation and Classes of Penalties

The archetypal penalized least squares estimator solves

$$\min_{\beta \in \mathbb{R}^p} \frac{1}{2n} \|y - X\beta\|_2^2 + \sum_{j=1}^p p_{\lambda}(|\beta_j|),$$

where $X \in \mathbb{R}^{n \times p}$ encodes covariates, $y \in \mathbb{R}^n$ is the response, $p_{\lambda}$ is a penalty parameterized by $\lambda$ promoting a desired structure (e.g., sparsity or smoothness), and the penalty may be convex (lasso, ridge), nonconvex (SCAD, MCP), or incorporate group and sorted/layered penalties. Particular cases include:

• Lasso (ℓ₁): $p_{\lambda}(t) = \lambda t$, achieving convex sparsity.
• Ridge: $p_{\lambda}(t) = \lambda t^2$, promoting regularization without sparsity.
• SCAD: Piecewise quadratic penalty flattening for large coefficients, reducing bias on strong signals.
• MCP: Minimax concave penalty with vanishing derivative beyond a threshold.
• Group/overlapping/grouped ℓ₁/ℓ₂: Penalization structured over variable blocks.
• Sorted-Concave (Slope/Sorted ℓ₁, MCP/SCAD-Slope): Penalty levels assigned adaptively according to order statistics, optimally adjusting thresholds according to signal strength and sparsity (Feng et al., 2017).
• Nonparametric PLS: Extends to function spaces, with penalty typically a seminorm enforcing smoothness or shape-constraints (Muro et al., 2015).

2. Optimization Algorithms and Computational Aspects

Algorithmic advances underpin PLS scalability and accuracy. Standard approaches include:

• Coordinate Descent (CD): Efficient for separable, closed-form thresholdable penalties, including MCP and SCAD. Algorithms for MCP/SCAD PLS are linearly convergent under restricted eigenvalue and other regularity conditions (Jiao et al., 2021).
• Semismooth Newton and Augmented Lagrangian Methods: For nonconvex/nonsmooth penalties, especially in adaptive lasso or semiparametric settings, fast local convergence is achieved with semismooth Newton updates within augmented Lagrangian frameworks. The dual problem formulation is often tractable, with constraints encoded as indicator functions (Yang et al., 2021).
• ADMM (Alternating Direction Method of Multipliers): Used when penalties are coupled across variables/components—e.g., joint/group sparsity in PLS regression. Row-wise soft-thresholding is used for group ℓ₁/ℓ₂ penalties; column updates are tackled via secular equations under orthogonality constraints (Liu et al., 2014).
• Active-set and LARS-type Algorithms: Particularly effective in enforcing exact zeros under sparsity constraints and incorporating sign/shape constraints, dramatically reducing computational cost in high sparsity regimes (Vega-Hernández et al., 2019).
• Iteratively Reweighted Least Squares (IRLS): Employed in PLS-based baseline correction/smoothing problems, especially for robust baseline fitting in spectroscopy. Efficient O(m) updates per iteration can be realized for banded difference penalties (Liu et al., 2022).
• Two-stage frameworks: Used for structural equations or instrumental variable settings (ridge step for prediction, followed by adaptive lasso for selection) (Chen et al., 2015).
• Proximal-gradient, Local Convex Approximation (LCA): For nonconvex or sorted penalties, surrogate convex subproblems are solved iteratively with fast projections and tailored thresholds (Feng et al., 2017).

3. Theoretical Guarantees and Statistical Properties

PLS estimators are rigorously characterized in terms of prediction error, support recovery, and oracle properties:

• Oracle Properties: Consistency in variable selection and asymptotic normality of coefficients under sufficiently strong penalization on inactive coefficients and controlled bias on active ones. For example, adaptive lasso, SCAD, and MCP all recover the true support and achieve root-n consistency for nonzero coefficients under RE-type conditions and β-min assumptions (Jiao et al., 2021, Suzuki et al., 2018, Zhang et al., 2014).
• Error Bounds: Finite-sample nonasymptotic bounds for prediction, ℓ₁/ℓ₂ estimation, and support estimation are derived under conditions such as restricted eigenvalue or incoherence (Pokarowski et al., 2013, Feng et al., 2017).
• Concentration Phenomena: In nonparametric settings, the trade-off between empirical error and penalization is shown to be sharply concentrated around a deterministic benchmark, enabling optimal calibration of tuning parameters for minimax rates (Muro et al., 2015).
• Minimax Rates and Adaptivity: Sorted-concave PLS achieves the minimax rate in prediction and estimation—i.e., order optimal (s log(p/s))/n for s-sparse signals—without requiring knowledge of true sparsity (Feng et al., 2017).
• Task-based Regularization: Inverse problems and imaging benefit from task-adapted penalties (e.g., for detectability under linear observer models), yielding superior preservation of task-relevant structure even without ground-truth signals (Chen et al., 30 Jan 2025).

4. Extensions: Function Spaces, Shape Restrictions, Baseline Correction, and Inverse Problems

PLS extends naturally to non-Euclidean and infinite-dimensional estimation:

• Nonparametric Function Estimation: PLS with smoothness seminorms or convex penalties recovers classical smoothing splines (e.g., m-th order penalty for Sobolev smoothness) with strong concentration properties and minimax-optimal empirical risk (Muro et al., 2015).
• Shape-restricted Regression: The penalized QP framework enforces concavity/convexity and monotonicity via slack penalties, drastically reducing computational complexity over naive constrained approaches. Dual formulations deliver further efficiency and scalability for large sample sizes (Keshvari, 2016).
• Baseline Correction in Spectroscopy: Penalized least-squares smoothers, particularly with adaptive, asymmetric weighting and spatially adaptive regularization, have shown quantitative superiority for baseline estimation in astronomical spectra, enabling robust detection of weak signals in high-noise contexts (Liu et al., 2022).

5. PLS for Partial Least Squares Regression and Integrative Analysis

PLS terminology is historically overloaded; within Partial Least Squares regression, penalization enforces sparsity and interpretability:

• Sparse and Regularized PLS: Sparse PLS imposes penalties directly on loading vectors (e.g., ℓ₁, group, fused, or structured penalties), yielding interpretable directions and selection of salient predictors even when p≫n (Allen et al., 2012).
• Joint-Sparsity and Group Structure: ADMM and group-sparsity penalties select variables globally across multiple PLS components, dramatically reducing model complexity without sacrificing predictive power (Liu et al., 2014).
• Multistudy and Integrative PLS: Extends variable selection to multi-dataset contexts, using group penalties (e.g., group MCP) and contrast penalties to enforce both shared and distinct signal recovery across studies (Liang et al., 2020). Closed-form and threshold-then-normalize strategies for dual-norm penalization provide efficient, tunable-sparsity paths (Alsouki et al., 2023).

6. Robust Regression, Outlier Detection, and Model Selection

Customized PLS approaches address robustness and model selection consistency:

• Penalized Weighted Least Squares (PWLS): Simultaneous outlier detection and estimation via shrinkage of observation weights, derived via Bayesian or M-estimation frameworks, with BIC-type and stability-based tuning strategies; often outperforming classical robust regression (Gao et al., 2016).
• Combined ℓ₁/ℓ₀ Selection: Hybrid screening, ordering, and greedy subset selection provides nonasymptotic selection guarantees with substantive reduction in computational burden compared to exhaustive search, especially for moderate n,p (Pokarowski et al., 2013).
• Best-Subset Selection via ℓ₀-PLS: Modern BnB algorithms with node-screening tests and nesting properties render exact ℓ₀-penalized least-squares feasible at moderate problem sizes, with empirical speed-ups over classic MIP approaches (Guyard et al., 2021).

7. Practical Recommendations, Tuning, and Comparative Performance

The empirical literature stresses the crucial role of penalty choice, regularization parameter selection, and design geometry:

• Penalty Selection: SCAD/MCP and adaptive lasso achieve the best compromise between selection accuracy (oracle property) and bias; lasso is suboptimal for highly correlated designs or strong signals due to bias (Zhang et al., 2014).
• Regularization Tuning: Cross-validation, information criteria (e.g., BIC), and stability selection are widely adopted. Sorted-penalties can bypass the need for knowledge of the underlying sparsity by adaptively decreasing thresholds (Feng et al., 2017).
• Scalability: CD, warm-starts, and active-set strategies make PLS models practical for ultra-high dimensional problems (p ~ 10⁵–10⁶), while dual and primal-dual solvers make high-resolution nonparametric estimation tractable (Jiao et al., 2021, Keshvari, 2016).
• Applicability: Applications include gene regulatory network inference, EEG/MEG source imaging, signal detection in astronomical spectroscopy, robust regression with contamination, and multi-study omics platforms (Chen et al., 2015, Liang et al., 2020, Chen et al., 30 Jan 2025, Vega-Hernández et al., 2019, Liu et al., 2022).

In summary, Penalized Least Squares is a unifying and deeply flexible framework governing high-dimensional estimation, robust inference, variable selection, and function approximation. Ongoing research blends statistical theory, optimization, and domain-specific constraints to extend PLS methodology to ever broader scientific applications, with rigorous performance guarantees established across convex and nonconvex regimes.