Partial Column-Wise Least Squares (PCLS)

• PCLS is a sparse least-squares regression method that selects and rescales columns to yield an r-sparse solution with strong theoretical approximation guarantees.
• It operates in two stages: first, a deterministic SVD-based column selection and rescaling, and then solving a restricted least squares problem for efficient reconstruction.
• Empirical results demonstrate that PCLS achieves competitive residual norms compared to truncated SVD, proving its effectiveness in high-dimensional and ill-conditioned settings.

Partial Column-Wise Least Squares (PCLS) is a deterministic algorithm for sparse least-squares regression under cardinality constraints. It is designed to find an approximately optimal solution to the minimization problem $\min_{x \in \mathbb{R}^n} \|A x - b\|_2$ with the additional requirement that the solution vector $x$ be $r$-sparse, i.e., $\|x\|_0 \leq r$. By explicitly selecting and rescaling a subset of columns from $A$ and then solving a restricted least-squares problem with these columns only, PCLS balances the goals of statistical robustness, computational tractability, and interpretability via sparsity. The method provides strong theoretical guarantees that the resulting residual norm is competitive with the best low-rank SVD-based regularization, justifying its utility for sparse regression even in high-dimensional or ill-conditioned settings (Boutsidis et al., 2013).

1. Sparse Least-Squares Problem Statement

Given $A \in \mathbb{R}^{m \times n}$ and $b \in \mathbb{R}^m$, with an integer $r \ll n$, the objective is to find $x$ satisfying: $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$ Standard unconstrained least squares, $x$0, often yields dense solutions and can be unstable if $x$1 is ill-conditioned. Alternative regularization via the truncated SVD, $x$2, where $x$3 is the best rank-$x$4 approximation of $x$5, achieves stability but does not enforce sparsity. The PCLS goal is to construct an $x$6-sparse $x$7 for $x$8 such that

$x$9

2. PCLS: Two-Stage Algorithmic Structure

PCLS proceeds according to a two-phase procedure:

Stage 1: Column Selection and Rescaling

A deterministic sampling procedure selects $r$0 columns from $r$1 based on its rank-$r$2 SVD structure, forming a sampling matrix $r$3 and rescaling diagonal matrix $r$4. The product $r$5 defines which columns of $r$6 are included and their scaling.

Stage 2: Restricted Least Squares and Reconstruction

A reduced $r$7 matrix $r$8 is formed. The least-squares problem

$r$9

is solved, and the sparse solution in $\|x\|_0 \leq r$0 is recovered as $\|x\|_0 \leq r$1, which by construction has at most $\|x\|_0 \leq r$2 nonzero entries.

Algorithm PCLS_Deterministic Pseudocode

$x$00

3. Deterministic Column Sampling

The deterministic column-selection mechanism was adapted from the column subset selection literature. The columns of $\|x\|_0 \leq r$3 and $\|x\|_0 \leq r$4 are denoted $\|x\|_0 \leq r$5 and $\|x\|_0 \leq r$6, respectively. At each iteration, a column $\|x\|_0 \leq r$7 and weight $\|x\|_0 \leq r$8 are selected to satisfy

$\|x\|_0 \leq r$9

where

$A$0

and the functions $A$1 depend on the spectrum of a running $A$2 matrix $A$3 and the vectors $A$4. After $A$5 iterations (allowing repeated columns), $A$6 and $A$7 encode the selection and rescalings required for the reduced problem.

Column Sampling Procedure Overview

Quantity	Description	Role
$A$8	$A$9th col. of $A \in \mathbb{R}^{m \times n}$0	SVD-informed importance
$A \in \mathbb{R}^{m \times n}$1	$A \in \mathbb{R}^{m \times n}$2th col. of $A \in \mathbb{R}^{m \times n}$3	Residual after SVD
$A \in \mathbb{R}^{m \times n}$4	Running $A \in \mathbb{R}^{m \times n}$5 matrix	Spectrum control
$A \in \mathbb{R}^{m \times n}$6	Inequalities to balance residual/Frobenius structure	Sampling condition

This guarantees control over both spectral and Frobenius aspects of the selected submatrix, ensuring downstream approximation quality.

4. Structural Bound and Approximation Guarantees

The key approximation theorems can be summarized as follows:

Structural Lemma

If the sampling matrices satisfy $A \in \mathbb{R}^{m \times n}$7, then for

$A \in \mathbb{R}^{m \times n}$8

with $A \in \mathbb{R}^{m \times n}$9, it holds that

$b \in \mathbb{R}^m$0

The second term quantifies error due to the portion of $b \in \mathbb{R}^m$1 orthogonal to its best rank-$b \in \mathbb{R}^m$2 approximation and how well the selected columns capture this residual.

Main Approximation Theorem

When PCLS_Deterministic is run with $b \in \mathbb{R}^m$3,

$b \in \mathbb{R}^m$4

where $b \in \mathbb{R}^m$5 is the $b \in \mathbb{R}^m$6th singular value of $b \in \mathbb{R}^m$7. If $b \in \mathbb{R}^m$8 is "effectively" rank $b \in \mathbb{R}^m$9 so that $r \ll n$0, then the PCLS residual approaches the SVD-truncated residual.

Proof Techniques

The proof utilizes the spectral–Frobenius bounds achieved by DeterministicSampling: $r \ll n$1 combined with submultiplicativity of operator norms and the choice of $r \ll n$2 to ensure that $r \ll n$3. Standard SVD projections and properties of the pseudoinverse complete the argument.

5. Computational Complexity Analysis

The overall computational cost consists of the following:

• Computing a rank-$r \ll n$4 SVD of $r \ll n$5: $r \ll n$6.
• DeterministicSampling for $r \ll n$7:
  • Each of $r \ll n$8 steps: $r \ll n$9 to update $x$0,
  • plus $x$1 to examine all $x$2.
  • Total: $x$3.
• Reduced $x$4 least-squares: $x$5.

The cost is dominated by the initial SVD and column sampling, $x$6. Memory requirements are $x$7 for storing $x$8 and $x$9 for the auxiliary structures.

6. Empirical Performance

Empirical evaluation was conducted on random Gaussian $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$0 and $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$1, using $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$2 and varying $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$3 from $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$4 to $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$5. The observed additive error

$$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$6

declined rapidly as $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$7 increased, routinely falling well below the theoretical upper bound. For instance, with $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$8 and $$\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.$$9, empirical errors were much smaller than the worst-case guarantee, demonstrating practical effectiveness and efficiency of the approach (Boutsidis et al., 2013).