---
title: Partial Column-Wise Least Squares (PCLS)
url: https://www.emergentmind.com/topics/partial-column-wise-least-squares-pcls
type: topic
---

# Partial Column-Wise Least Squares (PCLS)

Partial Column-Wise Least Squares (PCLS) is a deterministic algorithm for sparse least-squares regression under cardinality constraints. It is designed to find an approximately optimal solution to the minimization problem $\min_{x \in \mathbb{R}^n} \|A x - b\|_2$ with the additional requirement that the solution vector $x$ be $r$-sparse, i.e., $\|x\|_0 \leq r$. By explicitly selecting and rescaling a subset of columns from $A$ and then solving a restricted least-squares problem with these columns only, PCLS balances the goals of statistical robustness, computational tractability, and interpretability via sparsity. The method provides strong theoretical guarantees that the resulting residual norm is competitive with the best low-rank SVD-based regularization, justifying its utility for sparse regression even in high-dimensional or ill-conditioned settings [1312.7499].

## 1. Sparse Least-Squares Problem Statement

Given $A \in \mathbb{R}^{m \times n}$ and $b \in \mathbb{R}^m$, with an integer $r \ll n$, the objective is to find $x$ satisfying:
\[
\min_{x \in \mathbb{R}^n} \|A x - b\|_2 \quad \text{subject to} \quad \|x\|_0 \leq r.
\]
Standard unconstrained least squares, $x^* = A^\dagger b$, often yields dense solutions and can be unstable if $A$ is ill-conditioned. Alternative regularization via the truncated SVD, $x^*_k = A_k^\dagger b$, where $A_k$ is the best rank-$k$ approximation of $A$, achieves stability but does not enforce sparsity. The PCLS goal is to construct an $r$-sparse $\hat{x}_r$ for $r \approx O(k)$ such that
\[
\|A \hat{x}_r - b\|_2 \approx \|A_k^\dagger b - b\|_2.
\]

## 2. PCLS: Two-Stage Algorithmic Structure

PCLS proceeds according to a two-phase procedure:

### Stage 1: Column Selection and Rescaling
A deterministic sampling procedure selects $r$ columns from $A$ based on its rank-$k$ SVD structure, forming a sampling matrix $\Omega \in \{0,1\}^{n \times r}$ and rescaling diagonal matrix $S \in \mathbb{R}^{r \times r}$. The product $\Omega S$ defines which columns of $A$ are included and their scaling.

### Stage 2: Restricted Least Squares and Reconstruction
A reduced $m \times r$ matrix $C = A (\Omega S)$ is formed. The least-squares problem
\[
\hat{y} = \arg\min_{y \in \mathbb{R}^r} \|C y - b\|_2
\]
is solved, and the sparse solution in $\mathbb{R}^n$ is recovered as $\hat{x}_r = (\Omega S) \hat{y}$, which by construction has at most $r$ nonzero entries.

#### Algorithm PCLS\_Deterministic Pseudocode

```text
Input: A ∈ ℝ^{m×n}, b ∈ ℝ^m, target rank k < rank(A), accuracy ε ∈ (0,½) 
Output: sparse x̂_r ∈ ℝ^n with ‖x̂_r‖₀ ≤ r

1. Compute the rank-k SVD of A: A = U_k Σ_k V_k^T + ..., let E = A − U_k Σ_k V_k^T
2. Set r = ⌈9 k/ε²⌉
3. (Ω, S) = DeterministicSampling(V_k, E, r)
4. Form C = A (Ω S)
5. Compute ŷ = C^† b
6. Set x̂_r = (Ω S) ŷ
7. Return x̂_r
```

## 3. Deterministic Column Sampling

The deterministic column-selection mechanism was adapted from the column subset selection literature. The columns of $V_k \in \mathbb{R}^{n \times k}$ and $E = A - U_k \Sigma_k V_k^T$ are denoted $\{v_i\}$ and $\{e_i\}$, respectively. At each iteration, a column $i$ and weight $t$ are selected to satisfy
\[
U(e_i) \leq 1/t \leq L(v_i, B_\tau, \tau - \sqrt{rk}),
\]
where
\[
U(e_i) = \frac{\|e_i\|_2^2}{\|E\|_F^2(1 - \sqrt{k/r})},
\]
and the functions $L(\cdot), \varphi(\cdot)$ depend on the spectrum of a running $k \times k$ matrix $B_\tau$ and the vectors $v_i$. After $r$ iterations (allowing repeated columns), $\Omega$ and $S$ encode the selection and rescalings required for the reduced problem.

### Column Sampling Procedure Overview

| Quantity         | Description                                 | Role                |
|------------------|---------------------------------------------|---------------------|
| $v_i$            | $i$th col. of $V_k$                         | SVD-informed importance |
| $e_i$            | $i$th col. of $E$                           | Residual after SVD  |
| $B_\tau$         | Running $k \times k$ matrix                 | Spectrum control    |
| $U(e_i),L(v_i, ...)$ | Inequalities to balance residual/Frobenius structure | Sampling condition  |

This guarantees control over both spectral and Frobenius aspects of the selected submatrix, ensuring downstream approximation quality.

## 4. Structural Bound and Approximation Guarantees

The key approximation theorems can be summarized as follows:

### Structural Lemma

If the sampling matrices satisfy $\operatorname{rank}(V_k^T \Omega S) = k$, then for
\[
x̂_r = \Omega S(C^\dagger b)
\]
with $C = A \Omega S$, it holds that
\[
\|A x̂_r - b\|_2 \leq \|A_k^\dagger b - b\|_2 + \|(A - A_k)\Omega S(V_k^T \Omega S)^\dagger \Sigma_k^{-1} U_k^T b\|_2.
\]
The second term quantifies error due to the portion of $A$ orthogonal to its best rank-$k$ approximation and how well the selected columns capture this residual.

### Main Approximation Theorem

When PCLS\_Deterministic is run with $r=\lceil 9k/\epsilon^2 \rceil$,
\[
\|A x̂_r - b\|_2 \leq \|A_k^\dagger b - b\|_2 + (1 + \epsilon) \|b\|_2 \frac{\|A - A_k\|_F}{\sigma_k(A)},
\]
where $\sigma_k(A)$ is the $k$th singular value of $A$. If $A$ is "effectively" rank $k$ so that $\|A - A_k\|_F \ll \sigma_k(A)$, then the PCLS residual approaches the SVD-truncated residual.

#### Proof Techniques

The proof utilizes the spectral–Frobenius bounds achieved by DeterministicSampling:
\[
\sigma_k(V_k^T \Omega S) \geq 1 - \sqrt{k/r},\quad \|E \Omega S\|_F \leq \|E\|_F,
\]
combined with submultiplicativity of operator norms and the choice of $r$ to ensure that $(1-\sqrt{k/r})^{-1} \leq 1+\epsilon$. Standard SVD projections and properties of the pseudoinverse complete the argument.

## 5. Computational Complexity Analysis

The overall computational cost consists of the following:

- Computing a rank-$k$ SVD of $A$: $O(m n \min(m, n))$.
- DeterministicSampling for $r \approx O(k/\epsilon^2)$:
  - Each of $r$ steps: $O(k^3)$ to update $B_\tau$,
  - plus $O(n k^2)$ to examine all $v_i, e_i$.
  - Total: $O(r k^3 + n r k^2 + m n)$.
- Reduced $r \times r$ least-squares: $O(m r^2 + r^3)$.

The cost is dominated by the initial SVD and column sampling, $O(m n \min(m, n) + n k^3/\epsilon^2)$. Memory requirements are $O(m n)$ for storing $A$ and $O(n + r^2)$ for the auxiliary structures.

## 6. Empirical Performance

Empirical evaluation was conducted on random Gaussian $A \in \mathbb{R}^{2000 \times 1000}$ and $b \in \mathbb{R}^{2000}$, using $k=20$ and varying $r$ from $20$ to $200$. The observed additive error
\[
\|A x̂_r - b\|_2 - \|A_k^\dagger b - b\|_2
\]
declined rapidly as $r$ increased, routinely falling well below the theoretical upper bound. For instance, with $\|b\|_2 \approx 25$ and $\|A - A_k\|_F / \sigma_k(A) \approx 18$, empirical errors were much smaller than the worst-case guarantee, demonstrating practical effectiveness and efficiency of the approach [1312.7499].

Source: https://www.emergentmind.com/topics/partial-column-wise-least-squares-pcls