---
title: Variance Thresholding & Retained-Set PCA
url: https://www.emergentmind.com/topics/variance-thresholding-and-retained-set-pca
type: topic
---

# Variance Thresholding & Retained-Set PCA

Variance thresholding and retained-set principal component analysis (PCA) are algorithmic strategies for sparse PCA in high-dimensional settings, wherein the goal is to recover the support of sparse principal components (PCs) from noisy observations. These approaches combine thresholding of empirical variances or covariances with eigenanalysis restricted to a "retained" coordinate set, providing both computational tractability and theoretical guarantees for support recovery under the spiked covariance model. Covariance thresholding refines classical variance-thresholding by incorporating off-diagonal information and leveraging advanced probabilistic bounds, substantially expanding the feasible support size regime for accurate recovery [1311.5179].

## 1. Statistical Model: Sparse PCA and Spiked Covariance Framework

The core model for sparse PCA considered is the "spiked covariance" formulation. One observes $n$ independent samples $X_1,\ldots,X_n \in \mathbb{R}^p$:
$$
X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i
$$
where $v_1, \ldots, v_r \in \mathbb{R}^p$ are orthonormal population PCs, each with $\|v_q\|_0 \leq s_0$ ("$s_0$-sparse"), $u_{q,i} \sim \mathcal{N}(0,1)$, and $\xi_i \sim \mathcal{N}(0, I_p)$ are independent Gaussian noise vectors. The population covariance is thus $\Sigma = \mathbb{E}[X_i X_i^\top] = I_p + \sum_{q=1}^r \beta_q v_q v_q^\top$. Of particular interest is the high-dimensional regime, $p$ comparable to or much larger than $n$. The principal statistical objective is precise recovery of the supports $\mathrm{supp}(v_q)$.

## 2. Methodologies: Classical Variance-Thresholding vs. Covariance-Thresholding

Two principal retained-set PCA algorithms address sparse PC recovery in this setting:

A. **Variance-Thresholding (Johnstone–Lu Diagonal Method)**

- Compute the empirical covariance $\widehat{\Sigma} = n^{-1} \sum_{i=1}^n X_i X_i^\top$.
- Let $\widehat{\sigma}_{jj} = \widehat{\Sigma}_{jj}$. For threshold $\tau$, define the retained index set:
  $$
  \widehat{S} = \{ j \in [p] : \widehat{\sigma}_{jj} \geq 1 + \tau \}
  $$
  (Adjustments may be made for non-unit noise variance via centering or alternative baselines.)
- Restrict $\widehat{\Sigma}$ to the principal submatrix on $\widehat{S}$, then compute its top-$r$ eigenvectors, padding with zeros off $\widehat{S}$ for estimates $\widehat{v}_1, \ldots, \widehat{v}_r \in \mathbb{R}^p$.
- Optional secondary thresholding on $\widehat{v}_q$ can enforce exact sparsity.

B. **Covariance-Thresholding (Deshpande–Montanari Method)**

- Split data into two halves. From the first half, form $\Sigma^1 = n^{-1}\sum X_i X_i^\top$, and subtract the identity: $M = \Sigma^1 - I_p$.
- Entrywise apply soft-thresholding at $\tau/\sqrt{n}$:
  $$
  [\eta(M)]_{ij} = \begin{cases}
    M_{ij} - \tau/\sqrt{n}, & \text{if } M_{ij} \geq \tau/\sqrt{n} \\
    0, & \text{if } |M_{ij}| < \tau/\sqrt{n} \\
    M_{ij} + \tau/\sqrt{n}, & \text{if } M_{ij} \leq -\tau/\sqrt{n}
  \end{cases}
  $$
- Extract the top $r$ eigenvectors $w_1, \ldots, w_r$ of $\eta(M)$.
- On the second half, compute $\Sigma^2$ and coordinate-wise scores $c_{q,j} = (\Sigma^2 w_q)_j$.
- For threshold $\rho$, define the retained set:
  $$
  \widehat{S} = \bigcup_{q=1}^r \{ j : |c_{q,j}| \geq \rho \}
  $$
- Final support estimates are either principal eigenvectors of $\Sigma^2$ restricted to $\widehat{S}$ or thresholded $w_q$.

## 3. Support Recovery Theory

The statistical guarantees describe the support size regimes in which high-probability recovery is assured, and corresponding thresholds that yield optimal behavior:

| Method                              | Support Size Regime         | Threshold Choice        | Guarantee                            |
|--------------------------------------|-----------------------------|------------------------|--------------------------------------|
| Variance-thresholding + retained PCA | $s_0 \leq K_1 \sqrt{n/\log p}$ | $\tau \asymp C\sqrt{\log p/n}$ | $P(\widehat{S} = \mathrm{supp}(v)) \to 1$    |
| Covariance-thresholding + retained PCA | $s_0 \leq K \sqrt{n}$      | $\tau = C\sqrt{\log(p/s_0^2)}$ (or $C\sqrt{\log p}$), $\rho = \theta/(2\sqrt{s_0})$ | $P(\widehat{S} = \mathrm{supp}(v)) \geq 1-o(1)$ |

Here, $K_1, K$ are universal positive constants, $\theta$ is the minimal magnitude parameter on the support of $v$, and $\asymp$ indicates asymptotic proportionality.

A key finding is that covariance thresholding expands the feasible support regime for exact recovery from $s_0 = O(\sqrt{n/\log p})$ to $s_0 = O(\sqrt{n})$, under mild moment conditions and in certain $p \gg n$ scenarios. The sample complexity bound in the rank-one case is $n \geq K_0 s_0^2 \log(p/s_0^2)$, where $K_0$ is universal. Lower bounds from computational complexity theory indicate that no polynomial-time algorithm can succeed for substantially larger $s_0$ [1311.5179].

## 4. Analysis Techniques and Spectral Norm Bounds

The proof of tight support recovery relies on precise control of the spectral norm of the randomized matrices arising from the noise in $\widehat{\Sigma}$ and the effect of entrywise thresholding. Given $\widehat{\Sigma}-I = \text{signal} + \text{noise} = (\beta/n) vv^\top +$ cross-terms $+ (n^{-1}XX^\top - I)$, soft-thresholding balances retention of the rank-one signal and substantial attenuation of the noise. The challenging task is bounding $\|\eta(n^{-1}XX^\top - I)\|_{op}$, where $\eta$ is non-Lipschitz.

Earlier results addressed asymptotic spectral distributions or smooth functions; here, a non-asymptotic tail bound for the given kernel random matrix is required. The authors develop an $\varepsilon$-net argument combined with a new concentration lemma for non-Lipschitz Gaussian functionals, controlling $\sup_{\|y\|=1} |y^\top \eta(n^{-1}XX^\top - I) y|$ via discretization of the sphere and analyzing typical Gaussian behavior versus rare bad events [1311.5179].

## 5. Practical Considerations and Threshold Selection

In applied settings, the theoretical threshold prescriptions depend on unknown quantities such as $\beta$ and $s_0$. Suggested heuristics are:

- **Threshold $\tau$**: Select $\tau \asymp C(\beta \vee 1)\sqrt{\log(p/s_0^2)}$; when $s_0 = O(\sqrt{n})$ this is $O(\sqrt{\log(p/s_0^2)})$.
- **Pragmatic Rule**: Use an estimated noise-variance level: $\tau \approx \nu \widehat{\sigma}^2$, where $\widehat{\sigma}^2$ is estimated from the median absolute deviation (MAD) of off-diagonal or diagonal elements, with $\nu \sim 3 - 4$.
- Spectral diagnostics via the Marchenko–Pastur edge location on the thresholded matrix can also guide $\tau$ selection.

The table below summarizes the main computational steps and their leading costs for covariance-thresholding:

| Step         | Operation                           | Complexity                 |
|--------------|-------------------------------------|----------------------------|
| 1            | Split, form $\Sigma^1$, $M$         | $O(p^2)$                   |
| 2            | Soft-threshold $M$                  | $O(p^2)$                   |
| 3            | Top-$r$ eigenvectors                | $O(p^3)$ or $O(rp^2)$      |
| 4            | Form $\Sigma^2$, actions on $w_q$   | $O(np r)$                  |
| 5            | Submatrix eigendecomposition        | $O(|\widehat{S}|^3)$       |

For $p \gg n$ scenarios, randomized SVD or low-rank subspace iterations may provide efficient alternatives.

## 6. Connections to Literature and Limitations

Variance-thresholding and retained-set PCA, originally proposed by Johnstone and Lu, established foundational support-recovery thresholds in sparse PCA [Johnstone–Lu 2004]. Covariance-thresholding, as developed by Deshpande and Montanari, achieves strictly larger support recovery regimes without increased computational cost, and its guarantees match known computational lower bounds [berthet2013computational, ma2015sum]. These advances synthesize ideas from high-dimensional statistics, spectral random matrix theory, and computational complexity. No practical, polynomial-time algorithm currently exhibits better support recovery guarantees for the spiked covariance model under standard noise assumptions [1311.5179].

## 7. Summary and Contemporary Relevance

Variance-thresholding and retained-set PCA remain essential techniques for sparse principal component estimation in high dimensions. Covariance-thresholding generalizes diagonal thresholding, effectively leveraging both diagonal and off-diagonal information, and is distinguished by optimal support-recovery, robust performance in the high-dimensional regime, and refined non-asymptotic analysis. The introduction of sharp spectral norm bounds for thresholded random kernel matrices is a substantial technical contribution and informs the theoretical underpinnings of related algorithms in high-dimensional statistics and machine learning [1311.5179].

Source: https://www.emergentmind.com/topics/variance-thresholding-and-retained-set-pca