Hard Uniformity-Constrained Contrastive PCA

• The paper introduces PCA++, a method that extracts shared low-dimensional signal subspaces with a hard uniformity constraint to mitigate background noise.
• It employs a generalized eigenproblem to achieve a closed-form solution, ensuring identity covariance in the projected features for robust performance.
• Empirical evaluations on simulations, corrupted-MNIST, and single-cell RNA-seq data demonstrate PCA++’s superior signal recovery compared to standard PCA and alignment-only methods.

Hard Uniformity-Constrained Contrastive PCA (commonly denoted as PCA++) is a spectral method for extracting low-dimensional shared signal subspaces from paired high-dimensional observations, even under strong structured background noise. The method is rooted in contrastive learning principles and introduces an explicit hard uniformity constraint, ensuring the projected features have identity covariance and thereby regularizing against background interference. PCA++ is characterized by a closed-form solution via a generalized eigenproblem, enjoys provable robustness in high-dimensional regimes, and has demonstrated empirical effectiveness compared to standard PCA and alignment-only contrastive methods (Wu et al., 15 Nov 2025).

1. Problem Formulation and Optimization Objective

Given paired data matrices $X, X^+ \in \mathbb{R}^{n \times d}$, where each pair $(x_i, x_i^+)$ shares an identical low-dimensional signal but different background noise realizations, the objective is to recover the underlying shared subspace. Two covariance structures are central:

• Contrastive (alignment) covariance: $S_+ = \frac{1}{2n}(X^\top X^+ + {X^+}^\top X)$, quantifying the statistical alignment of positive pairs.
• Standard sample covariance: $S = \frac{1}{n} X^\top X$, capturing the overall variance structure in the observed data.

PCA++ is defined as the solution to the following constrained optimization problem:

$$\begin{aligned} & \underset{V \in \mathbb{R}^{d \times k}}{\text{maximize}} && \operatorname{Tr}(V^\top S_+ V) \ & \text{subject to} && V^\top S V = I_k \end{aligned}$$

The alignment term maximizes the signal correspondence across pairs, while the hard uniformity constraint enforces identity covariance in the projected subspace, preventing the solution from collapsing onto dominant background directions with high variance. For $d \gg n$, an optional truncation is employed whereby $S$ is replaced with a rank-$s$ approximation $S_s$, improving numerical stability by discarding near-zero modes.

2. Closed-Form Solution via Generalized Eigenproblem

The solution employs Lagrangian duality, introducing a symmetric multiplier $M$ for the covariance constraint. The stationarity condition $\partial L/\partial V = 0$ yields the generalized eigenproblem:

$S_+ v_j = \lambda_j S v_j, \quad j = 1, \ldots, d$

The top $k$ eigenvectors $v_1, \ldots, v_k$ associated with the largest real generalized eigenvalues $\lambda_1 \geq \ldots \geq \lambda_d$ constitute the columns of the optimal $V$. When using rank-$s$ truncation for $S$, the procedure involves first projecting into the dominant eigenspace of $S$ (with possible ridge regularization), then solving a much smaller eigenproblem in that subspace, and finally mapping back to $\mathbb{R}^d$.

3. Algorithmic Workflow

The algorithmic implementation of PCA++ follows these steps:

1. Compute covariance matrices:
   • $S \leftarrow X^\top X / n$
   • $S_+ \leftarrow (X^\top X^+ + {X^+}^\top X)/(2n)$
2. (Optional) Truncation in high dimensions:
   • Eigendecompose $S \approx V_x \Lambda_x V_x^\top$ with top $s$ eigenpairs
   • Form $R \leftarrow V_x (\Lambda_x + \epsilon I)^{-1/2}$
   • Set $M \leftarrow R^\top S_+ R$
   • Eigendecompose $M = U \Lambda U^\top$
   • Obtain generalized eigenvectors $V \leftarrow R U$
3. No truncation:
   • Directly solve $S_+ v = \lambda S v$ via a standard generalized eigenproblem solver
4. Post-processing:
   • Sort eigenvalues $\lambda$ in descending order
   • Return the $d \times k$ matrix $V$ whose columns are the top $k$ eigenvectors

This procedure yields a projection that maximally aligns paired structure while enforcing the dispersion regularization.

4. High-Dimensional Theoretical Properties

The recovery guarantees of PCA++ are analyzed under a linear contrastive factor model:

$$\begin{aligned} x &= A w + B h + \epsilon \ x^+ &= A w + B h' + \epsilon' \end{aligned}$$

where $A \in \mathbb{R}^{d \times k}$ encodes signals, $B \in \mathbb{R}^{d \times m}$ encodes backgrounds, $w,h,h'$ are low-dimensional latent variables, and $\epsilon,\epsilon'$ denote noise.

Theoretical results are provided under two high-dimensional regimes:

A. Fixed-Aspect Ratio ($d/n \to c \in (0,\infty)$):

Assume all population "spikes" (signal and background eigenvalues) $\lambda_{A,j},\lambda_{B,j} \geq \sqrt{c}$ are distinct (BBP detectability). Let $\widetilde{U}_A$ denote the recovered PCA++ subspace and $U_A$ the true signal subspace. Then, almost surely,

$$\operatorname{dist}^2(\widetilde{U}_A, U_A) \to 1 - \frac{1 - c/\lambda_{A,k}^2}{1 + c/\lambda_{A,k}}$$

where $\operatorname{dist}$ is the operator-norm sine of principal angles. When the weakest signal strength $\lambda_{A,k} \gg \sqrt{c}$, the error tends to zero.

B. Growing-Spike Regime ($\lambda_{A,j},\lambda_{B,j} \to \infty$):

With $d/(n \lambda_{A,j}) \to c_{A,j}$ and $d/(n \lambda_{B,j}) \to c_{B,j}$, under distinctness,

$$\operatorname{dist}^2(\widetilde{U}_A, U_A) \to \frac{c_{A,k}}{1 + c_{A,k}}$$

Uniformity, enforced via the covariance constraint, continues to regularize away background spikes and the limiting recovery performance is controlled by the same mechanism as in the fixed-aspect scenario.

5. Empirical Performance and Comparative Analysis

Experimental evaluations demonstrate the relative strengths and weaknesses of PCA++, standard PCA, and alignment-only PCA+ across various regimes:

• One-signal/one-background simulations: As background strength $\lambda_B$ or relative dimension $d/n$ increase, both PCA and PCA+ exhibit subspace drift towards background axes, while PCA++ remains stably aligned with the signal.
• High-dimensional simulations: Subspace error for PCA++ matches the asymptotic theoretical predictions.
• Corrupted-MNIST embedding: In two-dimensional embeddings where digits are obscured with added background (“digit+grass”), standard PCA fails to separate classes, PCA+ achieves partial separation, and PCA++ achieves clear separation of classes (specifically, distinguishing '0' from '1' along the principal component).
• Single-cell RNA-seq: For datasets containing invariant and condition-responsive cell types, PCA splits the same cell types by experimental condition, while PCA++ clusters invariant types (e.g., B cells) together and aligns responsive types with true biological variation.

The table below summarizes several empirical comparisons:

Method	Failure Mode	Alignment with Signal (Increasing Background)
Standard PCA	Captures background spikes	Declines
PCA+	Dominated by background if $\lambda_B \gg \lambda_A$ or $d \gg n$	Declines sharply
PCA++	Suppresses high-variance background directions	Remains stable

6. Practical Considerations and Implementation Guidelines

• Subspace Dimension ($k$): When unknown, select $k$ by inspecting the spectrum of generalized eigenvalues $\lambda_j$; look for a spectral gap beneath which $\lambda_j \to 0$.
• Truncation Rank ($s$): Invertibility and stability are maintained by choosing $s$ to capture approximately 90% of $S$'s variance but avoid near-zero eigenvalues. Monitoring the condition number $\operatorname{cond}(\Lambda_x)/\Lambda_x(s)$ is advised.
• Computational Complexity: Covariance computation scales as $O(nd^2)$ or $O(d n^2)$. Truncated eigendecomposition costs $O(sd^2)$ using IRLM or Lanczos when $s \ll d$. The remaining eigenproblem scales as $O(s^3)$. Total computational cost: $O(n d^2 + s d^2)$.
• Numerical Stability: Apply a small ridge $\epsilon$ before inversion and use truncated spectral decompositions along with iterative solvers when appropriate.

In summary, hard uniformity-constrained contrastive PCA (PCA++) provides a principled and scalable approach for signal recovery in paired high-dimensional datasets, with closed-form solutions, robust high-dimensional error guarantees, and empirically verified advantages over both classical and alignment-only contrastive PCA (Wu et al., 15 Nov 2025).