Anchor PCA

Abstract: Principal component analysis (PCA) is one of the most widely used unsupervised dimension reduction techniques. We study PCA for data from multiple related domains. Since principal components generally differ across domains, one way to obtain a shared low-rank embedding is to perform PCA on the pooled data. However, this approach can focus on spurious directions that exhibit high variation in only a few domains. To find a robust embedding that still explains most variance in unseen but similar domains, we propose instead to focus on shared directions of variation. To this end, we introduce Anchor PCA which trades off overall explained variance with agreement between the shared and domain-specific low-rank embeddings. Anchor PCA amounts to PCA on a modified target matrix and thus can be solved efficiently. Moreover, we show that Anchor PCA recovers a maximal invariant subspace and admits a minimax reconstruction interpretation under bounded domain-specific covariance inflations. On simulated and real-world gas sensor data with temporal drift, we demonstrate, respectively, that Anchor PCA recovers the maximally invariant subspace and yields embeddings that explain more variance on unseen domains than the pooling baseline and a worst-case alternative. Taken together, these findings establish Anchor PCA as a promising approach to robust unsupervised dimension reduction from multi-domain data.

• The paper introduces Anchor PCA, a robust framework that constructs invariant subspaces across multiple domains to mitigate issues from spurious, high-variance directions.
• It defines two variants—AnchorPCAₗ (soft penalty) and AnchorPCA∞ (hard constraint)—to trade off between maximizing explained variance and achieving subspace agreement.
• The study provides theoretical guarantees on invariant subspace containment and minimax robustness, validated through extensive synthetic and real-world evaluations.

Anchor PCA: Invariant Subspace Recovery and Robust Multi-Domain Dimension Reduction

Motivation and Problem Setting

Principal Component Analysis (PCA) is ubiquitous for unsupervised dimension reduction, but when data is collected across multiple heterogeneous domains, the leading principal directions can vary substantially. Conventional "pooled" PCA on the aggregated covariance matrix risks overemphasizing spurious directions, which may exhibit high variance in only a subset of domains and fail to generalize. The central challenge addressed by this paper is to construct low-rank linear embeddings that are robust to domain shifts—invariant to domain-specific variance inflations, and capable of explaining variance in unseen domains.

The Anchor PCA framework formalizes invariance as the intersection of top-$k$ principal subspaces across all domains. Two variants are defined: AnchorPCA$_\lambda$ (soft penalty) and AnchorPCA$_\infty$ (hard constraint), which trade off explained variance with agreement—the overlap between a candidate subspace and the domain-specific principal subspaces.

Figure 1: Geometric view of a motivating example, showing the local top-3 eigenspaces (left) and recovered rank-3 subspaces (right); AnchorPCA$_\lambda$ and AnchorPCA$_\infty$ recover the true invariant direction while pooled PCA fails to.

Formal Definitions and Algorithmic Framework

Let $E$ denote the set of domains, each with a covariance matrix $\Sigma_e \in \mathbb{R}^{p \times p}$ and principal projection $\Pi_k^{(e)}$. The maximal invariant subspace is defined as: $$S = \bigcap_{e=1}^E \mathrm{Im}(\Pi_k^{(e)}), \quad m = \dim(S)$$ The AnchorPCA approach constructs a shared rank-$k$ projector $_\lambda$0 as the maximizer of a penalized objective: $_\lambda$1 where $_\lambda$2 is the average covariance and $_\lambda$3 controls the variance/invariance tradeoff. AnchorPCA$_\lambda$4 enforces exact overlap, while AnchorPCA$_\lambda$5 interpolates between pooled PCA and full invariance.

A critical insight is that AnchorPCA reduces to PCA on a modified target matrix, facilitating efficient spectral computation.

Theoretical Guarantees

Two main theoretical results underpin AnchorPCA's robustness:

• Invariant Subspace Containment: If there exists a nontrivial invariant subspace $_\lambda$6, AnchorPCA$_\lambda$7 always includes $_\lambda$8, and as $_\lambda$9, AnchorPCA$_\infty$0 converges to $_\infty$1 in its leading directions.
• Minimax Robustness: AnchorPCA$_\infty$2 is minimax optimal for worst-case average reconstruction error under domain-specific covariance inflations bounded in the span of the local top-$_\infty$3 directions.

  Figure 2: Average reconstruction error along a perturbation path; AnchorPCA$_\infty$4 dominates for medium perturbations, AnchorPCA$_\infty$5 for strong inflations, outperforming worst-case and pooled baselines.

Explicitly, the reconstruction error $_\infty$6 for test domains perturbed by covariance inflation $_\infty$7 is minimized: $_\infty$8 AnchorPCA$_\infty$9 is provably optimal among all rank-$_\lambda$0 projectors under this uncertainty set.

Empirical Evaluation

Extensive synthetic experiments demonstrate that AnchorPCA$_\lambda$1 consistently recovers the invariant subspace $_\lambda$2 as sample size increases. Both block-based estimator and a sequential hypothesis testing procedure (FindS$_\lambda$3) are evaluated, confirming asymptotic recovery probabilities and subspace estimation errors.

Figure 3: Recovery of $_\lambda$4 by AnchorPCA$_\lambda$5 and FindS$_\lambda$6; probability of correct dimension recovery converges to 1, operator-norm error vanishes with increasing sample size.

On real-world data, the gas-sensor drift dataset (measurements across temporal batches with sensor aging effects), AnchorPCA$_\lambda$7 yields subspaces that explain significantly more variance in future held-out domains compared to pooled PCA and worst-case projection baselines. The method robustly discounts drift-specific high-variance directions and selects stable dimensions.

Figure 4: Gas sensor drift scenario; AnchorPCA$_\lambda$8 sacrifices source variance but achieves maximal target explained variance on unseen batches.

Practical Implications and Extensions

AnchorPCA introduces a principled, theoretically-grounded framework for robust PCA under domain heterogeneity, bridging classical dimension reduction and distributionally robust optimization. The minimax interpretation gives explicit guarantees under additive variance inflation shifts, relevant for applications with evolving measurement distributions, sensor drift, or biological heterogeneity.

Efficient implementation is enabled by spectral reformulation. Extensions to nonlinear invariant embeddings via autoencoders are discussed, opening further avenues for robust representation learning beyond linear PCA.

Limitations and Future Directions

• The invariance notion depends on the rank $_\lambda$9 and is sensitive to near eigenvalue ties.
• Robustness is currently confined to variance inflation in local principal subspaces; robustness to mean shifts or rotations is not covered.
• Selection of the penalty $_\infty$0 remains pragmatic; while $_\infty$1 is theoretically motivated, further study of finite penalty selection and block-separation diagnostics is warranted.
• Nonlinear invariant dimension reduction (AnchorAE, DPA) offers a promising but computationally heavier path.

  Figure 5: Consistency curves for AnchorPCA$_\infty$2 block-recovery in main synthetic configuration; Gaussian mixture sampling confirms robustness beyond standard parametric assumptions.

  

  Figure 6: Recovery performance in small-$_\infty$3 configuration; subspace error and dimension recovery show comparable convergence behavior.

  

  Figure 7: Small-$_\infty$4 configuration and agreement-separation gap; empirical block separation matches theoretical challenges in invariant recovery.

  

  Figure 8: Gas composition by temporal batch in sensor drift archive; domain heterogeneity and batch imbalances visualized.

Conclusion

Anchor PCA provides a theoretically justified and computationally tractable tool for robust unsupervised dimension reduction across multiple domains. By jointly penalizing disagreement from local principal dimensions and maximizing explained variance, it achieves minimax optimality under structured distributional shifts. Empirical evidence supports both invariant subspace recovery and improved generalization in challenging real-world settings. Extensions to nonlinear regimes and further theoretical robustness analyses represent active directions for the development of domain-robust representation learning (2606.06233).