A Geometric Analysis of PCA (2510.20978v1)

Abstract: What property of the data distribution determines the excess risk of principal component analysis? In this paper, we provide a precise answer to this question. We establish a central limit theorem for the error of the principal subspace estimated by PCA, and derive the asymptotic distribution of its excess risk under the reconstruction loss. We obtain a non-asymptotic upper bound on the excess risk of PCA that recovers, in the large sample limit, our asymptotic characterization. Underlying our contributions is the following result: we prove that the negative block Rayleigh quotient, defined on the Grassmannian, is generalized self-concordant along geodesics emanating from its minimizer of maximum rotation less than $\pi/4$.

• The paper establishes a central limit theorem for PCA’s excess risk, characterizing its asymptotic distribution under finite moment and eigengap assumptions.
• It leverages Grassmannian geometry to analyze PCA as an M-estimator and proves a generalized self-concordance of the block Rayleigh quotient.
• The study derives non-asymptotic risk bounds that connect sample complexity with eigengaps and fourth moments, offering robust insights for dimensionality reduction.

Geometric Analysis of Principal Component Analysis: Excess Risk and Self-Concordance

Introduction and Motivation

This paper provides a comprehensive geometric and statistical analysis of Principal Component Analysis (PCA), focusing on the excess risk under the reconstruction loss. The central question addressed is: Which property of the data distribution determines the excess risk of PCA? The authors develop a framework that leverages the geometry of the Grassmannian manifold, treating PCA as an M-estimator, and employ tools from asymptotic and non-asymptotic statistics to characterize both the asymptotic distribution and finite-sample behavior of the excess risk.

Problem Formulation and Geometric Framework

PCA is formulated as the minimization of the empirical reconstruction error over the Grassmannian $\mathrm{Gr}(d, k)$, the manifold of $k$-dimensional subspaces of $\mathbb{R}^d$. The population and empirical risks are defined as: $$\widetilde{R}(U) = \frac{1}{2} \mathbb{E} \|X - UU^\top X\|_2^2, \quad \widetilde{R}_n(U) = \frac{1}{2n} \sum_{i=1}^n \|X_i - UU^\top X_i\|_2^2$$ where $U \in \mathbb{R}^{d \times k}$ is an orthonormal basis for the subspace. The analysis is performed on equivalence classes $[U]$ in the Grassmannian, eliminating the redundancy of the orthonormal basis representation.

The geometry of the Grassmannian is exploited via the Riemannian metric, geodesics, principal angles, and the logarithmic and exponential maps. The Riemannian distance between subspaces is given by the sum of squared principal angles, and the tangent space is concretely represented for computational purposes.

Asymptotic Characterization of PCA Excess Risk

The first main result is a central limit theorem for the error of the principal subspace estimated by PCA, and the derivation of the asymptotic distribution of the excess risk under the reconstruction loss. Under a finite moment condition and an eigengap assumption ($\lambda_k > \lambda_{k+1}$), the following results are established:

• Consistency: The estimated subspace $[U_n]$ converges in Riemannian distance to the true subspace $[U_*]$.
• Asymptotic Normality: The fluctuations of the estimated subspace around the true subspace are asymptotically normal, with an explicit covariance structure determined by the fourth moments of the data and the eigengaps.
• Excess Risk Distribution: The scaled excess risk $n \cdot (R([U_n]) - R([U_*]))$ converges in distribution to a quadratic form in a Gaussian matrix, with the variance explicitly characterized.

The key property of the data distribution that determines the excess risk is identified as: $$\sum_{i=1}^{d-k} \sum_{j=1}^k \frac{\mathbb{E}[\langle u_{k+i}, X\rangle^2 \langle u_j, X\rangle^2]}{\lambda_j - \lambda_{k+i}}$$ where $u_j$ are the eigenvectors of the population covariance matrix.

The analysis generalizes previous results, which were either restricted to Gaussian data or provided only upper bounds, by giving an explicit asymptotic distribution for general distributions with finite moments.

Self-Concordance of the Block Rayleigh Quotient

A central technical contribution is the proof that the negative block Rayleigh quotient (the population or empirical reconstruction risk) is generalized self-concordant along geodesics emanating from its minimizer, provided the maximum principal angle is less than $\pi/4$. This property is analogous to self-concordance in convex analysis and is crucial for controlling the error in Taylor expansions of the risk function, which underpins the non-asymptotic analysis.

Formally, for $F([V]) = -\frac{1}{2} \mathrm{Tr}(V^\top A V)$, the third derivative along geodesics is controlled by the second derivative, up to a factor depending on the principal angle. This ensures that the second-order Taylor expansion is accurate within a neighborhood of the minimizer, with explicit constants.

Non-Asymptotic Excess Risk Bounds

Building on the self-concordance property, the authors derive a non-asymptotic upper bound on the excess risk of PCA that matches the asymptotic characterization in the large-sample limit. The bound holds with high probability, under a sample size requirement that depends explicitly on the eigengap, the fourth moments of the data, and the desired confidence level.

The non-asymptotic bound takes the form: $$R([U_n]) - R([U_*]) \leq \frac{C}{n \delta} \sum_{i=1}^{d-k} \sum_{j=1}^k \frac{\mathbb{E}[\langle u_{k+i}, X\rangle^2 \langle u_j, X\rangle^2]}{\lambda_j - \lambda_{k+i}}$$ for some explicit constant $C$, provided $n$ exceeds a threshold determined by the data distribution and eigengap. The dependence on $\delta$ (the failure probability) is unimprovable under weak moment assumptions, but can be improved to $\log(1/\delta)$ under boundedness.

The analysis reveals that the sample complexity of PCA is governed by the eigengap and the fourth moments of the data, and that the performance degrades under heavy-tailed distributions, highlighting the need for robust estimation in such settings.

Extensions and Applications

The framework and results extend beyond classical PCA to the estimation of leading eigenspaces of general symmetric matrices, including applications in spectral clustering, community detection, and contrastive learning. The analysis applies to empirical risk minimization problems on the Grassmannian where the loss is a negative block Rayleigh quotient.

The explicit characterization of the excess risk and its dependence on the data distribution provides a benchmark for evaluating and comparing PCA and related algorithms in various statistical and machine learning tasks.

Implications and Future Directions

The geometric and statistical analysis presented in this work clarifies the precise dependence of PCA's excess risk on the data distribution, particularly the interplay between eigengaps and higher-order moments. The self-concordance result opens avenues for improved optimization algorithms on the Grassmannian, potentially enabling faster or more robust convergence for PCA and related problems.

Two main limitations are identified:

1. Reliance on the Eigengap Assumption: The analysis requires a nonzero eigengap, which is mild but excludes degenerate cases where the leading eigenvalues are equal. Extending the results to this setting would require new techniques, as the minimizer set becomes a submanifold.
2. Global Analysis Tightness: The global component of the non-asymptotic analysis is likely loose, and tighter sample complexity bounds may require a deeper understanding of the global geometry of the reconstruction risk.

Potential future directions include:

• Extending the analysis to kernel PCA and functional PCA, which would require infinite-dimensional analogues of the Grassmannian.
• Providing end-to-end guarantees for downstream tasks that use PCA as a preprocessing step.
• Leveraging self-concordance for improved optimization methods on the Grassmannian.

Conclusion

This work provides a rigorous geometric and statistical analysis of PCA, identifying the key distributional property that determines its excess risk, establishing both asymptotic and non-asymptotic characterizations, and introducing a novel self-concordance property for the block Rayleigh quotient. The results have broad implications for the theory and practice of dimensionality reduction, spectral methods, and optimization on manifolds, and suggest several directions for further research in high-dimensional statistics and geometric machine learning.