Metric-Aware Principal Component Analysis (MAPCA):A Unified Framework for Scale-Invariant Representation Learning

Abstract: We introduce Metric-Aware Principal Component Analysis (MAPCA), a unified framework for scale-invariant representation learning based on the generalised eigenproblem max Tr(W^T Sigma W) subject to W^T M W = I, where M is a symmetric positive definite metric matrix. The choice of M determines the representation geometry. The canonical beta-family M(beta) = Sigma^beta, beta in [0,1], provides continuous spectral bias control between standard PCA (beta=0) and output whitening (beta=1), with condition number kappa(beta) = (lambda_1/lambda_p)^1-beta decreasing monotonically to isotropy. The diagonal metric M = D = diag(Sigma) recovers Invariant PCA (IPCA), a method rooted in Frisch (1928) diagonal regression, as a distinct member of the broader framework. We prove that scale invariance holds if and only if the metric transforms as M_tilde = CMC under rescaling C, a condition satisfied exactly by IPCA but not by the general beta-family at intermediate values. Beyond its classical interpretation, MAPCA provides a geometric language that unifies several self-supervised learning objectives. Barlow Twins and ZCA whitening correspond to beta=1 (output whitening); VICReg's variance term corresponds to the diagonal metric. A key finding is that W-MSE, despite being described as a whitening-based method, corresponds to M = Sigma^{-1} (beta = -1), outside the spectral compression range entirely and in the opposite spectral direction to Barlow Twins. This distinction between input and output whitening is invisible at the level of loss functions and becomes precise only within the MAPCA framework.

• The paper introduces MAPCA, a generalization of PCA leveraging a symmetric positive definite metric to achieve strict scale invariance under arbitrary rescaling.
• It constructs a β-family of metrics that provides a continuous trade-off between spectral bias and isotropy, clarifying the connections with self-supervised learning methods.
• Empirical validation on the Army Cadets dataset demonstrates that IPCA uniquely delivers exact scale invariance with a nondegenerate spectral structure.

Metric-Aware Principal Component Analysis (MAPCA): A Unified Framework for Scale-Invariant Representation Learning

Motivation and Limitations of Classical PCA

Traditional Principal Component Analysis (PCA) is a foundational technique for dimensionality reduction and representation learning. However, standard PCA is inherently scale-dependent: its principal components can change dramatically under feature rescaling, leading to instability and potentially misleading representations when variables differ in units or variance. Correlation PCA provides strict scale invariance but discards all variance information, creating a tension between preserving spectral structure and achieving invariance. The root cause of this trade-off is traced to the Euclidean geometry underlying classical formulations.

MAPCA: Generalized Eigenproblem and the Role of the Metric

This work introduces Metric-Aware Principal Component Analysis (MAPCA), generalizing PCA via the parameterization of the constraint with an arbitrary symmetric positive definite metric matrix $M$. The MAPCA optimization is

$$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$

where $\Sigma$ is the sample covariance matrix. The metric $M$ determines the geometry and invariance properties of the learned representation. Standard PCA emerges as $M=I$, correlation PCA as $M=D$ (the diagonal of marginal variances), and full output whitening as $M=\Sigma$. The core theoretical contribution is a precise characterization of scale invariance: MAPCA solutions are strictly invariant to feature rescaling if and only if $M$ transforms equivariantly as $M=C M C$ under diagonal rescaling $C=\operatorname{diag}(c_1,\ldots,c_p)$.

Figure 1: The geometric evolution of the MAPCA family, visualizing metric unit balls and principal directions across choices of $$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$0 and $$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$1.

$$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$2-Family: Spectral Bias Control and Continuous Trade-off

MAPCA introduces the canonical $$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$3-family of metrics $$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$4, with $$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$5, providing a continuous interpolation between standard PCA ($$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$6) and full output whitening ($$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$7). The effective operator is $$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$8, and the associated condition number $$\max_{W} \operatorname{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W^\top M W = I$$9 reduces monotonically as $\Sigma$0 increases. At the endpoints: $\Sigma$1 is maximally scale-sensitive (high spectral bias), $\Sigma$2 is maximally isotropic (scale-invariant).

Intermediate values of $\Sigma$3 compress the spectral gap, interpolating between bias and invariance, but only the endpoints satisfy strict scale invariance under general diagonal rescaling. This is proven via the invariance theorem, leveraging transformation properties of $\Sigma$4.

Diagonal Metric and IPCA: Uniquely Strict Scale Invariance

Setting $\Sigma$5 defines Invariant PCA (IPCA), which occupies a geometrically distinct position within the MAPCA family. IPCA corrects for marginal variances without rotating the metric and achieves strict scale invariance under arbitrary diagonal rescaling (as established by Theorem 1 and its corollaries). IPCA’s spectral structure, governed by the correlation matrix $\Sigma$6, remains nontrivial.

The strict invariance property is unique to IPCA. Other values in the $\Sigma$7-family ($\Sigma$8) fail to achieve invariance except under uniform scaling, since $\Sigma$9 is not equivariant under arbitrary diagonal similarity transformations. Whitening ($M$0) achieves invariance trivially through complete isotropy.

Connections to Self-Supervised Learning and Whitening Techniques

MAPCA provides a geometric language unifying several self-supervised learning (SSL) objectives:

• Barlow Twins: Corresponds to output whitening ($M$1, $M$2), pushing representations to maximal isotropy.
• VICReg's variance term: Interprets as enforcing a diagonal metric ($M$3), correcting marginal variances without enforcing isotropy.
• W-MSE: Performs input whitening, corresponding to $M$4 (i.e., $M$5), which amplifies spectral bias; this direction lies outside the $M$6-family spectral compression range and is theoretically distinct from output whitening techniques.
• ZCA Whitening: Also corresponds to output whitening ($M$7).

MAPCA reveals that these methods target different regions in the metric space, some of which are not directly comparable at the level of loss functions but are clarified in terms of representation geometry and spectral bias.

Empirical Verification: Army Cadets Dataset

MAPCA’s theoretical claims are numerically validated on the Army Cadets dataset (height, weight, chest circumference in both metric and imperial units). The experiment demonstrates:

• Standard PCA: Severe scale dependence; eigenvalues and principal coefficients change radically under unit conversion.
• IPCA (Diagonal Metric): Exact invariance in eigenvalues and correct coefficient scaling to machine precision.
• Intermediate $M$8: Eigenvalues and principal directions do not match across unit systems, confirming strict invariance fails.
• Whitening ($M$9): Eigenvalues invariant but eigenspace comparisons are degenerate due to nonuniqueness.

These results confirm the invariance hierarchy and clarify the practical impact of metric selection within MAPCA.

Theoretical and Practical Implications

The MAPCA framework generalizes classical PCA and connects scale-invariance, spectral bias, and geometric interpretation in a unified formalism. The invariance theorem distinguishes methods with strict invariance (IPCA, whitening) from those with only uniform invariance ($M=I$0-family, intermediate metrics). By clarifying geometric distinctions between input and output whitening, MAPCA provides guidance for principled spectral bias control in representation learning, and underscores that loss-function-level similarities can mask essential geometric differences.

Potential future directions include adaptive selection of $M=I$1 based on task-specific spectral geometry, extensions to kernel or nonlinear covariance operators, exploration of geometric geodesics in the Riemannian cone, and evaluation of the input whitening regime for anomaly detection.

Conclusion

MAPCA offers a unified, rigorously characterized framework that subsumes principal component and whitening-based representation methods. The core theoretical result—a necessary and sufficient condition for strict scale invariance—clarifies the invariance hierarchy for representation learning. The geometric language provided by MAPCA reveals subtle distinctions in contemporary SSL algorithms, especially between input and output whitening, and highlights IPCA’s unique standing as strictly scale-invariant with nontrivial spectral structure. The framework establishes a foundation for principled metric selection and geometry-aware representation learning, enabling systematic exploration of invariance, bias, and isotropy in both classical and modern learning scenarios.