Generalization of P=2, D=2 MMCR intuition to high-dimensional regimes

Determine the extent to which the closed-form intuition for Maximum Manifold Capacity Representations (MMCR) in the P=2, D=2 case—namely, that maximizing the norm of each mean vector and orthogonalizing the centers maximizes the nuclear norm—extends to general settings with arbitrary numbers of manifolds P and embedding dimensions D in the large-data, high-dimensional regime.

Background

MMCR defines a loss as the negative nuclear norm of a matrix of per-datum centers, motivating representations that increase the nuclear norm. For the special case P=2, D=2, a closed-form solution provides intuition: the nuclear norm is maximized when the mean vectors are unit-norm and orthogonal.

The paper notes that, although this low-dimensional case offers intuition, MMCR was theoretically derived and empirically implemented in regimes with large numbers of data and high embedding dimensions. The authors explicitly state uncertainty about how the P=2, D=2 intuition sheds light on general MMCR behavior, motivating their high-dimensional analysis to address this gap.

References

Yerxa et. al (2023) \citep{yerxa2023learning} note that no closed form solution exists for singular values of an arbitrary matrix, but when $P=2, D=2$, a closed form solution exists that offers intuition: $|C|_*$ will be maximized when (i) the norm of each mean is maximized i.e., $|_p|_2 = 1$ (recalling that $0 \leq |_p| < 1$ since the embeddings live on the hypersphere), and (ii) the means $_1, _2$ are orthogonal to one another. While we commend the authors for working to offer intuition, it is unclear to what extent the $P=2, D=2$ setting sheds light on MMCR in general, as MMCR was theoretically derived and numerically implemented in the large data and high dimension regime.

— Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations  (2406.09366 - Schaeffer et al., 2024) in Section 2.2 (Maximum Manifold Capacity Representations)