Manifold-Matching Autoencoders

Abstract: We study a simple unsupervised regularization scheme for autoencoders called Manifold-Matching (MMAE): we align the pairwise distances in the latent space to those of the input data space by minimizing mean squared error. Because alignment occurs on pairwise distances rather than coordinates, it can also be extended to a lower-dimensional representation of the data, adding flexibility to the method. We find that this regularization outperforms similar methods on metrics based on preservation of nearest-neighbor distances and persistent homology-based measures. We also observe that MMAE provides a scalable approximation of Multi-Dimensional Scaling (MDS).

• The paper introduces MMAE which augments reconstruction loss with a manifold-matching regularization term to preserve global geometry.
• It aligns pairwise Euclidean distances in the latent space with reference embeddings, thereby maintaining topological fidelity on diverse datasets.
• Experimental results demonstrate that MMAE outperforms traditional methods in metrics like Distance Correlation and Triplet Accuracy, ensuring scalability and robustness.

Manifold-Matching Autoencoders: Distance Alignment for Topology-Preserving Representations

Motivation and Background

Dimensionality reduction is fundamental in data-driven workflows, yet widely-employed autoencoders typically optimize reconstruction error without explicit constraints on the preservation of geometric or topological structure. The manifold hypothesis posits that high-dimensional data resides near a manifold embedded in the ambient space; however, loss of structural information in latent representations can disrupt both interpretability and downstream tasks such as anomaly detection or trajectory visualization. Classical Multidimensional Scaling (MDS) preserves global geometry but is impractical for large datasets due to computational bottlenecks, and nonparametric methods (UMAP, t-SNE, PHATE) often fail to recover nuanced topological relationships (e.g., nesting in high-dimensional nested spheres).

Recent advances have attempted to enforce topology preservation via persistent homology (TopoAE, RTD-AE) or local geometric fidelity (GeomAE, GGAE, SPAE). Topological autoencoders rely on persistent diagrams but suffer from computational and implementation limitations; geometric variants prioritize distortion minimization but may sacrifice global connectivity.

Methodology: MMAE Framework

Manifold-Matching Autoencoders (MMAE) introduce a straightforward, unsupervised distance-based regularization scheme. The approach augments the vanilla autoencoder reconstruction objective with a manifold-matching term ("MM-reg") that directly aligns the pairwise Euclidean distance matrix of the latent representation with a reference distance matrix derived from the input or a specified embedding (e.g., PCA, UMAP):

• For each minibatch, pairwise distances in the latent space $D_z$ are matched via MSE to reference distances $D_E$ (from input $X$ or $u(X)$).
• The reference embedding $u(\cdot)$ can vary in dimensionality and character, enabling global structure regularization even for low-dimensional latent spaces.
• Theoretical underpinning: stability properties of persistent homology and the Gromov-Hausdorff metric ensure that distance preservation bounds topological divergence. Maintaining batch-wise distance fidelity is a practical proxy for global topology, validated by convergence arguments as batch size increases.

The total loss is:

$$\mathcal{L} = \mathcal{L}_{recon} + \lambda \mathcal{L}_{MM}$$

where $\lambda$ controls the trade-off.

Comparative Analysis with Prior Methods

MMAE distinguishes itself as follows:

• Unlike TopoAE and RTD-AE, MMAE does not require persistent homology computation, thus scaling similarly to vanilla autoencoders and geometric methods. Topological regularizations are not strictly required for topology preservation if global metric structure is aligned.
• Unlike GeomAE and GGAE, MMAE focuses on global pairwise geometry rather than local volume distortion or graph Laplacian-based isometry, making it data-agnostic and less sensitive to hyperparameter tuning (e.g., kernel bandwidths).
• In contrast to SPAE, which regularizes the variance of log distance ratios and suffers from instability in high-dimensions, MMAE's direct alignment is more robust to dimensionality and noise (especially under PCA-projected reference spaces).

Experimental Evaluation

Synthetic datasets (nested spheres, linked tori, concentric spheres, Earth, mammoth) and real-world benchmarks (MNIST, FMNIST, CIFAR10, PBMC3K, Paul15) were employed to evaluate structural preservation, using metrics such as Distance Correlation (DC), Triplet Accuracy (TA), KL Density, Wasserstein distance on persistence diagrams (W0), Trustworthiness, and Continuity.

Key numerical findings include:

• On the nested spheres benchmark, MMAE and topological variants were the only methods to recover true nesting in 2D/3D latent spaces; geometric and vanilla variants failed.
• MMAE consistently achieved highest DC, TA, and lowest KL0.1 on linked tori and concentric spheres, maintaining recognizable global structure.
• On 3D datasets (mammoth, Earth), MMAE projections preserved global proportions and relative distances; competing methods produced distortions or lost structural information.
• In real-world gene expression and image datasets, MMAE outperformed or matched RTD-AE, TopoAE, and SPAE in topology-sensitive metrics (W0, DC, TA), especially in high-dimensional and noisy settings due to the flexible reference projection.
• MMAE's computational scaling is on par with vanilla autoencoders and geometric variants, enabling larger batch sizes and efficient training.

Practical and Theoretical Implications

MMAE operationalizes the principle that global geometry preservation is a proxy for topology preservation, aligning with the stability theorems of persistent homology. This offers several implications:

• MMAE can serve as a scalable, parametric analog to classical MDS, providing out-of-sample extensions and batch-wise optimization.
• The method is robust to noise and dimensionality effects (via PCA reference selection), making it suited to small or high-dimensional datasets.
• MMAE enables latent space "copying" from arbitrary source embeddings (UMAP, t-SNE, PCA), facilitating transfer learning or interpretability in complex domains.
• The approach does not unfold manifolds but constrains latent geometry; extensions may involve adaptive regularization schedules or hybridization with topological losses for enhanced connectivity.

Future developments may include randomized MM-reg application, hybrid training schedules (initial MMAE regularization followed by topological refinement), and generative model integration (e.g., VAEs or GANs), leveraging the global structure-preserving properties for improved sample quality and interpolation.

Conclusion

Manifold-Matching Autoencoders provide an efficient, unsupervised framework for global geometry regularization, yielding latent spaces that are topology-aware through metric alignment rather than explicit topological optimization. The approach circumvents computational burdens of persistent homology, facilitates accurate structural preservation across diverse datasets, and enables out-of-sample extension and embedding replication. MMAE demonstrates a promising trajectory for topology-aware representation learning, with broad applicability in visualization, clustering, generative modeling, and scientific data analysis.

(2603.16568)