Bi-Lipschitz Autoencoder With Injectivity Guarantee

Abstract: Autoencoders are widely used for dimensionality reduction, based on the assumption that high-dimensional data lies on low-dimensional manifolds. Regularized autoencoders aim to preserve manifold geometry during dimensionality reduction, but existing approaches often suffer from non-injective mappings and overly rigid constraints that limit their effectiveness and robustness. In this work, we identify encoder non-injectivity as a core bottleneck that leads to poor convergence and distorted latent representations. To ensure robustness across data distributions, we formalize the concept of admissible regularization and provide sufficient conditions for its satisfaction. In this work, we propose the Bi-Lipschitz Autoencoder (BLAE), which introduces two key innovations: (1) an injective regularization scheme based on a separation criterion to eliminate pathological local minima, and (2) a bi-Lipschitz relaxation that preserves geometry and exhibits robustness to data distribution drift. Empirical results on diverse datasets show that BLAE consistently outperforms existing methods in preserving manifold structure while remaining resilient to sampling sparsity and distribution shifts. Code is available at https://github.com/qipengz/BLAE.

• The paper introduces a novel injectivity guarantee that penalizes latent collisions to maintain topological fidelity during autoencoder training.
• It employs bi-Lipschitz regularization on the decoder to preserve global geometric consistency and reduce required latent dimensionality.
• Empirical evaluations on synthetic, image, and transcriptomics data demonstrate improved reconstruction accuracy and robust manifold learning under distribution shifts.

Bi-Lipschitz Autoencoder With Injectivity Guarantee: A Technical Exposition

Motivation and Problem Statement

Autoencoders constitute a foundational method for nonlinear dimensionality reduction, leveraging the theoretical expressivity of deep neural networks. However, in practice, vanilla autoencoders frequently learn latent representations that are topologically inconsistent with the underlying data manifold. This is largely due to non-injective encoder mappings, which introduce latent collisions, leading to irreversible information loss, suboptimal reconstruction, and failure to preserve manifold geometry. Existing regularization strategies—gradient-based (Jacobian constraints), graph-based (distance preservation), and embedding-based (classic manifold learning alignment)—do not address this injectivity issue in a rigorous or robust fashion. Gradient-based methods often succumb to local minima induced by topological collapses, while graph-based schemes are sensitive to sampling density and distributional drift.

Theoretical Contributions

The paper formalizes encoder injectivity as a critical bottleneck for manifold-preserving embedding with autoencoders. It introduces a theoretically motivated separation criterion—(δ, ε)-separation—guaranteeing injectivity if satisfied for all δ > 0. To promote injectivity during training, a penalty term $\mathcal{L}_{\text{inj}}(\delta, \epsilon)$ is proposed, which penalizes the collapse of distant points in the input manifold to nearby positions in latent space, complemented by a non-expansiveness constraint to avoid trivial scaling solutions.

Beyond injectivity, the work advocates for regularizations that are admissible: that is, minimization solutions should be independent of the data distribution. Via sufficient conditions on the loss function, the authors prove that injectivity and geometric regularity losses can be made admissible, guaranteeing robustness to distribution shifts—a key requirement for generalizable manifold learning.

A significant insight concerns the limitations of isometric regularization (i.e., enforcing local isometry via Jacobian or spectral norm constraints): Nash's embedding theorem ensures that isometric embeddings generally require $O(m^2)$-dimensional latent spaces for intrinsic manifold dimension $m$. To resolve this inefficiency and non-admissibility in low dimensions, the paper introduces a bi-Lipschitz relaxation. A mapping is $k$-bi-Lipschitz if distances in latent space and on the input manifold are preserved up to a factor $k$, formalized via bounds on the singular values of the decoder's Jacobian. The theoretical development shows that for compact $k$-dimensional Riemannian manifolds, $k$-bi-Lipschitz embeddings exist in as low as $2k$ dimensions, markedly reducing latent dimensionality compared to isometric embeddings.

Methodology: Bi-Lipschitz Autoencoder (BLAE)

BLAE introduces two complementary regularization modules for the autoencoder training objective:

1. Injective Regularization: Enforces the (δ, ε)-separation criterion for encoder mappings, eliminating topological collapses and associated suboptimal minima in the loss landscape.
2. Bi-Lipschitz Regularization: Applies to the decoder, penalizing violation of $k$-bi-Lipschitz conditions via the minimal/maximal singular values of the local Jacobian, ensuring global geometric consistency and distributional robustness with only linear latent dimensionality requirements.

The full BLAE objective is:

$$\mathcal{L}_{\text{BLAE}} = \mathcal{L}_{\text{recon}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}} + \lambda_{\text{bi-Lip}} \mathcal{L}_{\text{bi-Lip}},$$

where the regularization terms interplay to ensure both injectivity and geometric fidelity.

Notably, the manuscript underscores that the injective term synergizes with, but cannot be substituted by, traditional gradient-based or graph-based geometry regularizations. Through ablation, it is shown that injectivity is necessary (but not sufficient) for high-fidelity geometric preservation.

Empirical Evaluation

Experiments are conducted across synthetic (Swiss Roll), structured image (dSprites, MNIST), and real-world single-cell transcriptomics datasets. Evaluation metrics focus on reconstruction MSE, neighborhood preservation (k-NN recall), and multiscale geometry similarity (KL divergence over graph-derived geodesics), explicitly leveraging non-Euclidean metrics suited to manifold structure.

Numerical results indicate that BLAE consistently outperforms graph-based and gradient-based baselines:

• On Swiss Roll, BLAE attains lowest KL divergence and highest k-NN recall, indicating preservation of both local and global geometry even with sparse samples.
• In the MNIST rotation-scale manifold, only BLAE yields concentric latent embeddings robust to both uniform and non-uniform training distribution—demonstrating strong invariance to distribution shift.
• For dSprites and scRNA-seq, BLAE achieves superior out-of-distribution generalization and cell type separability.

Critical empirical findings include:

• Gradient-based autoencoders augmented with injective regularization escape pathologically collapsed minima and achieve proper topological unfolding.
• Combining SPAE-like global distance constraints with bi-Lipschitz regularization is inferior: rigid graph-distance preservation is highly sensitive to approximation errors, while the selective separation penalty of BLAE tolerates graph noise and integrates smoothly with smoothness regularization.

Extension to Probabilistic Models

BLAE regularization extends naturally to VAEs, where its integration (BL-VAE) yields both lower reconstruction error and higher mutual information gap (MIG)—indicative of better disentanglement—compared to standard VAEs, confirming the generality of the proposed injectivity and admissibility framework.

Implications and Future Directions

The paper resolves key obstacles in nonlinear dimension reduction with neural autoencoders:

• It formalizes the topological requirements for geometry-preserving embedding, identifying injectivity as a prerequisite for robust optimization and latent fidelity.
• The bi-Lipschitz relaxation model sharply reduces the theoretical demand for latent dimensionality while maintaining geometric consistency and robustness.
• Admissible regularization, strong separation penalties, and efficient approximations are shown to be crucial for scalability and extendibility to large, high-dimensional, or non-uniform real-world datasets.

Practically, BLAE presents a robust protocol for manifold learning, representation learning, and as a plug-in module to enhance generative modeling pipelines. A notable limitation is the quadratic time/space complexity for geodesic matrix computation, addressed via sampling strategies for scaling to larger datasets.

Theoretically, the work suggests that future research should further harmonize topological/topographic constraints with deep learning optimization, as well as develop scalable estimators for intrinsic distances and tangent spaces to improve regularization accuracy in highly nonlinear and high-dimensional domains.

Conclusion

Bi-Lipschitz Autoencoder with injectivity guarantee offers a principled synthesis of topology, geometry, and optimization for neural manifold learning. By unifying injective separation and bi-Lipschitz regularization, it achieves latent representations that are both topologically and geometrically faithful, robust to sampling artifacts and distributional shifts, and computationally feasible for practical datasets. The approach represents a substantial advance in ensuring the theoretical promises of deep autoencoders are met in realistic settings, reliably unlocking their use for scientific and engineering applications involving nonlinear, high-dimensional data (2604.06701).