Analytical Discovery of Manifold with Machine Learning

Abstract: Understanding low-dimensional structures within high-dimensional data is crucial for visualization, interpretation, and denoising in complex datasets. Despite the advancements in manifold learning techniques, key challenges-such as limited global insight and the lack of interpretable analytical descriptions-remain unresolved. In this work, we introduce a novel framework, GAMLA (Global Analytical Manifold Learning using Auto-encoding). GAMLA employs a two-round training process within an auto-encoding framework to derive both character and complementary representations for the underlying manifold. With the character representation, the manifold is represented by a parametric function which unfold the manifold to provide a global coordinate. While with the complementary representation, an approximate explicit manifold description is developed, offering a global and analytical representation of smooth manifolds underlying high-dimensional datasets. This enables the analytical derivation of geometric properties such as curvature and normal vectors. Moreover, we find the two representations together decompose the whole latent space and can thus characterize the local spatial structure surrounding the manifold, proving particularly effective in anomaly detection and categorization. Through extensive experiments on benchmark datasets and real-world applications, GAMLA demonstrates its ability to achieve computational efficiency and interpretability while providing precise geometric and structural insights. This framework bridges the gap between data-driven manifold learning and analytical geometry, presenting a versatile tool for exploring the intrinsic properties of complex data sets.

• The paper presents the GAMLA framework, employing a two-stage autoencoder to obtain both global parametric mappings and implicit analytical descriptions of low-dimensional manifolds.
• It details the character mapping G that creates a global coordinate system and the complementary mapping R that enforces constraint equations, ensuring R(x) ≈ 0 on the manifold.
• The analytical properties of R enable direct computation of geometric features like normal vectors, enhancing manifold visualization, anomaly detection, and curvature analysis.

The paper "Analytical Discovery of Manifold with Machine Learning" (2504.02511) introduces the GAMLA (Global Analytical Manifold Learning using Auto-encoding) framework. This approach aims to derive both a parametric representation (character representation) and an implicit analytical description (complementary representation) for a low-dimensional manifold $\mathcal{M}$ embedded within a high-dimensional ambient space $\mathbb{R}^n$.

GAMLA Methodology

GAMLA utilizes an autoencoder architecture trained in two sequential rounds. The primary objective is to learn two mappings:

1. Character Mapping ($G$): An encoder $G: \mathbb{R}^n \to \mathbb{R}^m$, where $m$ is the intrinsic dimension of the manifold $\mathcal{M}$. This map provides a global coordinate system for the manifold by projecting data points $\bm{x} \in \mathcal{M}$ onto an $m$-dimensional latent "character space", $\bm{z} = G(\bm{x})$. The corresponding decoder $\hat{G}: \mathbb{R}^m \to \mathbb{R}^n$ generates points $\hat{\bm{x}} = \hat{G}(\bm{z})$ that approximate the manifold, effectively providing a parametric representation.
2. Complementary Mapping ($R$): A map $R: \mathbb{R}^n \to \mathbb{R}^{n-m}$. This map is designed such that for points $\bm{x}$ lying on the manifold $\mathcal{M}$, $R(\bm{x}) = \bm{0}$. This provides $n-m$ constraint equations that implicitly define the manifold $\mathcal{M}$ within the ambient space $\mathbb{R}^n$. The output $\tilde{\bm{z}} = R(\bm{x})$ resides in the $(n-m)$-dimensional "complementary space".

The framework assumes the underlying manifold $\mathcal{M}$ is smooth, compact, and bounded, and that the available data adequately samples this manifold.

Two-Round Training Procedure

The core of GAMLA is its specific two-round training strategy:

Round 1: Character Representation Learning

• Objective: Learn the intrinsic manifold structure and the mapping $G$.
• Architecture: An autoencoder is constructed with an encoder $G$ mapping to an $m$-dimensional bottleneck layer and a decoder $\hat{G}$ mapping back to $\mathbb{R}^n$. Hidden layer dimensions are typically $\ge n$.
• Training Data: The network is trained exclusively using data points sampled from the manifold, denoted as $X_{\mathcal{M}} = \{\bm{x}_i \in \mathcal{M}\}$.
• Loss Function: The standard reconstruction loss is minimized: $$\mathcal{L}_1 = \frac{1}{|X_{\mathcal{M}}|} \sum_{\bm{x} \in X_{\mathcal{M}}} \|\bm{x} - \hat{G}(G(\bm{x}))\|^2$$.
• Outcome: After convergence, the weights and biases of the encoder $G$ and decoder $\hat{G}$ are fixed. This round establishes the parametric representation of the manifold.

Round 2: Complementary Representation Learning

• Objective: Learn the relationship between the manifold and the surrounding ambient space to derive the implicit description $R(\bm{x}) = \bm{0}$.
• Architecture Modification: The bottleneck layer is augmented by adding $n-m$ new nodes. The total latent dimension becomes $n$. The connections from the preceding layer to these new nodes and from these new nodes to the subsequent layer constitute the components that will learn the mapping $R$. The weights learned in Round 1 remain frozen.
• Training Data: The network is now trained using data points sampled uniformly from a larger domain $\mathcal{A} \supset \mathcal{M}$ (e.g., a hyperrectangle enclosing the manifold), denoted as $X_{\mathcal{A}} = \{\bm{x}_i \in \mathcal{A}\}$.
• Training Constraint: Crucially, only the weights and biases associated with the newly added $n-m$ nodes and their connections are trainable. All other parameters (from Round 1) are fixed.
• Loss Function: The reconstruction loss is minimized over the ambient data: $$\mathcal{L}_2 = \frac{1}{|X_{\mathcal{A}}|} \sum_{\bm{x} \in X_{\mathcal{A}}} \|\bm{x} - \hat{\bm{x}}\|^2$$. Here, $\hat{\bm{x}}$ is the output of the modified autoencoder.
• Outcome: This round trains the components representing the map $R: \mathbb{R}^n \to \mathbb{R}^{n-m}$. According to Proposition 1 in the paper, if the first round achieved near-perfect reconstruction for points on the manifold (i.e., $\hat{G}(G(\bm{x})) \approx \bm{x}$ for $\bm{x} \in \mathcal{M}$), then the training process forces the output of the complementary part $R(\bm{x})$ to be approximately zero for points $\bm{x}$ on or very near the manifold.

Representation Derivation and Properties

The two rounds yield the desired representations:

• Character Representation: The encoder $G(\bm{x})$ from Round 1 provides the mapping to the $m$-dimensional character space. The decoder $\hat{G}(\bm{z})$ provides the parametric representation $\hat{\bm{x}} = \hat{G}(\bm{z})$. This offers a global coordinate system for the manifold.
• Complementary Representation: The mapping $R(\bm{x})$ learned by the newly added components in Round 2 provides the implicit analytical description. The manifold $\mathcal{M}$ is approximated by the level set $$\{\bm{x} \in \mathbb{R}^n \mid R(\bm{x}) = \bm{0}\}$$.

Analytical Nature: Since $R$ is realized by neural network components (typically using smooth activation functions like tanh), it is an analytical function. This allows for the direct computation of geometric properties through differentiation.

Global Nature: Both $G(\bm{x})$ and $R(\bm{x})$ provide global descriptions, unlike methods relying on local neighborhoods or charts. $G(\bm{x})$ globally unfolds the manifold, while $R(\bm{x}) = \bm{0}$ defines it globally within $\mathbb{R}^n$.

Applications and Implications

The GAMLA framework enables several applications stemming from its dual representations and analytical nature:

Visualization and Interpretation

• The character coordinates $\bm{z} = G(\bm{x})$ provide a direct low-dimensional embedding suitable for visualization, effectively unfolding complex manifold structures (e.g., Swiss roll).
• The analytical form $R(\bm{x}) = \bm{0}$ potentially offers a more interpretable, "white-box" description of the manifold's constraints compared to purely black-box embedding methods.
• The full latent representation $[\bm{z}, \tilde{\bm{z}}] = [G(\bm{x}), R(\bm{x})]$ decomposes the information about a point $\bm{x}$ into its projection onto the manifold coordinates ($\bm{z}$) and its deviation from the manifold ($\tilde{\bm{z}}$).

Anomaly Detection and Categorization

• Points $\bm{x}$ significantly deviating from the manifold $\mathcal{M}$ will result in $R(\bm{x}) \neq \bm{0}$. The magnitude $\|\tilde{\bm{z}}\| = \|R(\bm{x})\|$ can serve as an anomaly score, quantifying the extent of deviation.
• Beyond detection, the value (vector) $\tilde{\bm{z}} = R(\bm{x})$ in the $(n-m)$-dimensional complementary space provides information about the nature or direction of the deviation from the manifold. This allows for anomaly categorization, distinguishing between different types of outliers based on how they violate the manifold constraints $R(\bm{x}) = \bm{0}$. The paper demonstrates this with mouse phenotyping data, categorizing anomalies based on the sign of a one-dimensional complementary coordinate ($n-m=1$).

Analytical Geometry Computation

• Normal Vectors: Since $$R(\bm{x}) = [R_1(\bm{x}), ..., R_{n-m}(\bm{x})]^T = \bm{0}$$ defines the manifold, the gradient vectors $\nabla R_1(\bm{x}), ..., \nabla R_{n-m}(\bm{x})$ are normal to the manifold at point $\bm{x} \in \mathcal{M}$. These $n-m$ vectors span the normal space $N_{\bm{x}}\mathcal{M}$. The Jacobian matrix $\nabla R(\bm{x}) \in \mathbb{R}^{(n-m) \times n}$ contains these gradients as rows. These can be computed analytically via automatic differentiation of the network components defining $R$.
• Tangent Space: The tangent space $T_{\bm{x}}\mathcal{M}$ is the orthogonal complement of the normal space. Alternatively, for a point $\hat{\bm{x}} = \hat{G}(\bm{z})$ on the reconstructed manifold, the columns of the Jacobian of the decoder $\nabla \hat{G}(\bm{z}) \in \mathbb{R}^{n \times m}$ span the tangent space $T_{\hat{\bm{x}}}\mathcal{M}$.
• Curvature: Higher-order derivatives of $R(\bm{x})$ or $\hat{G}(\bm{z})$ could potentially be used to compute curvature properties, although the paper primarily focuses on normal vectors derived from $R(\bm{x})$.

Conclusion

GAMLA provides a framework for learning manifold representations that bridge data-driven methods with analytical geometry. By employing a two-round autoencoder training strategy, it derives both a parametric character representation $G(\bm{x})$ useful for visualization and global coordinates, and an implicit analytical complementary representation $R(\bm{x})=\bm{0}$ that defines the manifold via constraint equations. The analytical nature of $R(\bm{x})$ facilitates the computation of geometric properties like normal vectors and enables a novel approach to anomaly detection and categorization based on the structure of deviations from the learned manifold.