Riemannian Multi-Manifold Modeling

• Riemannian Multi-Manifold Modeling is a framework that represents data as lying near multiple low-dimensional Riemannian submanifolds within a higher-dimensional space.
• It employs geodesic clustering via tangent space approximations to achieve precise clustering and effective feature extraction with strong theoretical guarantees.
• By integrating methods like normalizing flows, multi-kernel fusion, and intrinsic Riemannian metrics, RMMM enhances performance in applications such as image classification and brain network analysis.

Riemannian Multi-Manifold Modeling (RMMM) is a geometric data modeling framework in which data are assumed to be drawn from, or embedded near, a union of low-dimensional Riemannian submanifolds within a higher-dimensional ambient or feature space. This model generalizes classical linear subspace models to non-Euclidean data domains, including the sphere, Grassmannian, manifold of positive-definite matrices, and data cones parameterized via diffeomorphic flows. RMMM underpins advanced methodologies in clustering, feature extraction, interpolation, and manifold learning for complex datasets, especially where modalities and modes correspond to distinct geometric regimes.

1. Mathematical Foundations and Problem Setting

RMMM postulates that a dataset $X=\{x_i\}_{i=1}^N$ lies on or near a union $\bigcup_{k=1}^K S_k$ of $K$ unknown, smooth, low-dimensional submanifolds $S_k$ of a Riemannian manifold $(\mathcal M,g)$, each of dimension $d_k \ll \dim \mathcal M$ (Wang et al., 2014, Slavakis et al., 2017, Wang et al., 2019). Here, $g$ denotes the Riemannian metric, and $\mathrm{dist}_g$ is the geodesic distance induced by $g$. This abstraction extends the “manifold hypothesis” of high-dimensional data analysis by introducing the possibility of multimodal structure, i.e., multiple manifolds possibly intersecting or being well-separated, with each submanifold reflecting a different latent class, regime, or feature mode.

For time series and network applications, sequential samples $x_t$ are features extracted via windowing and mapped to points in a Riemannian manifold (e.g., the Grassmannian for dynamical subspaces or SPD for correlation structures) (Slavakis et al., 2017). In image set classification and machine learning with multi-modal datasets, each instance or collection yields manifold-valued descriptors (covariance matrices, subspaces, or Gaussians) (Wang et al., 2019).

2. Core Algorithms: Geodesic Clustering and Tangent-Information Methods

A central methodological paradigm in RMMM is “Geodesic Clustering by Tangent Spaces” (GCT) (Wang et al., 2014, Slavakis et al., 2017). The GCT workflow combines intrinsic manifold structure with sparse locality and tangent information:

• Neighbor selection: For each $x_i$, select $r$-nearest neighbors in geodesic distance.
• Logarithm map: Map neighbors to $T_{x_i}\mathcal M$ by $v_{ij} = \log_{x_i}(x_j)$.
• Sparse coding: Represent the origin $0 \in T_{x_i}\mathcal M$ as a sparse affine combination of $\{v_{ij}\}$ with locality-penalized weights.
• PCA-based tangent estimation: Estimate the local tangent subspace $\hat T_{x_i}S$ via eigen-decomposition of the local covariance.
• Directional discrimination: For each neighbor, compute geodesic angles between $v_{ij}$ and $\hat T_{x_i}S$; large angles signal transition to a different submanifold.
• Affinity construction: Define $$W_{ij} = \exp(|S_{ij}|+|S_{ji}|) \exp[-(\theta_{ij}+\theta_{ji})/\sigma_a]$$.
• Spectral clustering: Use normalized Laplacian eigenvectors followed by $K$-means to partition $X$.

This approach leverages both metric (distance-based) and structural (directional, subspace) information, yielding theoretical guarantees for exact recovery even when submanifolds intersect (Wang et al., 2014).

3. RMMM in Manifold Learning with Normalizing Flows

In contemporary machine learning, RMMM can be implemented via a pullback Riemannian geometry induced by a learned diffeomorphism $f: Z \to X$ between a latent space $Z \simeq \mathbb{R}^d$ and data space $X \subset \mathbb{R}^n$ (Diepeveen et al., 12 May 2025). The pullback metric at $z \in Z$ is $g(z) = J_f(z)^T J_f(z)$, where $J_f(z)$ is the Jacobian of $f$.

A tension arises between regularity (constant or slowly varying $g(z)$, stable geodesics) and expressivity (ability of $f$ to represent complicated multi-manifold data). To balance these, architectures combine additive-coupling layers (bounded-derivative activations such as low-order polynomials of $\tanh$) and invertible linear steps (Householder/orthogonal convolutions), maintaining moderate Jacobian norms without sacrificing density estimation capabilities.

Isometrization further refines RMMM by enforcing or approximating $\ell^2$-isometry on the data, i.e., $g(z) \approx I$ in neighborhoods of interest, often via regularization terms $$\mathcal{L}_{iso}(\theta) = \mathbb{E}_{x \sim p_{data}} \|J_f(f^{-1}(x))^T J_f(f^{-1}(x)) - I_d\|_F^2$$. Iso-geodesics and iso-log/exp maps (parameterized re-timings of geodesics to ensure constancy of Euclidean velocity in $X$) yield geodesic distances and principal geodesic analyses that more faithfully respect true data geometry.

This framework enables principled clustering and interpolation in multimodal settings, preserving the distinction between well-separated clusters and yielding interpretable geodesic paths and Riemannian PCA subspaces (Diepeveen et al., 12 May 2025).

4. Riemannian Descriptors, Kernels, and Multi-Manifold Feature Fusion

For image set or pattern recognition, RMMM can employ multiple manifold-valued descriptors—covariance matrices on SPD manifolds, principal subspaces on Grassmannians, Gaussian distributions on information manifolds (Wang et al., 2019). Each descriptor admits intrinsic Riemannian metrics (e.g., Affine-Invariant for SPD, projection metric for Grassmann, log-Euclidean on Gaussian-parameterized SPD):

Descriptor	Manifold	Riemannian Distance
Covariance matrix	$\mathrm{Sym}^+_d$	$\| \log(C_1^{-1/2} C_2 C_1^{-1/2}) \|_F$
Subspace	$\mathcal{G}(q,d)$	$\| \sin \Theta \|_2$
Gaussian	$\mathrm{Sym}^+_{d+1}$	$\| \log P_1 - \log P_2 \|_F$

Kernel functions based on these distances map each descriptor into a reproducing kernel Hilbert space (RKHS). Multi-kernel metric learning frameworks then fuse these representations, adaptively learning projection and gating weights to optimize within/between-class distance objectives in the joint RKHS. This approach exploits complementary geometric features and consistently outperforms single-descriptor or Euclidean methods in classification tasks such as video-based face recognition or dynamic scene analysis (Wang et al., 2019).

5. Theoretical Guarantees and Computational Aspects

RMMM methodologies often provide theoretical guarantees under sampling and regularity conditions. For two geodesic submanifolds with nonzero principal intersection angle, GCT exactly clusters all but an $O(r^{d-\dim(S_1 \cap S_2)})$ fraction of points in a neighborhood of the intersection (with high probability when sparse-coding/radius/angle thresholds are appropriately chosen) (Wang et al., 2014). These results extend to noisy settings under appropriate data conditions.

Computationally, the dominant costs are nearest neighbor search in $\mathrm{dist}_g$, local sparse coding (typically small neighborhood QPs), PCA in tangent spaces, and spectral clustering. Accelerations via approximate neighbor search, sparse Laplacian solvers, and parallelization render the approach scalable to moderately large datasets.

In manifold-learning-with-flows, training proceeds via optimization of a negative log-likelihood loss (for the learned diffeomorphism) with regularization, while downstream Riemannian analysis leverages derived log/exp/isometric operators for statistics and clustering (Diepeveen et al., 12 May 2025).

6. Applications and Empirical Performance

RMMM has been validated on a broad set of tasks:

• Synthetic and Real Data Clustering: GCT achieves $\geq95\%$ clustering accuracy for intersecting/non-intersecting submanifolds in Grassmann, SPD, and spherical data (Wang et al., 2014).
• Brain Network Time Series: RMMM with Grassmann and SPD features clusters brain-state windows from synthetic and real neuroimaging data with near-perfect accuracy, outperforming sparse manifold clustering and spectral Euclidean methods (Slavakis et al., 2017).
• Image Set Classification: Multi-descriptor, multi-kernel fusion architectures yield superior performance in video-based face recognition, object categorization, emotion recognition, and dynamic scene classification, consistently surpassing approaches relying on a single geometric model (Wang et al., 2019).
• Manifold Learning with Normalizing Flows: Regularized flows capture nontrivial multi-manifold geometry, enabling improved interpolation, clustering, and structured generative modeling, as demonstrated on synthetic mixtures, hemispherical manifolds, and MNIST (Diepeveen et al., 12 May 2025).

7. Open Directions and Extensions

Notable extensions and open questions in RMMM include:

• Generalization to non-geodesic and highly curved submanifolds, where tangent-based approximations may require refinement.
• Adaptive, data-driven selection of neighborhood, sparsity, and kernel parameters in unsupervised settings.
• Large-scale scalability via approximate nearest neighbors and low-rank/Nyström embeddings in spectral clustering.
• Robustification against non-Gaussian noise, heavy-tailed outliers, and submanifold intersections with small angles.
• Deeper integration of Riemannian multi-manifold feature modeling with global manifold embedding and representation learning (Wang et al., 2014, Diepeveen et al., 12 May 2025).

RMMM constitutes a rigorous geometric framework for the modeling, analysis, and clustering of complex data exhibiting multi-modal, manifold-structured variability across a spectrum of real-world settings.