Riemannian Manifold Transformer

• The paper introduces a Riemannian Manifold Transformer that integrates a geometry-aware projector and a Mixture-of-Experts layer, achieving up to a 3% accuracy boost on benchmarks.
• Manifold-aware designs preserve data structure by employing techniques such as QR, SVD, and Log-Euclidean mappings to maintain intrinsic geometric features in graph and SPD data.
• This approach combines Euclidean and non-Euclidean features, adaptively routing nodes based on local topology to capture heterogeneous and curved data spaces.

Searching arXiv for the primary paper and closely related manifold-aware transformer work. A Riemannian Manifold Transformer denotes a transformer-based model in which representation learning is explicitly organized around non-Euclidean geometry rather than a single latent Euclidean space. In the graph setting, the clearest recent instantiation is a graph transformer augmented with a lightweight Riemannian Mixture-of-Experts (MoE) layer that routes each node to manifold types such as spherical, flat, and hyperbolic according to local topology, thereby providing both greater representational flexibility and intrinsic geometric explanations of the latent space (Jyothish et al., 9 Jul 2025). In a broader sense, the term also covers transformer architectures that preserve or exploit manifold structure throughout computation, including transformers for sequences of Symmetric Positive Definite matrices under Log-Euclidean geometry (Seraphim et al., 2023), and transformer analyses formulated on quotient manifolds using Riemannian geometry to account for attention symmetries (Silva et al., 8 May 2025). Taken together, these works place transformers within a geometric program in which curvature, pullback structure, quotient structure, and manifold-aware projection are treated as first-class modeling objects.

1. Conceptual scope and definition

The central motivation is that standard transformers operate in Euclidean vector spaces, whereas many data modalities have heterogeneous, curved, or otherwise non-Euclidean structure. In the graph case, the stated problem is that standard graph transformers embed all node features in a single Euclidean space, which cannot capture the rich, heterogeneous, and often curved topology of real-world graphs (Jyothish et al., 9 Jul 2025). The corresponding remedy is to prepend a geometry-aware projector that adaptively maps each node to the manifold type best matching its local structure.

Within this usage, the most direct exemplar is R-SGFormer, which prepends a geometry-aware projector—Stiefel, Grassmann, or Riemannian MoE—to a scalable transformer backbone, SGFormer (Jyothish et al., 9 Jul 2025). The projector is lightweight, node-wise, and adaptive. Its purpose is not merely to replace Euclidean embeddings, but to coexist with them: the ensemble makes sure that both Euclidean and non-Euclidean features are captured (Jyothish et al., 9 Jul 2025).

A broader reading of the term is supported by related transformer research. SPDTransNet is a Transformer architecture for sequences of SPD matrices that preserves their Riemannian geometry throughout the analysis by working in the Log-Euclidean representation and restricting operations to structure-preserving ones (Seraphim et al., 2023). Separately, recent sharpness analysis for transformers argues that meaningful perturbation geometry must be defined on a quotient manifold induced by transformer symmetries, rather than in the raw Euclidean parameter space (Silva et al., 8 May 2025). This suggests that “Riemannian Manifold Transformer” can refer either to a concrete architecture with manifold-aware layers or to a more general transformer paradigm whose data representations, optimization geometry, or theoretical analysis are formulated on manifolds.

2. Geometric foundations

The graph-based formulation explicitly draws on several manifold families. The Stiefel manifold $V_k(\mathbb{R}^n)$ consists of orthogonal $k$-frames in $\mathbb{R}^n$, with $X^\top X = I_k$, and is used to enforce orthogonality in features, which is described as beneficial for attention mechanisms (Jyothish et al., 9 Jul 2025). The Grassmann manifold $G_k(\mathbb{R}^n)$ is the set of $k$-dimensional subspaces, expressed as equivalence classes $[X] = \{XQ : Q \in O(k)\}$, and encodes subspace structure while ignoring basis order (Jyothish et al., 9 Jul 2025). The same paper also places these within the larger category of general Riemannian manifolds $(\mathcal{M}, g)$, including spaces with non-zero curvature such as hyperbolic spaces (Jyothish et al., 9 Jul 2025).

A second geometric strand comes from SPD-valued sequence modeling. The set $SPD(n)$ forms a nonlinear Riemannian manifold, and the Log-Euclidean metric is used to map SPD matrices to the vector space of symmetric matrices through the matrix logarithm, yielding

$$\delta_{LE}(A, B) = \lVert \log_{mat}(A) - \log_{mat}(B) \rVert_2 .$$

Because this mapping is bijective, computations in $k$0 can be interpreted geometrically on the SPD manifold (Seraphim et al., 2023). This is a different design pattern from graph-manifold routing, but it serves the same principle: transformer computation should remain compatible with the intrinsic geometry of the objects being processed.

A third strand is theoretical. A singular Riemannian geometry approach to deep networks treats a network as a sequence of smooth maps between manifolds, with pullback metrics inducing possibly degenerate geometry on earlier layers (Benfenati et al., 2021). In that formulation, quotient structures and vertical bundles describe invariances and equivalence classes of inputs mapped to the same output. This suggests a conceptual bridge to manifold-aware transformers: once a transformer is regarded as a sequence of maps between manifolds, one can ask whether each block preserves, distorts, or quotients geometric structure.

3. Architecture and routing mechanisms

In the graph formulation, the defining mechanism is the Riemannian Mixture-of-Experts layer. Each node’s local topology is measured, for example by sampling its subgraph and extracting descriptors, and those descriptors are used to compute gating weights over a set of $k$1 Riemannian experts (Jyothish et al., 9 Jul 2025). The gating rule is

$k$2

with $k$3, so that each node receives a convex mixture over experts (Jyothish et al., 9 Jul 2025). The resulting node embedding is

$k$4

where $k$5 is the embedding function of the selected manifold expert (Jyothish et al., 9 Jul 2025).

Projection into specific manifolds is implemented by structure-preserving linear-algebraic operations. For the Stiefel case, projection is performed via QR decomposition: $k$6 and only the $k$7 factor is retained as the mapped feature (Jyothish et al., 9 Jul 2025). For the Grassmann case, projection is performed via SVD,

$k$8

and the left singular vectors $k$9 span the target subspace (Jyothish et al., 9 Jul 2025). The framework also introduces pairwise alignment, described as normalizing distances across different manifolds to ensure global geometric consistency (Jyothish et al., 9 Jul 2025).

The projector is integrated into the transformer backbone through feature fusion. In R-SGFormer, the original features and the Riemannian embeddings are fused through a cross-attention block and then passed to the main transformer; a regularizer ensures that the output remains structured (Jyothish et al., 9 Jul 2025). This yields a hybrid design rather than a fully manifold-only stack. The paper explicitly attributes part of the method’s effect to the ensemble structure, which preserves both “flat” and “curved” signals (Jyothish et al., 9 Jul 2025).

For SPD-valued data, the architectural analogue is Structure-Preserving Multi-Head Attention (SP-MHA). There, standard concatenation is avoided; instead, attention maps are combined via a linear weighted sum so that each token remains equivalent to a point on the SPD manifold via the exp/log correspondence (Seraphim et al., 2023). Triangular linear maps act only on independent symmetric-matrix entries, ensuring that outputs stay within symmetric matrix space and that proximity under the Log-Euclidean metric is respected (Seraphim et al., 2023). The architectural commonality with graph-manifold routing is that the attention mechanism is not left geometrically unconstrained.

4. Structure preservation, regularization, and interpretability

A recurrent theme is that manifold-aware transformers are not defined solely by non-Euclidean embeddings, but by explicit structure preservation. In the graph setting, orthogonality or subspace regularization is used to maintain manifold structure. For the Stiefel case, the loss includes

$\mathbb{R}^n$0

where $\mathbb{R}^n$1 is the learned node representation (Jyothish et al., 9 Jul 2025). A similar constraint is described for Grassmann outputs (Jyothish et al., 9 Jul 2025). The stated role of these regularizers is to keep representations in a meaningful geometric subspace and improve generalization.

Interpretability is framed geometrically. The MoE layer provides intrinsic geometric meaning to node representations because each node’s embedding is a mixture of manifold experts, and the expert weights are interpretable as probabilities reflecting local topology (Jyothish et al., 9 Jul 2025). The paper further states that the benefit, or lack thereof, of orthogonal or subspace projections in validation loss gives practitioners a concrete test for which manifold is most appropriate for their data (Jyothish et al., 9 Jul 2025). This is a stronger interpretability claim than conventional feature attribution: the representation itself is given a geometric explanation.

The SPD sequence work makes a related claim at the operator level. Because tokens are mapped into log-space and every transformer block is restricted to operations compatible with the SPD manifold, the geometry is preserved from input to classification head (Seraphim et al., 2023). This offers a different form of interpretability, namely that each intermediate token continues to correspond to a valid geometric object rather than an unconstrained Euclidean surrogate.

At the theoretical level, the singular Riemannian framework provides language for understanding why structure-preserving designs matter. Pulling back an output metric through a network can induce degenerate metrics on earlier manifolds, giving rise to pseudometrics, equivalence classes, and Kolmogorov quotients (Benfenati et al., 2021). A plausible implication is that manifold-aware transformer layers can be interpreted as attempts to control or exploit these induced geometric structures instead of allowing them to arise implicitly and opaquely.

5. Empirical findings and benchmark behavior

The graph-manifold study reports experiments on node-classification benchmarks including Cora, Citeseer, PubMed, Airport, Actor, Squirrel, Chameleon, Deezer, using Weighted-F1 and Macro-F1 accuracy as metrics (Jyothish et al., 9 Jul 2025). The compared methods include GCN, GAT, SAGE, HGCN, Q-GCN, MotifRGCN, GraphMoRE with various backbones, and baseline SGFormer (Jyothish et al., 9 Jul 2025). The primary result is that R-SGFormer (with GraphMoRE) consistently outperforms all baselines, with example scores of Cora: Weighted-F1 82.44%, Macro-F1 80.66%, Airport: Weighted-F1 93.53%, Macro-F1 93.18%, and PubMed: Weighted-F1 81.01%, Macro-F1 80.21% (Jyothish et al., 9 Jul 2025). The reported performance uplift is up to 3% accuracy boost over strong baselines (Jyothish et al., 9 Jul 2025).

The same paper emphasizes that the manifold choice is data-dependent. The Stiefel and Grassmann variants each excel on different datasets depending on intrinsic graph geometry; Stiefel is described as best when data is nearly orthogonal, and Grassmann for rank-invariant tasks (Jyothish et al., 9 Jul 2025). This does not imply a single optimal manifold family across benchmarks. Rather, it supports the adaptive routing premise of the MoE formulation.

For SPD-valued sequences, SPDTransNet is evaluated on automatic sleep staging using the MASS SS3 dataset, with Macro-F1, macro accuracy, and class-wise F1 as reported metrics (Seraphim et al., 2023). The paper states that SPDTransNet achieves competitive or superior results in macro-F1 and N1 F1 relative to both a vanilla Transformer and several state-of-the-art baselines, and that replacing SP-MHA with classic non-structure-preserving MHA yields worse performance (Seraphim et al., 2023). The explicit interpretation given is that strict preservation of SPD structure improves representation quality.

A distinct empirical line concerns transformer generalization rather than representation tasks. Quotient-manifold sharpness analysis reports that geodesic sharpness improves correlation with generalization for real-world transformers on both text and image classification tasks, whereas adaptive sharpness performs weakly or inconsistently (Silva et al., 8 May 2025). Although this is not an architecture paper, it strengthens the case that Riemannian geometry is not only a representational device but also a tool for analyzing transformer behavior in a symmetry-aware manner.

6. Relation to adjacent research programs

The term intersects several neighboring research directions without being identical to any one of them. One nearby line concerns universal metric embeddings implemented by small transformers. A probabilistic transformer can bi-Hölder embed any $\mathbb{R}^n$2-point dataset from an arbitrary metric space, and for datasets from compact Riemannian manifolds stronger bi-Lipschitz guarantees are obtained, with effective dimension depending on intrinsic manifold dimension rather than dataset size (Kratsios et al., 2022). In that work, the representation target is a space of Gaussian mixtures with a transport metric, and the transformer's role is to realize a geometry-preserving embedding rather than to perform manifold routing in the latent space.

Another nearby line concerns learned pullback geometry via normalizing flows. There, the goal is to learn a Riemannian structure in ambient space whose geodesic subspace aligns with the intrinsic data manifold, while addressing distortions and modeling errors through isometrization and a balance between regularity and expressivity of the diffeomorphism parameterization (Diepeveen et al., 12 May 2025). The paper explicitly describes these developments as relevant to future scalable and interpretable manifold-based deep learning models, including transformer-based models operating on learned Riemannian geometry (Diepeveen et al., 12 May 2025). This suggests a division of labor: normalizing flows can learn the geometry, and a manifold-aware transformer can consume or refine it.

The singular Riemannian geometry framework provides a more abstract umbrella. By modeling a deep network as a sequence of maps between manifolds and analyzing pullback-induced pseudometrics and quotient manifolds, it offers a general language for invariance, robustness, and equivalence classes in neural computation (Benfenati et al., 2021). This suggests that the manifold-aware transformer literature can be read as a concrete specialization of a broader geometric view of deep architectures.

7. Open interpretations and conceptual boundaries

The current literature supports a precise but relatively narrow meaning for the term. The graph paper explicitly describes a geometry-aware input projector based on a Riemannian Mixture-of-Experts layer attached to a scalable graph transformer, and the paper summary states that it pioneers “Riemannian Manifold Transformers” for graphs (Jyothish et al., 9 Jul 2025). At the same time, the available works do not establish a single standardized architecture family under that name across all modalities. A plausible implication is that the term presently functions more as a design pattern than as a fixed model class.

One common misconception is that any transformer processing manifold-valued data is automatically a Riemannian Manifold Transformer. The cited works suggest a stricter criterion: the geometry must be explicitly modeled, preserved, or used for routing. In the graph case, this occurs through node-wise manifold assignment and projection (Jyothish et al., 9 Jul 2025). In SPD sequence modeling, it occurs through SP-MHA, triangular linear maps, and the log-domain formulation that preserve manifold structure end-to-end (Seraphim et al., 2023). In symmetry-aware sharpness analysis, it occurs by redefining perturbation geometry on a quotient manifold rather than in the raw parameter space (Silva et al., 8 May 2025).

Another misconception is that manifold-aware design necessarily replaces Euclidean representations. The graph formulation states the opposite: the ensemble makes sure that both Euclidean and non-Euclidean features are captured (Jyothish et al., 9 Jul 2025). This suggests that current practice favors hybridization over exclusivity, especially when real-world data contain both approximately flat and strongly curved local structure.

More broadly, the literature indicates three converging trajectories. One trajectory is architecture, in which manifold-aware projectors or structure-preserving attention layers are inserted into transformer pipelines (Jyothish et al., 9 Jul 2025, Seraphim et al., 2023). Another is theory, in which quotient manifolds, pullback geometry, and singular Riemannian structures are used to analyze transformer invariances and neural equivalence classes (Silva et al., 8 May 2025, Benfenati et al., 2021). The third is representation geometry, in which transformers serve as universal metric embedding mechanisms or downstream consumers of learned pullback geometry (Kratsios et al., 2022, Diepeveen et al., 12 May 2025). Taken together, these works define the present research landscape of the Riemannian Manifold Transformer: a transformer paradigm in which geometry is treated as latent structure to be preserved, selected, or made explicit rather than flattened away.