Riemannian Mixture-of-Experts Layer

• The Riemannian Mixture-of-Experts layer is a geometric neural module that leverages multiple constant-curvature manifolds to address graph topological heterogeneity.
• It employs a learned gating mechanism to weight expert outputs based on local topology, enabling curvature-specific adaptation in models like GraphMoRE and GAD-MoRE.
• The module integrates manifold-aware operations, routing strategies, and reconstruction losses to produce metric-faithful embeddings and support effective anomaly detection.

Searching arXiv for the named papers and closely related Riemannian mixture/product-manifold graph representation work. I’ll look up the specific arXiv entries and a few directly related graph-geometry papers to ground the article. A Riemannian Mixture-of-Experts layer is a geometric neural module that distributes a representation across multiple constant-curvature manifolds, assigns expert-specific weights through a learned gating or routing mechanism, and combines the resulting outputs into a mixed embedding or reconstruction. In recent graph learning, this construction is used precisely where a single curvature space is insufficient: GraphMoRE applies a mixture of Riemannian experts to mitigate topological heterogeneity in graph representation learning, while GAD-MoRE uses specialized Riemannian experts to support zero-shot generalizable graph anomaly detection by scoring inputs in the geometry where their anomaly signal is most detectable (Guo et al., 2024, Zhao et al., 6 Feb 2026).

1. Geometric motivation and problem setting

The central motivation for a Riemannian Mixture-of-Experts layer is the observation that graph data frequently exhibits heterogeneous geometric structure. GraphMoRE states that real-world graphs have inherently complex and diverse topological patterns, termed topological heterogeneity, and argues that learning graph representations in a single constant-curvature space is insufficient to match these complex geometric shapes, leading to low-quality embeddings with high distortion. It further states that, although recent studies indicate that a product manifold may help address topological heterogeneity, the product manifold remains homogeneous and is therefore inadequate and inflexible for representing mixed heterogeneous topology (Guo et al., 2024).

GAD-MoRE advances a closely related geometric argument in the anomaly-detection setting. It states that anomaly detectability is highly dependent on underlying geometric properties and that embedding graphs from different domains into a single static curvature space can distort the structural signatures of anomalies. Its proposed remedy is again a mixture of constant-curvature experts, but now with the explicit aim that each anomaly pattern be modeled in the Riemannian space where it is most detectable (Zhao et al., 6 Feb 2026).

Taken together, these formulations define the Riemannian Mixture-of-Experts layer as a response to curvature mismatch. A single geometry is treated not as a universal latent substrate, but as a restricted hypothesis class. The layer therefore replaces global geometric uniformity with input-dependent expert allocation. This suggests an interpretation of the mechanism as a localized curvature-selection procedure, although the two papers operationalize that idea differently.

2. Core architectural pattern

Across the two formulations, the layer consists of $K$ experts, each attached to a manifold $\mathcal M_{\kappa_i}$ or $\mathcal M^d_{\kappa_i}$ with constant curvature $\kappa_i$. Each expert receives an input representation, performs manifold-aware computation in its own curvature space, and contributes an output weighted by a gating or routing network. The common structure is therefore a geometric MoE, but the implementation details differ materially between representation learning and anomaly detection (Guo et al., 2024, Zhao et al., 6 Feb 2026).

Aspect	GraphMoRE	GAD-MoRE
Expert type	Riemannian GNN on $\mathcal M^d_{\kappa_i}$	Small manifold-aware MLP on $\mathcal M_{\kappa_i}$
Router	Topology-aware gating MLP with softmax weights	Memory-based dynamic router with sparse top-$k$ selection
Output combination	Weighted manifold scalar multiplication, then concatenation in product-manifold form	Weighted sum of selected experts’ tangent-space reconstructions

In GraphMoRE, each expert is implemented as a Riemannian GNN, for example using the $\kappa$-stereographic model, and its forward pass maps Euclidean node features into $\mathcal M^d_{\kappa_i}$ before message passing in that manifold. The gating network $\phi$ produces a probability vector $\mathcal M_{\kappa_i}$0, which determines how much each expert is trusted for a node (Guo et al., 2024).

In GAD-MoRE, each expert is specialized to a distinct constant-curvature manifold and the router computes a sparse top-$\mathcal M_{\kappa_i}$1 mixture so that only the most relevant experts are activated per node. Each expert receives an input embedding in Euclidean tangent space, projects it onto its manifold, processes it with manifold-aware layers, and maps the output back to tangent space. The final reconstructed embedding is the weighted sum of the selected experts’ outputs, and the anomaly score is the reconstruction error (Zhao et al., 6 Feb 2026).

A common misconception is that a Riemannian Mixture-of-Experts layer is simply a conventional MoE with non-Euclidean embeddings substituted into its experts. In these formulations, the geometric operations are not incidental: the gating objective, the distance definitions, the fusion rule, and the loss design are all explicitly curvature-aware.

3. GraphMoRE: topology-aware gating, manifold experts, and aligned distances

GraphMoRE defines $\mathcal M_{\kappa_i}$2 experts on $\mathcal M_{\kappa_i}$3-dimensional constant-curvature manifolds

$\mathcal M_{\kappa_i}$4

Its gating mechanism is topology-aware rather than feature-only. For each node $\mathcal M_{\kappa_i}$5, a multi-resolution local topology encoding is first extracted: $\mathcal M_{\kappa_i}$6 where $\mathcal M_{\kappa_i}$7 is the induced subgraph of radius $\mathcal M_{\kappa_i}$8, $\mathcal M_{\kappa_i}$9 is a small GNN encoder, and concatenation is taken over radii $\mathcal M^d_{\kappa_i}$0. A gating MLP then produces logits $\mathcal M^d_{\kappa_i}$1, followed by a softmax

$\mathcal M^d_{\kappa_i}$2

This construction ties expert assignment to local topology rather than only to raw node attributes (Guo et al., 2024).

Within expert $\mathcal M^d_{\kappa_i}$3, Riemannian message passing proceeds by mapping manifold states to the tangent space at the origin through $\mathcal M^d_{\kappa_i}$4, aggregating and combining there, and mapping back with $\mathcal M^d_{\kappa_i}$5. If $\mathcal M^d_{\kappa_i}$6 denotes the layer-$\mathcal M^d_{\kappa_i}$7 state, GraphMoRE writes

$\mathcal M^d_{\kappa_i}$8

followed by

$\mathcal M^d_{\kappa_i}$9

After $\kappa_i$0 layers, the expert output is $\kappa_i$1.

Fusion is performed by a weighted manifold “scalar multiplication”

$\kappa_i$2

followed by concatenation across experts: $\kappa_i$3 The explicit interpretation given is that each expert’s embedding is pulled back to tangent space, scaled by $\kappa_i$4, and pushed forward again onto the manifold, so experts with small $\kappa_i$5 are down-weighted.

A further technical difficulty is cross-space distance comparison. GraphMoRE therefore introduces an alignment strategy for node pairs. For nodes $\kappa_i$6, it defines aligned expert weights

$\kappa_i$7

and then the squared mixed distance

$\kappa_i$8

This quantity is compared to graph shortest-path distance $\kappa_i$9 via the distortion loss

$\mathcal M^d_{\kappa_i}$0

Because $\mathcal M^d_{\kappa_i}$1 depends on the gating weights, minimizing $\mathcal M^d_{\kappa_i}$2 trains the gating network to put more mass on experts whose curvature better preserves local topology (Guo et al., 2024).

4. GAD-MoRE: multi-curvature alignment, memory routing, and reconstruction-based scoring

GAD-MoRE instantiates the Riemannian Mixture-of-Experts layer in a different regime: unsupervised zero-shot graph anomaly detection. Its experts live in three types of constant-curvature geometries—Euclidean $\mathcal M^d_{\kappa_i}$3, hyperbolic $\mathcal M^d_{\kappa_i}$4, and spherical $\mathcal M^d_{\kappa_i}$5—with explicit distance and origin-centered exponential/logarithmic maps specified for each. In practice, the paper initializes the $\mathcal M^d_{\kappa_i}$6 experts with curvatures

$\mathcal M^d_{\kappa_i}$7

so as to cover Euclidean, hyperbolic, and spherical geometries (Zhao et al., 6 Feb 2026).

Before routing, GAD-MoRE introduces an anomaly-aware multi-curvature feature alignment module. Raw features $\mathcal M^d_{\kappa_i}$8 are described as geometry-agnostic and are aligned in two stages. First, a tangent-space projection is applied for each curvature index $\mathcal M^d_{\kappa_i}$9: $\mathcal M_{\kappa_i}$0 If $\mathcal M_{\kappa_i}$1, these projected features are lifted to the manifold and retracted back to the tangent space at the origin by

$\mathcal M_{\kappa_i}$2

while for $\mathcal M_{\kappa_i}$3, $\mathcal M_{\kappa_i}$4. Second, PCA is performed on $\mathcal M_{\kappa_i}$5, and candidate feature dimensions are ranked by the Laplacian score

$\mathcal M_{\kappa_i}$6

with the top-$\mathcal M_{\kappa_i}$7 dimensions of smallest score selected to form $\mathcal M_{\kappa_i}$8. These aligned features are concatenated and used as input to the GNN backbone.

Routing is memory-based and performance-conditioned. After the GNN backbone and residual construction, each node has an Euclidean embedding $\mathcal M_{\kappa_i}$9. Each expert $k$0 maintains a memory bank $k$1 of past embeddings it reconstructed well. The raw routing logit is

$k$2

which measures how close the current node is to the expert’s specialty region. These logits are converted to temperature-controlled mixture weights and then sparsified to top-$k$3 experts: $k$4 followed by $k$5 outside the top-$k$6 set $k$7 and renormalization within $k$8.

Expert $k$9 reconstructs the node representation by

$\kappa$0

and the final reconstructed embedding is

$\kappa$1

The node-level anomaly score is the Euclidean reconstruction error

$\kappa$2

Unlike GraphMoRE, which fuses expert outputs into a product-manifold representation, GAD-MoRE maps expert outputs back to tangent space and forms a weighted Euclidean reconstruction. The layer is therefore not tied to a single fusion regime; its defining feature is the routing across curvature-specialized experts, not a particular downstream decoder (Zhao et al., 6 Feb 2026).

5. Training objectives, optimization, and differentiability

In GraphMoRE, training combines a task-specific decoder loss $\kappa$3 with the distortion term $\kappa$4. The pseudo-code describes the total loss as

$\kappa$5

This makes geometry preservation an explicit optimization target rather than a by-product of task supervision. The paper also remarks that softmax and MLP operations are Euclidean, while gradients through $\kappa$6, $\kappa$7, Möbius addition, and related operations are obtained automatically by differentiating their closed-form formulas; modern frameworks identified in the description are PyTorch + geoopt, with support for Riemannian Adam (Guo et al., 2024).

GAD-MoRE is trained end-to-end without labels and uses a weighted sum of five losses: embedding reconstruction $\kappa$8, feature reconstruction $\kappa$9, structure reconstruction $\mathcal M^d_{\kappa_i}$0, structure-contrastive $\mathcal M^d_{\kappa_i}$1, and gate entropy regularization $\mathcal M^d_{\kappa_i}$2. The total objective is

$\mathcal M^d_{\kappa_i}$3

Optimization is by Adam over 40 epochs, and the memory banks are updated dynamically after a cold-start period by keeping high-quality reconstructions as prototypes (Zhao et al., 6 Feb 2026).

These two training regimes expose an important design split. In GraphMoRE, routing is directly shaped by a graph-distance distortion criterion. In GAD-MoRE, routing is shaped by reconstruction fidelity, memory proximity, and gate regularization. A plausible implication is that the same abstract layer can be specialized either for metric-faithful representation learning or for reconstruction-based detection, depending on which training signal determines expert assignment.

6. Relations, distinctions, and common misunderstandings

One recurring misunderstanding is that a product manifold is already sufficient to model heterogeneity. GraphMoRE explicitly rejects this position, stating that the product manifold is still homogeneous and therefore inadequate for representing mixed heterogeneous topology. Its contribution is not merely to concatenate multiple manifold coordinates, but to create personalized mixed curvature spaces for nodes through learned gating weights and aligned pairwise distance computation (Guo et al., 2024).

A second misunderstanding is that Riemannian MoE routing must be hard selection. The two formulations show otherwise. GraphMoRE uses a dense softmax $\mathcal M^d_{\kappa_i}$4 over all $\mathcal M^d_{\kappa_i}$5 experts and down-weights each expert continuously through manifold scalar multiplication. GAD-MoRE, by contrast, computes a dense temperature-controlled distribution $\mathcal M^d_{\kappa_i}$6 and then sparsifies it to top-$\mathcal M^d_{\kappa_i}$7 experts through $\mathcal M^d_{\kappa_i}$8. Both are MoE layers, but they occupy different points on the dense-to-sparse routing spectrum (Zhao et al., 6 Feb 2026).

A third misunderstanding is that geometry enters only at the expert level. In both formulations, geometry also governs input preparation, output fusion, and loss construction. GraphMoRE makes pairwise distance alignment part of the layer semantics; GAD-MoRE makes anomaly-aware multi-curvature feature alignment and memory-based routing part of the same semantics. This broadens the notion of a Riemannian Mixture-of-Experts layer from a local expert block to a curvature-conditioned computational pathway.

Within the cited literature, the layer is positioned as an architectural response to tasks requiring uniform handling of diverse graph data or generalization to unseen graph domains. GraphMoRE frames it as a novel architectural perspective for graph foundation models, while GAD-MoRE frames it as a mechanism for zero-shot cross-domain graph anomaly detection (Guo et al., 2024, Zhao et al., 6 Feb 2026). In both cases, the defining principle is the same: instead of committing the entire graph or node population to one global geometry, the model learns which curvature-specific expert or combination of experts should dominate for each input.