---
title: Mixture of Subgraph Experts (MoSE)
url: https://www.emergentmind.com/topics/mixture-of-subgraph-experts-mose
type: topic
---

# Mixture of Subgraph Experts (MoSE)

Searching arXiv for the MoSE graph paper and closely related mixture-of-experts graph work.
Mixture of Subgraph Experts (MoSE) is a graph representation-learning framework that combines anonymous-walk-based subgraph extraction, sparse mixture-of-experts routing, and random walk kernel (RWK) matching against learnable hidden graphs in order to improve structural expressiveness across both node classification and graph classification tasks. It was introduced to address the limitation that standard message-passing GNNs rely on local, pairwise neighborhood aggregation and are therefore upper-bounded by the 1-dimensional Weisfeiler-Lehman test (1-WL), which restricts their ability to capture high-order subgraph structures such as triangles, cycles, and more complex local configurations [2509.09337].

## 1. Problem setting and conceptual basis

MoSE is motivated by the observation that stacking more message-passing layers does not directly encode the relational arrangement among neighbors. The framework therefore replaces plain neighbor aggregation with adaptive subgraph processing. Its starting point is the claim that modern MPNNs are upper-bounded by 1-WL and inherit its inability to reliably distinguish many higher-order structural patterns. This motivates a shift from pairwise message passing toward subgraph-based structural probes.

The immediate precursor identified for this shift is graph-kernel-based GNN design, especially models based on the random walk kernel. In those models, an input graph is compared against a set of learnable hidden graphs that act as structural probes. MoSE adopts that basic intuition but argues that prior RWK-based neural models have two limitations: they are largely designed for graph-level tasks, and they use fixed kernel configurations and fixed hidden graph designs. The first limitation makes them poorly suited to node-level tasks, especially in large graphs; the second reduces flexibility because different nodes or graphs may require different structural probes.

MoSE addresses these issues through a pipeline in which each node first receives an informative local subgraph, then that subgraph is routed to specialized experts, and each expert compares the subgraph against its own learnable hidden graphs. This suggests a division of labor absent from monolithic RWK-based models: subgraph extraction determines what local structure is exposed, routing determines which expert family is relevant, and the experts perform structurally specialized matching.

## 2. Anonymous-walk subgraph extraction and structural routing

MoSE does not use all local context indiscriminately. Instead, it extracts rooted subgraphs using anonymous walks. An anonymous walk preserves the structural revisit pattern of a random walk while discarding node identities:

$$
\textbf{Definition [Anonymous Walk]}
$$

Given a random walk $w=(v_0, v_1, \ldots, v_L)$ on a graph, the corresponding anonymous walk is defined as a sequence of integers $a=(\gamma_0, \gamma_1, \ldots, \gamma_L)$, where $\gamma_i=\min\mathrm{pos}(w, v_i)$, and $\mathrm{pos}(w, v_i)$ returns the position at which the node $v_i$ first appears in the walk $w$. We denote the mapping of a random walk $w$ to the anonymous walk $a$ by $w\mapsto a$.

The paper gives a reconstruction result to justify the use of anonymous walks: a sufficiently rich distribution over anonymous walks starting at a node is sufficient to reconstruct the rooted ego graph centered at that node. This is used to argue that anonymous walks can carry enough local structural information to recover the rooted local neighborhood and are not merely a heuristic compression.

The extraction procedure is specified as

$$
\mathcal{A}_v=\{w\mapsto a: w\in \mathcal{W}_v\}. \tag{6}
$$

$$
\mathcal{P} = \mathrm{TopK}\left(\bigcup_{v\in\mathcal{V}}\mathcal{A}_v, k_{walk} \right). \tag{7}
$$

$$
G_v = G[\{v\} \cup \mathcal{N}_{v}^{aw}]. \tag{8}
$$

Here the model samples random walks from each node, converts them into anonymous walks, keeps the most frequent patterns globally, collects nodes appearing in walks whose anonymous patterns belong to that retained set, and induces a final rooted subgraph $G_v$. The result is a subgraph extraction policy that is pattern-driven rather than radius-driven.

Routing is then performed by a subgraph-aware gating network. The paper defines a topology-aware pooled representation

$$
\bm{\eta} (v) = \sigma \left(\bm{x}_v + \sum\nolimits_{u \in \mathcal{V}_v}\alpha_{uv}\bm{x}_u \right), \tag{9}
$$

with non-parametric attention weights

$$
\alpha_{uv}=\exp(\bm{x}_u^\top\bm{x}_v)/\sum_{u\in\mathcal{V}_v}\exp(\bm{x}_u^\top\bm{x}_v).
$$

The routing logits and sparse gate are

$$
\bm{\psi}(v) =\bm{\eta}(v)\bm{W}_g+\epsilon\cdot \mathrm{Softplus}(\bm{\eta}(v)\bm{W}_n), \tag{10}
$$

$$
\bm{\zeta}(v) = \mathrm{Softmax} \left(\mathrm{TopK}\left(\bm{\psi}(v), k_{ept}\right)\right). \tag{11}
$$

Only the top-$k_{ept}$ experts are activated. In the paper’s terminology, routing is based on the “structural semantics” of the extracted subgraph. This refers to the topology and feature configuration that indicate what kind of local pattern the node belongs to.

## 3. Expert construction and random-walk-kernel encoding

Each MoSE expert is a bank of learnable hidden graphs. The experts are indexed by size:

$$
E=\{E_s: s=1, 2, \ldots, K\}.
$$

Each expert $E_s$ contains $N$ hidden graphs of size $s$,

$$
E_s=\{H_1^s, H_2^s, \ldots, H_N^s\},
$$

and each hidden graph has a learnable adjacency $ReLU(\bm W_i)\in\mathbb R^{s\times s}$ and a learnable feature matrix $\bm Z_i\in\mathbb R^{s\times f}$.

The expert computation is built on the $P$-step random walk kernel. The paper writes

$$
\mathcal{K}(G, G')= \sum_{p=0}^{P} \mathcal{K}^{(p)}(G, G') =\sum_{p=0}^{P}  \sum_{i=1}^{\vert \mathcal{V}_\times \vert} \sum_{j=1}^{\vert \mathcal{V}_\times \vert} \lambda_p  [\bm{A}_\times^p]_{ij}, \tag{1}
$$

with differentiable form

$$
\mathcal{K}^{(p)}(G, G')= \sum_{i=1}^{\vert \mathcal{V}_\times \vert} \sum_{j=1}^{\vert \mathcal{V}_\times \vert} \bm{s}_i \bm{s}_j [\bm{A}_\times^p]_{ij} = \bm{s}^\top \bm{A}_\times^p \bm{s}, \tag{2}
$$

where $\bm S=\bm X' \bm X^\top$ and $\bm s=\mathrm{vec}(\bm S)$.

MoSE applies this machinery not to the whole graph but to each extracted rooted subgraph. For node $v$ and expert $s$,

$$
\bm{h}_s(v) = \mathrm{MLP}\left(\concat_{i=1}^N \mathcal{K}^{(p)}\left(G_v, H_i^s\right)\right). \tag{12}
$$

The expanded RWK computation given in the paper is

$$
\begin{aligned}
\mathcal{K}^{(p)}(G_v, H_i) = \bm{s}^\top (\bm{A}_{G_v}^p \otimes ReLU(\bm{W}_i)^p)\bm{s}
= \bm{s}^\top \mathrm{vec}(ReLU(\bm{W}_i)^p \mathrm{vec}^{-1}(\bm{s})\bm{A}_{G_v}^p) \\
= \bm{1}^\top [\bm{Z}_i \bm{X}_{G_v}^\top \odot ReLU(\bm{W}_i)^p \bm{Z}_i \bm{X}_{G_v}^\top \bm{A}_{G_v}^p] \bm{1}.
\end{aligned} \tag{13}
$$

The node representation is then formed by combining the outputs of the selected experts:

$$
\bm{h}(v) = Comb\left(\{\bm{h}_s(v) : s\in\mathcal{M}\}, \bm{\zeta}\left(v\right)\right), \tag{14}
$$

where the paper states that $Comb(\cdot)$ can be either a weighted sum or a concatenation followed by an MLP. For graph-level tasks, the node embeddings are pooled by a permutation-invariant readout:

$$
\bm{h}(G)=Readout(\{\bm{h}(v) \mid v \in \mathcal{V}\}). \tag{15}
$$

This architecture makes expert specialization explicit. Different experts can learn different families of hidden graphs, and sparse top-$k$ routing means each input subgraph only pays computation for a few selected experts.

## 4. Training objective, balancing, and expressivity

MoSE is trained with a task loss plus an importance-balancing loss that discourages router collapse. The importance term is written as

$$
\mathrm{Importance}(\mathcal{V}) = \sum_{v\in\mathcal{V}} \sum_{m\in\mathcal{M}} E_m^v,
$$

$$
\mathcal{L}_{importance}(\mathcal{V}) = \mathrm{CV}(\mathrm{Importance}(\mathcal{V}))^2, \tag{16}
$$

and the final objective is

$$
\mathcal{L} = \mathcal{L}_{task} + \beta\cdot \mathcal{L}_{importance}. \tag{17}
$$

This makes the routing mechanism a sparse MoE in the usual sense: expert selection is conditional, top-$k$, and regularized for balanced utilization.

The paper analyzes expressivity through the Subgraph Weisfeiler-Lehman (SWL) test. It contrasts 1-WL refinement,

$$
c_v^{(t+1)} = \operatorname{hash}\left(c_v^{(t)}, \{\!\!\{c_u^{(t)} \mid u \in \mathcal{N}(v)\}\!\!\}\right),
$$

with SWL refinement on rooted subgraphs,

$$
c^{(t+1)}_v = \operatorname{hash}\bigl(G^{(t)}_v\bigr).
$$

The main proposition states that if the hidden graph family is sufficiently large, the RWK is positive-definite and distinguishes all non-isomorphic subgraphs of bounded size, and the MLP $\rho$ is injective, then the MoSE update

$$
c_v^{(t+1)} = \rho\left(\Phi(G_v)\right) = \rho\left( \left[\mathcal{K}^{(p)}(G_v, H_i)\right]_{i=1}^N \right)
$$

is injective over the set of rooted subgraphs $G_v$. The corollary then states that if SWL distinguishes two graphs under a fixed subgraph extraction policy, MoSE also distinguishes them under the same policy, provided the proposition’s conditions hold [2509.09337].

The appendix further phrases this as “more powerful than SWL” under the paper’s refinement-based definition. A plausible implication is that the paper views MoSE not simply as a practical mixture architecture but as a learned, kernelized refinement procedure whose stable coloring can be at least as fine as SWL’s when the hidden-graph family and RWK are sufficiently expressive.

## 5. Interpretability and empirical profile

Interpretability is one of MoSE’s defining claims. The hidden graphs act as structural probes, while routing exposes which expert family is selected for each extracted subgraph. The visualizations described in the paper show that learned experts are diverse and compact, and that they are less redundant than fixed-size hidden graphs in earlier RWK-based models. On Chameleon and IMDB-MULTI, the selected experts differ across subgraphs; on Wisconsin and MUTAG, visualizing all experts shows higher redundancy on simpler datasets like MUTAG and less redundancy on more structurally nuanced datasets like Wisconsin. The paper explicitly connects the learned expert families to synthetic structural patterns such as caveman, cycle, grid, ladder, and star [2509.09337].

The empirical scope spans 19 datasets across graph classification and node classification.

| Task | Datasets |
|---|---|
| Graph classification | MUTAG, DD, NCI1, PROTEINS, ENZYMES, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, GraphFive, GraphCycle |
| Node classification | Cora, Citeseer, Pubmed, Chameleon, Squirrel, Actor, Cornell, Texas, Wisconsin |

The baselines cover classical graph kernels, MPNN-based GNNs, and graph-kernel-based neural models, including GL, SP, WL, GCN, GAT, GIN, GraphSAGE, RWNN, KerGNN, GNN-AK, RWK$^+$CN, GIP, and GKNN.

On graph classification, MoSE achieves the best or near-best results on almost all datasets. The paper reports, for example, **92.7 ± 2.5** on MUTAG, **83.7 ± 1.6** on DD, **78.6 ± 4.4** on PROTEINS, **83.0 ± 3.2** on IMDB-BINARY, **52.6 ± 1.1** on IMDB-MULTI, and **92.1 ± 1.4** on REDDIT-BINARY. On the synthetic datasets, it reports **62.3 accuracy / 57.4 F1** on GraphFive and **82.3 accuracy / 78.7 F1** on GraphCycle.

On node classification, the paper emphasizes especially strong performance on heterophilous graphs, reporting **58.8 ± 0.7** on Squirrel, **39.6 ± 1.1** on Actor, **73.0 ± 3.4** on Cornell, **83.7 ± 3.8** on Texas, and **87.5 ± 4.3** on Wisconsin, while also giving **64.5 ± 1.1** on Chameleon and competitive results on Cora, Citeseer, and Pubmed.

The ablation study isolates three components: anonymous-walk subgraph extraction, the MoE mechanism, and subgraph-aware gating. The paper states that the MoE component has the largest impact on performance, that subgraph-aware gating also contributes significantly, and that anonymous-walk-based extraction improves the ability to isolate discriminative local patterns. It also reports about **10.84% average performance improvement** with around **30% runtime reduction** relative to other RWK-based models. Sensitivity analysis indicates that increasing the number of hidden graphs per expert helps initially but can later cause diminishing returns or overfitting, and that performance often improves up to **3 or 4 steps** in the random walk kernel.

## 6. Position within the MoE literature and acronym ambiguities

In the graph-learning literature, MoSE occupies a specific niche: it is a subgraph-based, RWK-driven mixture-of-experts model in which the routed object is an extracted rooted subgraph and the experts are banks of hidden graphs specialized by structural pattern. This distinguishes it from other graph MoE formulations that route nodes to filter experts or receptive-field experts rather than routing subgraphs to RWK-based subgraph experts.

A close neighboring formulation is Node-MoE, which performs node-wise filtering by routing nodes to experts with different spectral responses such as low-pass, constant, and high-pass filters; its motivation is mixed homophilic and heterophilic local structure rather than RWK-based subgraph matching [2406.03464]. Another related model is Graph Mixture of Experts (GMoE), which replaces each GNN layer with sparse node-level routing over hop-1 and hop-2 aggregation experts and uses importance and load balancing losses, but it does not explicitly construct or route extracted subgraphs [2304.02806]. These comparisons indicate that MoSE is more structurally explicit than hop-based or filter-based graph MoEs because its experts operate on induced subgraphs extracted through anonymous walks.

The acronym “MoSE” is also overloaded outside this graph-specific setting. In segmentation uncertainty, “MoSE” refers to “Mixture of Stochastic Experts,” a method for modeling multimodal aleatoric uncertainty in semantic segmentation [2212.07328]. In autonomous driving, “MoSE” has been used to denote “Mixture of Sparse Experts,” where the experts are organized by driving function and use different sparse attention mechanisms [2511.11740]. Those uses are unrelated to Mixture of Subgraph Experts except for the shared MoE principle.

Within graph representation learning itself, MoSE’s distinctive contribution is the combination of three elements: anonymous-walk subgraph extraction, sparse subgraph-aware routing, and RWK comparison against learnable hidden graphs. This suggests a synthesis between kernel methods and conditional computation: instead of using one fixed bank of structural probes for every local region, the model dynamically chooses which expert family should evaluate the extracted subgraph, thereby improving flexibility, interpretability, and expressiveness.

Source: https://www.emergentmind.com/topics/mixture-of-subgraph-experts-mose