---
title: 'GMoPE: Expert Block Framework for Graph Generalization'
url: https://www.emergentmind.com/topics/expert-block-framework
type: topic
---

# GMoPE: Expert Block Framework for Graph Generalization

GMoPE, short for **Graph Mixture of Prompt-Experts**, is a framework for graph foundation models that combines Mixture-of-Experts (MoE) with prompt-based learning in order to improve cross-domain and cross-task generalization for graph neural networks. It is designed to address three limitations identified for existing graph transfer methods: negative transfer, scalability issues, and high adaptation costs. The framework assigns each expert its own prompt vector, uses a structure-aware routing mechanism to determine which experts should contribute to a batch or task, introduces a soft orthogonality constraint to prevent prompt collapse, and adopts a prompt-only fine-tuning strategy during transfer. Reported experiments show that it consistently outperforms state-of-the-art baselines and can reach performance comparable to full parameter fine-tuning while requiring only a small fraction of the adaptation overhead [2511.03251].

## 1. Problem Formulation and Design Rationale

Graph Neural Networks have demonstrated strong performance on task-specific benchmarks, but their ability to generalize across diverse domains and tasks remains limited. GMoPE is presented as a response to this limitation, especially in settings where a pretrained graph model must be adapted efficiently to multiple downstream tasks without incurring the full cost of tuning all backbone parameters [2511.03251].

The framework is built around a specific interpretation of expert specialization in graph learning. Rather than instantiating experts as entirely separate architectures, GMoPE keeps a shared expert architecture family and differentiates experts through expert-specific prompt vectors and routing. This design couples two ideas: prompt-based adaptation, which lowers transfer cost, and MoE-style specialization, which aims to reduce negative transfer by allowing different experts to focus on different subdomains. A plausible implication is that the framework treats domain diversity as a routing-and-conditioning problem rather than only as a parameter-sharing problem.

The paper characterizes the resulting system as a principled and scalable framework for generalizable and efficient graph foundation models. Its central claim is not merely that multiple experts improve accuracy, but that expert-specific prompts, structure-aware routing, orthogonality regularization, and prompt-only tuning operate as a coordinated mechanism.

## 2. Prompt-Conditioned Expert Architecture

For each expert $E_m$ among a total of $M$ experts, GMoPE maintains a learnable prompt vector
$$
\mathbf{p}_m \in \mathbb{R}^{d_p}.
$$
Given a graph $\mathcal{G}^{(i)} = (V^{(i)}, E^{(i)}, X^{(i)})$, the raw node features are first aligned by
$$
\tilde X^{(i)} = \mathrm{Proj}\bigl(X^{(i)}\bigr) \in \mathbb{R}^{|V^{(i)}| \times d_0},
$$
where $\mathrm{Proj}$ is implemented via SVD [2511.03251].

The expert prompt is then broadcast-concatenated to every node:
$$
\hat X_m^{(i)} = \bigl[\tilde X^{(i)} \;\|\; \mathbf{p}_m \mathbf{1}^\top \bigr] \in \mathbb{R}^{|V^{(i)}| \times (d_0 + d_p)}.
$$
This augmented feature matrix is fed into a standard GNN backbone such as GCN, GIN, or GAT. All $M$ experts share the same architecture, but they differ in their prompt vectors $\{\mathbf{p}_m\}$ and, during pretraining, in their own parameters $\theta_m$.

This expert design makes the prompt vector the primary conditioning interface. The expert is therefore not defined solely by a separate routing gate or by a full parameter fork, but by a prompt-conditioned view of the same graph. The empirical ablations reported for prompt removal indicate that these prompt vectors are not incidental: removing expert prompts causes substantial performance degradation across link prediction, node classification, and graph classification.

## 3. Structure-Aware Routing and Expert Aggregation

GMoPE uses a structure-aware router that computes routing weights from each expert’s performance on the current mini-batch. The raw score for expert $m$ is defined as
$$
\mathrm{Rawscore}_m
=
\frac{1}{B} \sum_{s \in \mathcal B} \mathcal L\bigl(E_m(\hat X_m^{(i)}; s)\bigr),
$$
where $\mathcal B$ is a mini-batch of $B$ samples and $\mathcal L$ is the pretraining loss, such as contrastive or reconstruction loss [2511.03251].

Two router variants are used.

The **soft router**, intended for pretraining in data-rich settings, first selects the top-$K$ experts $\mathcal K$ by highest $\mathrm{Rawscore}_m$, and then assigns for $i \in \mathcal K$
$$
g_i(x)
=
\frac{\exp(\mathrm{Rawscore}_i / \tau)}
{\sum_{j \in \mathcal K} \exp(\mathrm{Rawscore}_j / \tau)},
$$
with $g_i(x)=0$ for $i \notin \mathcal K$.

The **hard router**, used in low-resource transfer, assigns uniform mass over the selected experts:
$$
g_i(x) =
\begin{cases}
1/K, & i \in \mathcal K, \\
0, & \text{otherwise.}
\end{cases}
$$

At inference, each expert $E_m$ produces a prediction $\hat{\mathbf y}_m$ and a confidence
$$
\alpha_m = 1 - \mathcal H(\hat{\mathbf y}_m),
\qquad
\mathcal H(p) = -\frac{1}{\log C} \sum_{c=1}^C p_c \log p_c.
$$
The normalized aggregation weight is
$$
\omega_m = \alpha_m / \sum_j \alpha_j,
$$
and the final embedding is
$$
\mathbf h_{\rm final} = \sum_{m=1}^M \omega_m \mathbf h_m.
$$

This routing strategy couples training-time specialization with inference-time uncertainty weighting. The paper’s interpretation is that expert-specific prompts and structure-aware routing jointly drive specialization. This suggests that routing is intended not only to save computation but to maintain a meaningful partition of graph subdomains across experts.

## 4. Orthogonality, Specialization, and Training Objectives

To prevent prompt collapse and encourage experts to specialize in different subspaces, GMoPE adds a soft orthogonality constraint across prompt vectors:
$$
\mathcal L_{\rm ortho}
=
\frac{1}{M(M-1)}
\sum_{\substack{m,n=1 \\ m \neq n}}^M
\exp\!\left(
\frac{\mathbf p_m^\top \mathbf p_n}
{\|\mathbf p_m\|_2 \, \|\mathbf p_n\|_2}
\right).
$$
The stated purpose of this term is twofold: to encourage expert specialization and to facilitate more balanced expert utilization [2511.03251].

During pretraining, GMoPE jointly optimizes expert parameters and prompt vectors under a pretraining loss combined with the orthogonality regularizer. During transfer, the expert backbone parameters $\theta_m$ are frozen, and only the prompt vectors $\{\mathbf p_m\}$ and downstream task head $\phi$ are optimized, again with the orthogonality term retained.

The ablation results make the role of this regularization explicit. When the orthogonality coefficient $\lambda$ is too small, specialization is poor; when it is too large, prompts become unstable. The reported optimal range is approximately $\lambda \approx 0.1\mbox{--}3$. This is one of the central technical claims of the framework, because it ties expert diversity to a directly controlled objective term rather than to implicit routing behavior alone.

A common misconception in MoE-style systems is that expert diversity will emerge automatically from sparse routing. The GMoPE results argue against that assumption. In this framework, prompt diversity is explicitly regularized, and balanced expert use is treated as an optimization objective rather than an emergent by-product.

## 5. Prompt-Only Fine-Tuning and Computational Efficiency

A defining feature of GMoPE is its prompt-only fine-tuning strategy. In transfer, all expert GNN parameters $\{\theta_m\}$ are frozen, and only the prompt vectors $\{\mathbf p_m\}$ together with the downstream prediction head $\phi$ are updated [2511.03251].

The reported parameter savings are substantial. For node and edge tasks on citation networks, full fine-tuning updates approximately 20,928 parameters, whereas GMoPE tunes only 192 parameters, or approximately $0.9\%$. For graph classification, full fine-tuning updates approximately 3,584 parameters, whereas GMoPE tunes only 12 parameters, or less than $0.4\%$.

The complexity analysis distinguishes forward and backward behavior. The forward pass still involves all $M$ experts, but during backpropagation only the top-$K$ experts receive gradients. As a result, fine-tuning complexity becomes
$$
O\bigl(K(L|V|d^2 + L|E|d)\bigr)
$$
per batch instead of
$$
O\bigl(M \cdot \bigr).
$$
The paper therefore frames the transfer procedure as a reduction in spatiotemporal complexity rather than as a reduction in model expressivity. The empirical claim is stronger than parameter efficiency alone: prompt-only tuning attains, and in some settings exceeds, full-fine-tuning performance with less than $1\%$ of the parameters and with lower computational cost.

## 6. Experimental Results, Ablations, and Task-Specific Behavior

GMoPE is evaluated under multiple pretraining strategies and downstream tasks, including link prediction, node classification, and graph classification [2511.03251]. Under DGI pretraining for link prediction, the citation-network average AUC is reported as 88.04 for full fine-tuning, 86.63 for GPF, 84.70 for GraphPrompt, 85.46 for AnyGraph, and **88.22 for GMoPE**, corresponding to gains of +1.6 points over GPF and +3.7 over GraphPrompt. Under GAE pretraining, the average values are 89.35 for full fine-tuning, 89.58 for GPF, 88.59 for AnyGraph, and **91.80 for GMoPE**, giving a +2.22-point improvement over GPF.

For node and graph classification under DGI or GraphCL pretraining, the reported node average is 62.67 for full fine-tuning, 63.02 for GPF, 53.31 for GraphPrompt, and **64.57 for GMoPE**, a +1.55 gain over GPF. The graph average is 65.92 for full fine-tuning, 64.95 for GPF, and **66.72 for GMoPE**, a +1.77 improvement. Under EdgePred pretraining, node classification averages are 69.07 for full fine-tuning, 63.89 for GPF, 58.41 for GraphPrompt, and **69.25 for GMoPE**.

The ablations isolate the importance of the expert-block components. Removing expert prompts causes AUC on DGI link prediction to drop from 88.22 to 79.96. For classification, node ACC drops from 64.57 to 58.41, and graph ACC drops from 66.72 to 58.19. These are among the clearest indications that prompt conditioning is a structural necessity rather than a superficial add-on.

The framework also exhibits task-dependent routing preferences. For the number of experts $M$ and the number of activated experts $K$, the best trade-off is reported as $M \approx N_{\rm datasets}$, with $K=M$ for node and edge tasks, but $K=1$ for graph tasks. This suggests that graph-level prediction benefits from sharper expert selection, whereas node and edge tasks benefit from broader expert participation. Because this interpretation follows from the reported trade-off rather than from a formal theorem, it is best read as an empirical regularity of the studied benchmarks.

Taken together, the experiments support three claims. First, expert-specific prompts combined with structure-aware routing drive specialization. Second, soft orthogonality promotes balanced expert usage. Third, prompt-only tuning yields transfer performance comparable to or better than full fine-tuning while drastically reducing adaptation cost.

Source: https://www.emergentmind.com/topics/expert-block-framework