MoRE-GNN: Modular Graph Architectures

• MoRE-GNN is a family of graph neural network architectures that combine graph structures with expert specialization to enhance model expressiveness.
• It includes S’MoRE for efficient LLM fine-tuning, a heterogeneous autoencoder for multi-omics integration, and GNNMoE for flexible node classification.
• Each variant optimizes task-specific objectives through tailored gating, routing, and modular design, achieving improved accuracy and parameter efficiency.

MoRE-GNN is not a single canonical architecture. In 2025 usage, the label denotes at least three related but distinct constructions: S’MoRE, a Structural Mixture of Residual Experts for LLM fine-tuning whose residual experts form a graph and propagate by a GNN-like rule; MoRE-GNN, the Multi-omics Relational Edge Graph Neural Network, a heterogeneous graph autoencoder for paired single-cell multi-omics integration; and GNNMoE, a Mixture of Decoupled Message Passing Experts for node classification that is described as a MoRE-GNN in the sense of a Mixture-of-(message-passing)-Experts GNN (Zeng et al., 8 Apr 2025, Wang et al., 8 Oct 2025, Chen et al., 12 Feb 2025). This suggests that “MoRE-GNN” functions less as a unique model name than as a family label for architectures that combine graph structure with expert specialization.

1. Nomenclature and scope

The term has three established referents in the supplied literature. Two are interpretive uses of the name for models whose original titles do not contain “MoRE-GNN,” and one is an explicit paper title.

Usage	Expansion	Primary setting
S’MoRE	Structural Mixture of Residual Experts	PEFT for LLM fine-tuning
MoRE-GNN	Multi-omics Relational Edge Graph Neural Network	Paired single-cell multi-omics integration
GNNMoE	Mixture of Decoupled Message Passing Experts with Entropy Constraint	General node classification

S’MoRE is defined as a parameter-efficient fine-tuning architecture that combines LoRA-style low-rank adapters, MoE-style routing, and a hierarchical, graph-structured residual composition. The supplied description states that S’MoRE is “exactly what you are calling a MoRE-GNN”: a Mixture-of-Residual-Experts whose experts are organized as a graph and updated by a GNN-style message passing scheme. The multi-omics MoRE-GNN is a heterogeneous graph autoencoder over cell nodes and modality-specific edge types. GNNMoE is described as “exactly GNNMoE” in the sense of a MoRE-GNN for general node classification, with message-passing experts and node-wise gating (Zeng et al., 8 Apr 2025, Wang et al., 8 Oct 2025, Chen et al., 12 Feb 2025).

The shared motif is not a shared task, dataset family, or graph semantics. Instead, the commonality is the use of graph structure to organize either experts, relations, or message-passing regimes, together with some form of conditional or relation-aware aggregation.

2. S’MoRE as a graph-structured mixture of residual experts

S’MoRE places MoRE-GNN in the context of LLM adaptation. Each transformer weight to be adapted is approximated by a sum of residuals of multiple orders,

$$\mathbf{W}^i \approx \sum_{\ell=0}^{L-1}\Delta \mathbf{W}^i_\ell,$$

with each residual expert parameterized in low rank form,

$$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$

Rather than using a single low-rank adapter per layer or a flat set of independent low-rank experts, S’MoRE builds a multi-layer tree or graph of residual experts and routes each token through a token-specific sub-tree. The inter-layer update is

$$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$

where $\alpha^{(\ell)}_{i,n}$ is the router score on edge $n\to i$ and $\sigma(\cdot)$ is ReLU or a small MLP. The model defines a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ per token whose nodes are residual experts and whose directed edges are determined by routing, and the authors explicitly relate the update to GIN and the Weisfeiler-Lehman test (Zeng et al., 8 Apr 2025).

This graph interpretation underwrites the paper’s notion of structural flexibility. For a fixed token $\mathbf{x}$ and expert parameters $\Theta$, the relevant quantity is the number of distinct outputs $\text{dist}(\mathbf{x};\Theta)$ realizable by varying routing while keeping expert parameters fixed. The paper contrasts S’MoRE with MoMOR, a flat mixture of multi-order residuals. For MoMOR, structural flexibility is upper-bounded by combinations of selected residuals within each layer’s pool. For S’MoRE, when $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$0 is an MLP and the layer acts like a GIN step, there exist parameters such that

$$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$1

so the combinatorial terms appear as exponents. The supplied description emphasizes the resulting point: as depth grows, S’MoRE’s structural flexibility grows exponentially faster than that of the flat baseline.

The same section also places S’MoRE relative to MoLRE and MoMOR. S’MoRE with $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$2 is equivalent to MoLRE, and S’MoRE reduces to MoMOR if $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$3 is the identity and embedding dimensions avoid information loss. Under the PEFT regime described in the paper, total trainable parameters and expert-side FLOPs satisfy

$$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$4

so the cost is essentially the same as a single LoRA of rank $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$5, with router overhead reported as at most $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$6 of expert cost in the experiments.

Empirically, the reported fine-tuning results use LLaMA-3.2-1B and LLaMA-3-8B on ARC-c/e, CommonsenseQA, OpenBookQA, and Winogrande. On LLaMA-3.2-1B, the best LoRA average accuracy is reported as $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$7 with $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$8B trainable parameters, while a 2-layer S’MoRE with Switch gate reaches up to $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$9 with $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$0B parameters. On LLaMA-3-8B, LoRA is reported at $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$1, HydraLoRA and MixLoRA at $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$2–$$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$3, and S’MoRE at $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$4–$$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$5. The supplied summary states that S’MoRE yields roughly $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$6 absolute accuracy gain while using about $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$7 fewer parameters on average relative to the best baselines, and that 3-layer variants can improve accuracy further, as in the Winogrande comparison from $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$8 to $$\mathbf{x}^{(\ell+1)}_i = \sum_{n \in \mathcal{N}_\ell(i)} \alpha^{(\ell)}_{i,n}\, \sigma\!\Big(\mathbf{UP}[\ell][n]\mathbf{DOWN}[\ell][n]\mathbf{x} + \mathbf{W}[\ell]\mathbf{x}^{(\ell)}_n\Big),$$9 while reducing parameters from $\alpha^{(\ell)}_{i,n}$0B to $\alpha^{(\ell)}_{i,n}$1B (Zeng et al., 8 Apr 2025).

3. MoRE-GNN as a heterogeneous graph autoencoder for multi-omics integration

In single-cell multi-omics, MoRE-GNN denotes the Multi-omics Relational Edge Graph Neural Network. It is designed for paired multi-omics data in which each cell has measurements in all modalities, such as RNA plus protein, RNA plus ATAC, or tri-modal RNA+ATAC+protein. The graph is heterogeneous in edge type rather than node type: nodes are cells, node features are the concatenation of modality-specific feature matrices $\alpha^{(\ell)}_{i,n}$2, and for each modality $\alpha^{(\ell)}_{i,n}$3 there is a distinct cell-cell adjacency $\alpha^{(\ell)}_{i,n}$4 and edge set $\alpha^{(\ell)}_{i,n}$5, giving

$\alpha^{(\ell)}_{i,n}$6

Each $\alpha^{(\ell)}_{i,n}$7 is constructed from cosine similarity followed by a top-$\alpha^{(\ell)}_{i,n}$8 k-NN sparsification, and large datasets are handled by seed-node mini-batches with 1-hop and 2-hop neighborhood sampling. The model is explicitly described as avoiding fixed biological priors such as gene-peak networks and instead constructing relational graphs directly from data (Wang et al., 8 Oct 2025).

The encoder consists of an initial GCN block, a stack of $\alpha^{(\ell)}_{i,n}$9 GATv2 layers, and a final GCN block. All layers operate per modality graph, and outputs are summed across modalities. The intermediate update is

$n\to i$0

The autoencoder uses one decoder per modality. Given latent embeddings $n\to i$1 and $n\to i$2, the decoder takes the Hadamard product $n\to i$3, applies a modality-specific MLP, and predicts the probability that an edge exists in modality $n\to i$4. The total training loss is

$n\to i$5

with $n\to i$6 and $n\to i$7 cluster centers in the reported setup. The reconstruction term is a binary cross-entropy over positive and negative edges, and the clustering term is a nearest-center Euclidean objective.

The implementation details given in the supplied description are specific. The hidden dimension is $n\to i$8; the network uses an initial GCN block, $n\to i$9 GATv2 layers with $\sigma(\cdot)$0 heads, and a final GCN block. Optimization uses Adam with $\sigma(\cdot)$1, $\sigma(\cdot)$2, $\sigma(\cdot)$3, no weight decay, learning rate $\sigma(\cdot)$4 with linear warm-up for $\sigma(\cdot)$5 epochs and cosine decay, batch size $\sigma(\cdot)$6, and neighborhood sizes $\sigma(\cdot)$7. Training runs for up to $\sigma(\cdot)$8 epochs with early stopping if loss fails to improve by at least $\sigma(\cdot)$9 over $\mathcal{G}=(\mathcal{V},\mathcal{E})$0 epochs, typically converging in $\mathcal{G}=(\mathcal{V},\mathcal{E})$1–$\mathcal{G}=(\mathcal{V},\mathcal{E})$2 epochs, with reported runtime of approximately $\mathcal{G}=(\mathcal{V},\mathcal{E})$3–$\mathcal{G}=(\mathcal{V},\mathcal{E})$4 minutes per dataset on a single NVIDIA RTX 4070 with $\mathcal{G}=(\mathcal{V},\mathcal{E})$5GB. The authors also report that PCA preprocessing degraded performance, so the model is trained directly on normalized raw features.

The experimental evaluation covers six datasets: BM-CITE, LUNG-CITE, PBMC-Multiome, PBMC-TEA, PBMC-DOGMA, and Skin-SHARE. The principal baseline is MOJITOO. On BM-CITE, MoRE-GNN reports ARI/NMI of $\mathcal{G}=(\mathcal{V},\mathcal{E})$6 versus $\mathcal{G}=(\mathcal{V},\mathcal{E})$7 for MOJITOO; on LUNG-CITE, $\mathcal{G}=(\mathcal{V},\mathcal{E})$8 versus $\mathcal{G}=(\mathcal{V},\mathcal{E})$9. On PBMC-Multiome, the baseline is stronger, with $\mathbf{x}$0 versus $\mathbf{x}$1 for MoRE-GNN; on Skin-SHARE, the gap is larger, with $\mathbf{x}$2 versus $\mathbf{x}$3. PBMC-TEA is close in ARI but slightly lower in NMI for MoRE-GNN, and PBMC-DOGMA favors MOJITOO.

Cross-modal prediction is evaluated only after training, by fitting lightweight MLPs from the learned cell embeddings to each modality. The reported pattern is modality-dependent: ADT is best predicted, with PCC up to $\mathbf{x}$4 on BM-CITE and $\mathbf{x}$5 on PBMC-TEA; RNA is moderate, with PCC around $\mathbf{x}$6–$\mathbf{x}$7; ATAC or peaks are harder, with PCC around $\mathbf{x}$8–$\mathbf{x}$9. The qualitative analyses further report triangular latent structures in several datasets, matching three main immune lineages, and an amorphous latent space in Skin-SHARE, consistent with continuous differentiation rather than discrete clusters.

4. GNNMoE as a Mixture-of-Message-Passing-Experts GNN

For general node classification, the MoRE-GNN designation refers to GNNMoE, a universal framework intended to operate across both homophilous and heterophilous graphs. Its premise is that different graphs, and even different nodes in the same graph, prefer different orderings of propagation and transformation. The model therefore constructs four message-passing experts from decoupled operators,

$\Theta$0

where PP applies propagation twice, PT applies transformation after propagation, TP propagates after transformation, and TT is purely transformation-based. A per-node soft gate computes expert weights

$\Theta$1

and expert outputs are mixed node-wise, followed by an adaptive residual connection to the initial embedding and LayerNorm. The model then adds an enhanced FFN whose activation itself is hard-gated among SwishGLU, GEGLU, and REGLU via Gumbel-Softmax (Chen et al., 12 Feb 2025).

The training objective combines task loss and gate regularization,

$\Theta$2

The supplied technical description notes that the paper’s typesetting of the entropy term appears sign-inconsistent, and interprets the intended objective as entropy minimization for sharpening the soft gates. The conceptual role of the regularizer is explicit in the description: on homophilous datasets it encourages sharper, near-Top-$\Theta$3/Top-$\Theta$4 routing, while on heterophilous datasets smaller $\Theta$5 permits broader mixtures of experts.

The evaluation uses $\Theta$6 datasets, divided into six homophilous benchmarks and six heterophilous benchmarks. Reported hyperparameters include hidden dimension $\Theta$7, AdamW, maximum $\Theta$8 epochs, early stopping patience $\Theta$9, $\text{dist}(\mathbf{x};\Theta)$0 PT-blocks for most datasets, and a search over $\text{dist}(\mathbf{x};\Theta)$1. The framework is instantiated with GCN-like, SAGE-like, and GAT-like propagation operators. The supplied summary reports that GNNMoE variants are consistently top-$\text{dist}(\mathbf{x};\Theta)$2 and often best on homophilous data, including Coauthor CS at $\text{dist}(\mathbf{x};\Theta)$3 for the GCN-like variant and ogbn-arxiv at $\text{dist}(\mathbf{x};\Theta)$4 for the GAT-like variant. On heterophilous data, the reported best scores include Actor at $\text{dist}(\mathbf{x};\Theta)$5 for the SAGE-like variant, Chameleon-fix at $\text{dist}(\mathbf{x};\Theta)$6 for the GCN-like variant, Squirrel-fix at $\text{dist}(\mathbf{x};\Theta)$7 for the GCN-like variant, Tolokers at approximately $\text{dist}(\mathbf{x};\Theta)$8 ROC-AUC for the GCN-like or GAT-like variants, Roman-empire at $\text{dist}(\mathbf{x};\Theta)$9 for the GAT-like variant, and Penn94 at $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$00 for the SAGE-like variant.

The ablation results summarized in the supplied description are central to the model’s interpretation. Removing the entropy constraint worsens global rank for the GCN-like variant from $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$01 to $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$02. Removing the FFN MoE reduces accuracy substantially on ogbn-arxiv, Actor, Tolokers, and Roman-empire. Comparisons among learnable-parameter gating, mean gating, Top-$$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$03 gating, and entropy-constrained gating show that the entropy-constrained formulation is usually best or near-best, while Top-$$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$04 gating is strong on some homophilous datasets but weaker on heterophilous ones. Efficiency plots on ogbn-arxiv and Penn94 are reported to show $$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$05–$$\Delta\mathbf{W}^{(\ell,i)}=\mathbf{UP}[\ell][i]\mathbf{DOWN}[\ell][i].$$06 less training time than FSGNN and several graph transformers, and the model remains stable at greater depth than vanilla GNNs and H2GCN.

5. Shared design motifs and major differences

Across the three usages, MoRE-GNN always combines graph structure with specialized submodules, but the graph carries different semantics in each case. In S’MoRE, graph nodes are residual experts and edges are router-selected child-to-parent links inside a token-specific expert tree. In the multi-omics MoRE-GNN, graph nodes are cells and edges are modality-specific cell-cell relations derived from cosine similarity and k-NN construction. In GNNMoE, the graph is the original task graph, and the “experts” are alternative message-passing schemes applied to the same nodes (Zeng et al., 8 Apr 2025, Wang et al., 8 Oct 2025, Chen et al., 12 Feb 2025).

Routing and aggregation also differ materially. S’MoRE performs hierarchical top-down routing followed by bottom-up message passing over experts. The multi-omics model does not route among experts; instead it aggregates per-modality graph messages and reconstructs relational edges with modality-specific decoders. GNNMoE uses per-node soft gating over PP/PT/TP/TT experts and hard gating over FFN activation experts. A plausible implication is that the phrase “MoRE-GNN” names three different levels of specialization: expert-graph specialization in LLM PEFT, relation-type specialization in multi-omics graph autoencoding, and message-encoding specialization in node classification.

The objectives are likewise non-interchangeable. S’MoRE is optimized for supervised fine-tuning of LLM adapters under PEFT constraints. The multi-omics model optimizes graph reconstruction plus clustering regularization and evaluates clustering and cross-modal prediction. GNNMoE optimizes node classification cross-entropy with entropy-regularized gating. This difference matters because superficially similar terms—expert, routing, graph, relation—refer to distinct algorithmic roles across the three systems.

6. Limitations, misconceptions, and prospective directions

A common misconception is that “MoRE-GNN” denotes one settled architecture. The supplied literature does not support that reading. It supports three distinct usages whose overlap is conceptual rather than taxonomic. This suggests that any reference to MoRE-GNN requires immediate disambiguation by domain and paper identifier (Zeng et al., 8 Apr 2025, Wang et al., 8 Oct 2025, Chen et al., 12 Feb 2025).

The limitations are domain-specific. For S’MoRE, the reported issues are design complexity, more hyperparameters, hierarchical routing complexity, the need for careful load balancing under sparse gates, and the possibility that scaling depth further may require careful engineering and could introduce training instabilities. For the multi-omics MoRE-GNN, the reported limitations include sensitivity to learning rate, negative sampling, graph architecture, and clustering-loss weight; degradation on ATAC-heavy, noisy, or tri-modal datasets and on continuous trajectories such as Skin-SHARE; clustering bias toward discrete structure; no explicit trajectory modeling; and fixed adjacency matrices after initial construction. For GNNMoE, the supplied practical guidance states that gains may be limited on very small graphs with strong homophily and abundant labels, on extremely sparse or feature-poor graphs, or when latency and memory budgets are extremely tight.

The extension paths are equally differentiated. S’MoRE is explicitly associated with deeper and larger MoRE-GNNs, alternative graph structures beyond trees, possible application beyond language to vision or multimodal models, and adaptive depth. The multi-omics model points toward trajectory-aware metrics and losses, additional regularization for noisy modalities, and broader biological contexts. For GNNMoE, the supplied description emphasizes adaptation through expert design, entropy control, depth scaling, and optional extension of the FFN expert set. Taken together, these directions indicate that “MoRE-GNN” is best understood not as a single method but as a recurring architectural principle: graph-structured or graph-conditioned modularity used to expand expressiveness without collapsing into a single fixed computation pattern.