Refining GNN Architectures

• GNN Architecture Refinement Methods are a suite of techniques that enhance model expressivity, efficiency, and robustness using high-order derivatives, individualized node selection, and automated search.
• They employ both discrete and continuous NAS frameworks along with class-aware augmentations and robust structure learning to tailor architectures for specific tasks.
• Advanced plug-and-play modules and adaptive residual schemes further mitigate over-smoothing and degradation in deep GNNs, ensuring improved overall performance.

Graph Neural Network (GNN) Architecture Refinement Methods encompass a diverse set of strategies aimed at improving model expressivity, efficiency, robustness, and generalization. These methodologies include expressivity-boosting schemes via high-order derivatives and individualized refinement, architectural search and optimization (both discrete and continuous), class-aware module augmentations, advanced regularization, model simplification, robust structure learning, and foundational residual mechanisms for deep architectures. The following sections present a technical synthesis of principal refinement technique categories and their operational characteristics.

1. Expressivity Enhancement via Derivatives and Individualization

Recent work has demonstrated that the representational limitations of message-passing neural networks (MPNNs) can be overcome by systematically extracting and encoding high-order derivative information or employing individualization with refinement procedures.

High-Order Derivative GNNs (HOD-GNN):

HOD-GNN (Eitan et al., 2 Oct 2025) leverages mixed partials of the base MPNN output with respect to node features, i.e., high-order derivatives: $$D^{(T)}_k[v,(u_1, ..., u_k),i,(α_1, ..., α_k)] = \left. \frac{∂^{|\boldsymbol{α}|} h^{(T)}_{v,i}}{∂X_{u_1}^{α_1} … ∂X_{u_k}^{α_k}} \right|_{X}$$ These tensors act as structure-aware descriptors, capturing how feature perturbations at multiple nodes propagate throughout the graph. HOD-GNNs process the derivative tensors using downstream invariant GNNs to yield node and graph embeddings with expressivity matching or exceeding the Weisfeiler-Lehman (WL) hierarchy, theoretically matching the power of subgraph GNNs without their combinatorial overhead. For instance, even first-order HOD-GNNs with ReLU strictly dominate random walk positional encoding MPNNs (RWSE-MPNNs) and can distinguish vertex-transitive counterexamples unsolvable by 1-WL (Eitan et al., 2 Oct 2025).

Individualization and Refinement (IR-GNN):

The IR paradigm (Dupty et al., 2022) replaces the k-tuple enumeration of higher-order GNNs with strategically individualized nodes followed by refinement—propagation of the individualized label via message passing. Repeated rounds of adaptive node selection and aggregation of refinement results produce embeddings with strictly higher expressiveness than 1-WL GNNs at far reduced cost compared to k-WL GNNs. Selection policies are learned, and aggregation is permutation-invariant, yielding plug-in refinement modules for any backbone.

2. Automated Architecture and Block-wise Search

Automated Neural Architecture Search (NAS) frameworks for GNNs support systematic exploration and refinement of architectures to discover data- and task-specific models.

Discrete/RL-based NAS (SNAG, DeepGNAS):

SNAG (Zhao et al., 2020) defines a compact, yet expressive, search space comprising various message-passing aggregators, skip-connection patterns, and global layer aggregators. RL controllers (LSTM-based) sample candidate architectures, which are then trained and evaluated, with policy-gradient updates guiding the sampling distribution toward high-performing regions. The DeepGNAS pipeline (Feng et al., 2021) introduces two-stage block-wise and architecture-wise search, using deep Q-learning over block-encoded DAGs and multi-block stacking schemes. Residual and identity mappings are incorporated at both block and layer granularity, supporting very deep architectures (20–30 layers) and mitigating oversmoothing. These search spaces subsume most common hand-crafted designs and have yielded state-of-the-art performance in both node and graph classification.

Continuous/Latent-space NAS (GraphVAE-based):

Graph VAE-based NAS (Li et al., 2020) embeds discrete architectures as graphs into a continuous latent space via a GNN encoder-decoder framework coupled with performance and cost predictors. Refinement proceeds by gradient ascent in the latent space of encodings, guided by joint optimization of predicted accuracy and computational complexity, before decoding back to new candidate architectures. This continuous refinement not only enables efficient gradient-based search but tends to yield architectures with superior accuracy-FLOPs trade-offs compared to two-stage or purely discrete approaches.

3. Plug-and-Play and Augmentation Modules

Class-Aware Representation rEfinement (CARE):

CARE (Xu et al., 2022) augments any GNN by introducing a class prototype mechanism: learned class-wise representative embeddings (constructed from subgraph bags using permutation-invariant set encoders) are injected into the graph representations post-GNN via concatenation and MLP transformation. Training utilizes a composite loss encouraging intra-class cohesion and inter-class separation in embedding space. Theoretically, CARE reduces the upper VC-dimension bound of the overall model relative to its backbone, thus tightening generalization guarantees. This module yields consistent accuracy and ROC-AUC improvements across a variety of backbones and benchmarks with only constant-factor computational overhead.

Network In Graph Neural Network (NGNN):

NGNN (Song et al., 2021) enhances GNN expressiveness and robustness by deepening each message-passing layer with additional (1–2) non-linear feedforward sublayers post-aggregation, but before the standard non-linearity. This model-agnostic approach increases local functional capacity without expanding the receptive field or inducing oversmoothing.

4. Robustness via Structure Refinement and Adversarial Defenses

Unsupervised Structure Refinement (STABLE):

The STABLE pipeline (Li et al., 2022) addresses the vulnerability of GNNs to poisoned graph structures. It constructs robust node representations using a contrastive, unsupervised pretraining objective over multiple randomly corrupted adjacency views, yielding node embeddings insensitive to small structural perturbations. The induced similarity matrix supports pruning and addition of edges to refine the graph before downstream GNN classification, and an advanced degree-aware GCN update amplifies the effect of high-confidence/high-degree links. This unsupervised approach raises robustness by 5–20% in adversarial scenarios, incurring no extra asymptotic computational cost.

5. Model Simplification and Linear GNNs

LightKG:

LightKG (Li et al., 12 Jun 2025) exemplifies radical model simplification: for knowledge graph recommendation, it reduces per-relation parameters to two scalars (forward, reverse) and employs a purely linear aggregation—eschewing all dense matrix parameters and non-linearities—while incorporating an efficient, subgraph-free self-supervised contrastive loss. This approach achieves superior accuracy and up to 84% reduction in training time compared to standard state-of-the-art GNN-based recommenders reliant on complex message functions and subgraph augmentations, especially in sparse interaction regimes.

6. Enhanced Expressivity via Reconstruction and Higher-Order Schemes

Reconstruction GNNs and Color Refinement:

Reconstruction GNNs (Arvind et al., 2024) lift the color refinement paradigm by running 1-WL-equivalent GNNs on all vertex-deleted subgraphs and aggregating the results. This approach maintains a direct correspondence to multiset color refinement (D-CR) and strictly exceeds standard 1-WL expressiveness—it can, for example, decide graph connectedness, a function not expressible by any 1-WL-bounded architecture. The computational trade-off is quadratic in the number of vertices, but it achieves expressivity that is otherwise unattainable without explicitly invoking 2-WL machinery.

7. Residual Techniques and Deep GNN Stabilization

Recent analyses have disentangled GNN depth into propagation vs. transformation components. The Adaptive Initial Residual (AIR) mechanism (Zhang et al., 2022) inserts node-adaptive skip connections after each propagation or transformation block. For a propagation block, node-wise gating mixes the block input with the propagated features; for transformation blocks, an explicit residual maintains a direct path for the original features. This dual-residual scheme both combats over-smoothing (large propagation depth, $D_p$) and model degradation (large transformation depth, $D_t$), enabling successful training of GNNs at depths previously unattainable in practice, with only minor time overhead.

---

References Table: Principal Refinement Methods

Refinement Class	Key Paper (arXiv ID)	Salient Feature or Mechanism
High-Order Derivatives	(Eitan et al., 2 Oct 2025)	Inject high-order input-feature derivatives as node/graph descriptors, boosting expressivity to WL-hierarchy
Individualization & Refinement	(Dupty et al., 2022)	Adaptive node selection and refinement post-message-passing; strict super-1-WL expressivity
NAS – Discrete/RL	(Zhao et al., 2020, Feng et al., 2021)	RL over compact space of aggregators, skip/JK patterns, and block-wise stacking
NAS – Continuous/VAE	(Li et al., 2020)	GNN-VAE latent space, gradient-based refinement for accuracy and cost
Plug-in Augmentation	(Xu et al., 2022, Song et al., 2021)	Class-aware prototype embedding (CARE); deep in-layer MLPs (NGNN)
Structure Refinement	(Li et al., 2022)	Unsupervised contrastive pre-training for robust edge weighting and advanced GCN
Model Simplification	(Li et al., 12 Jun 2025)	Scalar-only relation encoding, linear aggregation, and subgraph-free contrastive SSL
Higher-order via Subgraphs	(Arvind et al., 2024)	Deck-based (D-CR) subgraph GNNs, reconstructive color refinement
Deep Residuals/AIR	(Zhang et al., 2022)	Node-adaptive residual gating in propagation/transformation blocks

---

These methodologies form a modular and rapidly evolving toolkit for overcoming the classic limitations of GNNs in expressivity, scalability, and robustness. They have established performance gains across domains including graph-level prediction, node classification, recommendation, and adversarial resistance, often establishing new state-of-the-art results or guaranteed expressivity properties as validated by theory and extensive empirical benchmarking.