Explainable Graph Neural Networks

• Explainable Graph Neural Networks (XGNNs) are frameworks that enhance GNN interpretability through subgraph analysis, counterfactual reasoning, and natural language narratives.
• They encompass a range of methods—from post-hoc and self-interpretable designs to model-level generation and representation probing—that address both local and global explanations.
• Recent advancements focus on improving explanation faithfulness, robustness under perturbations, and user-centered narrative outputs for practical, domain-specific insights.

Explainable Graph Neural Networks (XGNNs) are methods and frameworks designed to elucidate the decision-making processes of Graph Neural Networks, providing interpretability for their predictions on graph-structured data. Across the recent literature, XGNNs are organized by explanation scope, explanation method, domain applicability, and mode of explanation, and they span post-hoc and self-interpretable designs, local and global explanations, factual and counterfactual reasoning, and outputs ranging from node, edge, feature, and subgraph importance to concept-level, example-based, and natural-language explanations (Li et al., 2022, Kakkad et al., 2023, Tewari et al., 23 Sep 2025).

1. Conceptual scope and formalization

In the broadest formulation, XGNNs treat the graph as both a predictive object and an explanatory substrate. In dementia-oriented graph learning, for example, the graph is written as $G = (V, E, \mathbf{A})$, where $V$ is the set of brain ROIs, $E$ is the set of edges, and $\mathbf{A} \in \mathbb{R}^{N \times N}$ is a weighted adjacency matrix; a generic GNN layer is expressed as

$$h_v^{(l+1)} = \gamma^{(l)} \left(h_v^{(l)}, \bigoplus_{u \in \mathcal{N}(v)} \phi^{(l)}(h_v^{(l)}, h_u^{(l)}, A_{vu}) \right),$$

while an explainer function maps model, input, and prediction to an explanation object $\mathcal{E}$, which may contain node or edge importance scores, feature masks, or related explanatory artifacts (Tewari et al., 23 Sep 2025).

Two survey papers systematize this space. One proposes a two-part taxonomy distinguishing XAI-based XGNNs from non-XAI-based XGNNs, with non-XAI-based methods further divided into subgraph-based explanations, graph generation-based explanations, and intermediate-level injection (Li et al., 2022). Another survey introduces a hierarchy along the axes of factual versus counterfactual explanation, post-hoc versus self-interpretable integration modality, and methodological families including decomposition-based, gradient-based, surrogate, perturbation-based, generation-based, information constraint-based, and structural constraint-based methods (Kakkad et al., 2023). The dementia review adds four operational dimensions—explanation scope, explanation method, domain applicability, and mode of explanation—thereby making explicit the distinction between local and global explanation, intrinsic and extrinsic explanation, model-specific and model-agnostic explanation, and visual, score-based, and example-based outputs (Tewari et al., 23 Sep 2025).

A recurring implication of these taxonomies is that XGNN is not a single architecture class. It is a family of explanatory strategies for graph learning, and different members target different explanatory questions: which subgraph caused an instance-level prediction, what structural prototype maximally activates a class, what graph property is linearly encoded in hidden states, which counterfactual graph edit would change a label, or how to translate graph explanations into a form that domain experts or non-experts can assimilate (Li et al., 2022, Kakkad et al., 2023).

2. Model-level, instance-level, and representation-level explanation

Early work emphasized the distinction between instance-level and model-level explanation. "XGNN: Towards Model-Level Explanations of Graph Neural Networks" formulates model-level interpretation as graph generation: for a trained GNN classifier $f(\cdot)$ and a target class $c_i$, the method seeks a graph $G^*$ that maximizes the predicted probability for that class,

$G^* = \argmax_G P(f(G) = c_i).$

Rather than directly optimizing over discrete graph structure, XGNN casts graph generation as a reinforcement learning task in which the graph generator predicts how to add an edge into the current graph and is trained via a policy gradient method. Graph rules are incorporated to encourage the generated graphs to be valid, and the generated patterns are intended to reveal what graph motifs or structures the trained GNN has learned for a class (Yuan et al., 2020).

Instance-level explanation, by contrast, is dominated by subgraph- and mask-based methods. The surveys identify GNNExplainer, PGExplainer, GraphMask, PGM-Explainer, and SubgraphX as representative methods that learn or infer important subgraphs, node features, or probabilistic edge masks for a specific prediction or across multiple instances (Li et al., 2022, Li et al., 2022). A distinct line adapts established CNN explainers to graphs. "Edge-Level Explanations for Graph Neural Networks by Extending Explainability Methods for Convolutional Neural Networks" extends LIME, Gradient-Based Saliency Maps, and Grad-CAM to GNNs in order to identify which edges in the input graphs are important for GNN decisions; in its experimental results, the LIME-based approach is described as the most efficient explainability method for multiple tasks in the real-world situation, outperforming even the state-of-the-art method in GNN explainability (Kasanishi et al., 2021).

A third level of analysis is representation-level or model-internal probing. "Do graph neural network states contain graph properties?" presents a model agnostic explainability pipeline for GNNs employing diagnostic classifiers. Instead of asking which subgraph is important for one output, the method probes hidden states to determine whether well-known structural graph properties are linearly encoded in embeddings across layers, architectures, and datasets. The pipeline is explicitly model-level, property-centric, and scalable, and it reframes explanation as the quantitative recovery of graph-theoretic structure from intermediate representations rather than only as post-hoc masking or graph generation (Pelletreau-Duris et al., 2024).

These levels are complementary rather than mutually exclusive. Instance-level explanation answers why this prediction occurred; model-level generation answers what a trained model has learned in general; representation-level probing answers what kinds of graph information are organized inside the network. This suggests that a comprehensive account of XGNN behavior often requires more than one explanatory lens.

3. Intrinsic, self-explaining, and training-time XGNNs

A major development in XGNN research is the shift from post-hoc explanation toward explanation by design. "L2XGNN: Learning to Explain Graph Neural Networks" proposes a framework that provides faithful explanations by design by learning a mechanism for selecting explanatory subgraphs that are exclusively used in the GNNs message-passing operations. The selected subgraph can be constrained to be sparse and connected, and only that subgraph is used for prediction. In the paper’s formulation, the upstream model defines a distribution over feasible subgraphs $V$0,

$V$1

and the downstream GNN performs message passing on $V$2. The central claim is that the explanation is faithful because the prediction is entirely based on the explanation subgraph (Serra et al., 2022).

Training-time explainability can also be induced without replacing the base predictor. "A Meta-Learning Approach for Training Explainable Graph Neural Networks" introduces MATE, which meta-trains the model’s parameters to quickly minimize the error of an instance-level GNNExplainer trained on-the-fly on randomly sampled nodes. The framework steers optimization toward what it calls interpretable minima, and experiments on synthetic and real-world datasets for node and graph classification show that models can be made consistently easier to explain by different algorithms, with no cost for the accuracy of the model (Spinelli et al., 2021).

A more radical ante-hoc formulation appears in "X-Node: Self-Explanation is All We Need." X-Node is described as a self-explaining GNN framework in which each node generates its own explanation as part of the prediction process. For every node, a structured context vector encodes interpretable cues such as degree, centrality, clustering, feature saliency, and label agreement within its local topology; a lightweight Reasoner module maps this context into a compact explanation vector, which is then used to reconstruct the node’s latent embedding via a decoder, generate a natural language explanation using a pre-trained LLM, and guide the GNN itself via a text-injection mechanism that feeds explanations back into the message-passing pipeline (Sengupta et al., 14 Aug 2025).

Intrinsic explainability is also being built into graph architectures through explicit priors. "Structural Graph Neural Networks with Anatomical Priors for Explainable Chest X-ray Diagnosis" reconstructs CNN feature maps into patch-level graphs, augments nodes with normalized spatial coordinates, and uses a custom structural update rule,

$V$3

together with a learned node importance score

$V$4

The graph is treated as an inductive bias for structured inference rather than a passive relational representation, and the explainability is intrinsic rather than post-hoc (Berkani, 17 Jan 2026).

4. Evaluation, faithfulness, and theoretical limits

The evaluation of XGNNs is itself a substantial research area because existing graph datasets often lack reliable ground-truth explanations. "Evaluating Explainability for Graph Neural Networks" addresses this gap through ShapeGGen, a synthetic graph data generator that can produce benchmark datasets with controllable graph properties and ground-truth explanations, and through GraphXAI, an open-source library that provides datasets, data loaders, visualizers, GNN model implementations, and evaluation metrics. The paper formalizes Graph Explanation Accuracy (GEA) with the Jaccard index, Graph Explanation Faithfulness (GEF) through KL divergence, Graph Explanation Stability (GES) with cosine distance, and fairness mismatch metrics including GECF and GEGF (Agarwal et al., 2022).

A related experimental survey emphasizes the distinction between plausibility and faithfulness and proposes Explanation Confidence (EC) as a human-free, faithfulness-oriented, and normalized metric: $V$5 In experiments on citation networks and several explainers, GraphMask is reported to outperform GNNExplainer and PGExplainer in many settings, especially at higher EC, thereby foregrounding the importance of concise and faithful explanations rather than explanation sparsity alone (Li et al., 2022).

Beyond benchmarking, the recent theoretical literature questions whether some intrinsic XGNN architectures are faithful in principle. "How Interpretable Are Interpretable Graph Neural Networks?" introduces the subgraph multilinear extension (SubMT),

$V$6

and argues that extracting the desired interpretable subgraph requires an accurate approximation of SubMT. The paper states that existing XGNNs can have a huge gap in fitting SubMT, that SubMT approximation failure will lead to the degenerated interpretability of the extracted subgraphs, and that the proposed Graph Multilinear neT (GMT) is provably more powerful in approximating SubMT. Empirically, GMT is reported to outperform the state-of-the-art up to 10% in terms of both interpretability and generalizability across 12 regular and geometric graph benchmarks (Chen et al., 2024).

One common misconception in the area is that attention or soft edge weights are automatically explanations. The SubMT analysis directly disputes that assumption: if the learned soft subgraph is not a good approximation to the expected prediction over hard subgraphs, the resulting explanation can be structurally plausible yet theoretically unfaithful (Chen et al., 2024).

5. Robustness, distribution shift, and prerequisite-free explanation

Recent XGNN work has moved beyond standard IID evaluation toward distribution shift, black-box constraints, and adversarial robustness. "Towards Comprehensive and Prerequisite-Free Explainer for Graph Neural Networks" identifies two major limitations of existing XGNN methods: failure to capture the complete decision logic of GNNs across diverse distributions in the entire dataset’s sample space, and strict prerequisites on edge properties and GNN internal accessibility. OPEN addresses these limitations through a Non-Parametric Analysis Framework (NPAF), which infers and partitions the sample space into environments, and a Graph Variational Generator (GVAG), which samples and analyzes subgraphs within each environment. The paper characterizes OPEN as the first work in the literature that can infer and partition the entire dataset’s sample space into multiple environments, each containing graphs that follow a distinct distribution, and states that OPEN captures nearly complete decision logic of GNNs, outperforms state-of-the-art methods in fidelity while maintaining similar efficiency, and enhances robustness in real-world scenarios (Zhang et al., 20 May 2025).

OPEN is explicitly designed for settings in which prior assumptions are invalid. It does not require internal access to the GNN, and it does not require learnable edge weights to replace real edge features. This is significant for black-box deployment, privacy-sensitive domains, and settings in which edge semantics are not perturbable without corrupting the data-generating process (Zhang et al., 20 May 2025).

Robustness of the explanations themselves is the focus of "Provably Robust Explainable Graph Neural Networks against Graph Perturbation Attacks." XGNNCert is introduced as the first provably robust XGNN. Its framework builds hybrid subgraphs, applies majority-voting based classification and explanation, and derives a deterministic certificate ensuring that, under bounded edge perturbation, the GNN prediction does not change and at least $V$7 of the key explanation edges remain the same. The certified perturbation size is written as

$V$8

with the explanation-side margin determined by an additional voting inequality. The paper positions this as the first step toward explanation robustness under graph perturbation attacks and evaluates the method on multiple graph datasets and GNN explainers (Li et al., 6 Feb 2025).

Together, OPEN and XGNNCert mark a shift in XGNN research from explanation quality in static benchmarks toward explanation validity under environmental heterogeneity, prerequisite-free deployment, and worst-case attack models. A plausible implication is that future XGNN systems will be judged not only by fidelity and sparsity, but also by environment coverage and certified stability.

6. Natural-language, concept-level, and explainee-adaptive explanation

Another active direction is the transformation of technical graph explanations into forms that are more directly assimilable by human readers. "GraphXAIN: Narratives to Explain Graph Neural Networks" introduces GraphXAIN as the first model-agnostic and explainer-agnostic framework to automatically generate natural language narratives that explain individual predictions of GNNs. The method takes technical outputs from graph explainers—explanatory subgraphs and feature importance scores—formats them together with dataset description, target node features, edge connections, and model prediction into a prompt, and uses an LLM to generate a narrative explanation. The paper explicitly distinguishes a XAI Narrative from a XAI Description and defines GraphXAIN as “A GraphXAIN is a XAI Narrative tailored for graph-structured data.” Its evaluation reports that GraphXAIN enhances understandability, satisfaction, convincingness, and suitability for communicating model predictions, and that when combined with another graph explainer method it further improves trustworthiness, insightfulness, confidence, and usability; notably, 95% of participants found GraphXAIN to be a valuable addition to the GNN explanation method (Cedro et al., 2024).

The move toward human-centered explanation is not limited to LLM narration. "EiX-GNN : Concept-level eigencentrality explainer for graph neural networks" models explanation as a social explainer-to-explainee dependence. Its key construct is explainee concept assimibility $V$9, which controls explanation granularity via concept size $E$0. Multiple subgraph concepts are sampled, a concept-relevance matrix is built, and PageRank-style eigencentrality is used to order concepts and reduce redundancy. The final explanation combines central concept ordering with local Shapley values, yielding node-level relevance scores tailored to the explainee’s assimibility (Raison et al., 2022).

This line of work reframes the goal of XGNNs. Instead of treating explanation only as faithful subgraph recovery, it treats explanation as a knowledge-transfer process whose success depends on context, narrative structure, and the explainee’s background. At the same time, the GraphXAIN paper notes that LLMs may hallucinate when working on complex or unstructured graphs, suggesting further research on prompt engineering or augmented LLMs (Cedro et al., 2024). That caveat underscores that communicative adequacy and faithfulness are related but not identical objectives.

7. Domain applications and open research directions

XGNNs have been deployed or surveyed in a wide range of application domains. In meteorology, CloudNine combines a GNN for atmospheric state estimation with Layer-wise Relevance Propagation to analyze individual meteorological observations’ impacts on specific predictions. It represents observation points and NWP grid points as nodes, uses spatial adjacency within a 50 km radius for edges, and supports local and global explanations of observation impact across space and time through an interactive web application (Jeon et al., 2024). In automated driving, "Explainable Scene Understanding with Qualitative Representations and Graph Neural Networks" integrates Qualitative Explainable Graphs (QXGs) with a GAT-based architecture for relevant object identification in traffic scenes, preserving qualitative spatio-temporal relationships and using attention weights as an interpretability resource (Belmecheri et al., 17 Apr 2025). In medical imaging, structural graph reasoning with anatomical priors yields intrinsic explainability for chest X-ray diagnosis through patch-level graphs, coordinate-aware message passing, and learned node importance scores (Berkani, 17 Jan 2026). In brain connectomics, a recent review presents the first comprehensive review dedicated to XGNNs in dementia research and argues that XGNNs enable the identification of disease-relevant biomarkers, analysis of brain network disruptions, and provision of transparent insights for clinicians across Alzheimer’s disease, Parkinson’s disease, mild cognitive impairment, and multi-disease diagnosis (Tewari et al., 23 Sep 2025).

The application literature also clarifies persistent open problems. Surveys emphasize the lack of reliable ground-truth explanations for real-world graphs, limited scalability of many perturbation and search-based explainers, the difficulty of evaluating fairness and robustness, and the absence of unified principles for extending XAI methods to graph-structured data (Agarwal et al., 2022, Li et al., 2022). The broader explainability survey highlights additional gaps in combinatorial algorithms, global explainability, temporal graphs, visualization and human-centric tools, and mixed or integrated approaches that combine factual, counterfactual, and causal explanation (Kakkad et al., 2023). The dementia review stresses limited generalizability, underexplored domains, the lack of temporal and multimodal explanations, and the clinical interpretation gap, while also identifying LLM integration as an emerging direction (Tewari et al., 23 Sep 2025).

Taken together, these developments indicate that XGNNs are evolving from local subgraph attribution methods into a heterogeneous research program that includes model-level generation, probing of learned states, explanation-aware training, self-explaining nodes, concept-level and narrative explanations, environment-aware and certified-robust frameworks, and domain-specific systems built around structural priors and human interpretability constraints. This suggests that the future of XGNNs will likely depend on simultaneously satisfying several criteria that earlier work often treated separately: faithfulness, communicability, robustness, generalizability, and domain alignment.