B-XAIC: Benchmark for Explainable GNNs in Chemistry

• B-XAIC is a real-world benchmark that evaluates explainable AI methods for GNNs in molecular chemistry using a curated dataset from ChEMBL 35.
• It introduces dual metrics—Null Explanation (NE) and Subgraph Explanation (SE)—to assess both uniform and targeted attributions via IQR analysis and AUROC.
• Results indicate that while GNNs achieve high classification accuracy, current explainers often fail to consistently identify chemically meaningful substructures.

B-XAIC is a real-world benchmark designed to rigorously evaluate Explainable AI (XAI) methods for Graph Neural Networks (GNNs) within molecular chemistry. Developed to address the shortcomings of existing evaluation frameworks—namely, reliance on artificial data and weak metrics—B-XAIC provides a curated, diverse dataset from ChEMBL 35 and introduces ground-truth local rationales at both the atom and bond level. Its dual-metric protocol reveals significant limitations of leading GNN explainers in faithfully identifying chemically meaningful substructures, even when classification performance is near perfect (Proszewska et al., 28 May 2025).

1. Data Composition and Annotation

B-XAIC draws 50 000 molecules from ChEMBL 35 (∼2.5 million drug-like compounds), filtering for single, connected molecular graphs and removing invalid or duplicate SMILES, solvents, and counter-ions. Selection weights are proportional to the product of majority/minority task label ratios to mitigate class imbalance across tasks.

Each molecule averages 34.56 atoms and is partitioned into 80% training, 10% validation, and 10% test splits. Molecules are annotated with seven binary tasks, each grounded in a specific chemical motif or property, as summarized below.

Task	Positive Rate	Rationales: Subgraph Size
Boron (B)	2.18%	4.13% ± 2.16% nodes
Phosphorus (P)	12.78%	4.35% ± 2.44% nodes
Halogen (X)	56.46%	6.05% ± 4.07% nodes
Indole ring	36.94%	31.34% ± 12.15% nodes / 31.49% edges
PAINS patterns	32.88%	34.07% ± 14.22% nodes / 31.37% edges
Rings-count	30.06%	64.04% ± 16.03% nodes / 61.85% edges
Max-ring size	5.54%	50.22% ± 17.44% nodes / 47.12% edges

For positive labels, ground-truth is provided as subgraph-level annotations (“rationales”) specifying atoms and bonds participating in the chemical motif. For negative cases (“Null Explanations” or NE), no atom or bond is privileged.

2. Evaluation Metrics and Their Formalization

B-XAIC introduces two complementary attribution-regime metrics:

A. Null Explanation (NE) Regime:

For negative examples, correct explanations should assign uniform attribution across all atoms/bonds. Outlier detection is performed using interquartile range (IQR) analysis of the set of attributions $\{a_i\}$:

• Define $Q_1$ and $Q_3$ as the first and third quartiles; $IQR = Q_3 - Q_1$.
• Identify outliers as $$O = \{i: a_i < Q_1 - 1.5\,IQR \ \text{or} \ a_i > Q_3 + 1.5\,IQR \}$$.
• NE score for a graph: $NE = 1$ if $|O| = 0$ (no outliers), else $NE=0$.
• Overall NE: fraction of correctly uniform explanations across all negative graphs.

B. Subgraph Explanation (SE) Regime:

For positive examples, a subset $S$ of nodes (or edges) is labeled as relevant. The quality of an explainer is assessed by the AUROC between indicator ground-truths $y_i = \mathbb{I}_{i \in S}$ and predicted importances $Q_1$0:

• $Q_1$1
• No explicit thresholding is required, allowing for arbitrary rationale subgraph sizes.
• SE is averaged over all positive test examples.

Node- and edge-level explanations are evaluated separately. This protocol reveals deficiencies in disproportionately highlighting irrelevant substructures (low NE) or failing to focus on important subgraphs (low SE).

3. Benchmark Tasks, GNNs, and XAI Methodology

The seven annotation tasks target atomic motifs (B, P, X), complex subsystems (indole rings, PAINS), and global molecular properties (ring count, maximum ring size). Each task presents distinct challenges for both classification and explanation.

Model architectures:

• Standard GNNs: GCN, GAT, GIN.
• Prototype-based: ProtGNN atop each backbone.

XAI Methods are grouped as:

• Gradient-based: Saliency (Simonyan et al.), Deconvolution, GuidedBackprop, Input×Gradient, Integrated Gradients.
• Graph-specific: GNNExplainer, PGExplainer, GraphMask.
• Perturbation-based: ShapleyValueSampling.

Experimental protocol utilizes a 40 000/5 000/5 000 train/val/test split. Classification is performed to near-perfect F1 (GIN: F1 > 98% on most tasks) and explanations are scored post-hoc using NE/SE. Statistical significance is assessed with a one-sided Wilcoxon test. Experiments are executed on NVIDIA H100 hardware with 80 GB HBM3.

4. Observed Outcomes and Methodological Insights

• Classification: GIN outperforms GCN and GAT (F1 > 98% except indole ≈ 88%, PAINS ≈ 79%). Tasks involving ring counting and PAINS detection are most difficult for classifiers.
• Explanation Performance:
  • Node-level: Saliency and GuidedBackprop deliver the highest SE (AUROC ≈ 0.82–0.84) but low NE (frequent spurious highlighting). GNNExplainer and GraphMask excel in NE (≈ 0.70), but have reduced SE (≈ 0.50). The best overall node-level trade-off (avg = (NE+SE)/2 ≈ 0.69) is attained by gradient-based explainers with ProtGNN+GIN or GIN backbone.
  • Edge-level: GNNExplainer reaches the highest NE (≈ 0.80), while GuidedBackprop achieves the top SE (≈ 0.65). The overall best edge-level average is ≈ 0.68 for ProtGNN+GIN+GuidedBackprop.
• Task-dependency: Explanatory precision decays sharply for large or composite motifs (indole, PAINS) and counting tasks, but remains high for tasks linked to presence of single atoms.
• Architectural observation: Message-passing diffusion in GNNs disperses attribution, impeding localization even when prediction is accurate.
• Limitations: State-of-the-art explainers exhibit a trade-off—some over-highlight (low NE), others under-highlight (low SE); none consistently achieve high scores on both. Even in presence of robust prediction, explanation mechanisms often do not recover chemically relevant logic.

5. Constraints, Benchmark Scope, and Known Limitations

B-XAIC confines itself to local explanations—ground-truth rationales are provided at a per-molecule level, with global dataset-level drivers of model behavior left unaddressed. Real-molecule data ensure clinical relevance but introduce uncertainty regarding the specific logic a model has captured, especially in ambiguous scenarios; model mislearning, rather than explainer deficiency, may underlie poor explanations.

The protocol’s dependence on ChEMBL 35 compounds, as well as discrete binary task definitions, frame what aspects of explanation can be benchmarked. Use of atom- and bond-level subgraphs (rather than, for instance, learned hierarchical motifs) defines the ground-truth granularity.

6. Implementation, Distribution, and Prospective Extensions

B-XAIC is openly distributed under a CC-BY-SA license, with full dataset, predefined splits, and evaluation scripts hosted at https://huggingface.co/datasets/mproszewska/B-XAIC and https://github.com/mproszewska/B-XAIC. Researchers are encouraged to apply both NE and SE protocols, ensuring comparative rigor along orthogonal axes of faithfulness.

Potential extensions proposed include:

• Introduction of activity-cliff scenarios, testing explanation robustness to minimal but property-critical chemical edits.
• Inclusion of global (dataset-level) explanation evaluation.
• Supplementation with fidelity metrics (e.g., insertion/deletion analysis) in addition to accuracy-based AUROC/IQR-based scores.
• Architectural innovations or training modifications to minimize information diffusion—tightening attribution locality.

B-XAIC establishes a high-fidelity, real-world standard for GNN explainability assessment, diagnosing core deficits in localization and grounding of chemical explanations, and provides a scalable, reproducible protocol for future advances in molecular graph XAI (Proszewska et al., 28 May 2025).