GraphIP-Bench: GNN Extraction & Defense

• GraphIP-Bench is a unified benchmark that standardizes evaluation of GNN extraction attacks and defenses under a query-only black-box protocol.
• It integrates extraction attacks, defenses, public graphs, and standardized metrics to measure fidelity, task utility, and computational cost.
• The benchmark reveals that medium query budgets allow high extraction fidelity, yet many defenses fail to preserve ownership verification after extraction.

GraphIP-Bench is a unified benchmark for evaluating graph neural network model extraction and defenses under a single black-box protocol. It was introduced to answer two questions that prior work could not answer under comparable conditions: how hard it is to steal a GNN, and whether existing defenses can stop such theft. The benchmark integrates extraction attacks, defenses, datasets, backbones, tasks, budgets, and reporting conventions into a common experimental framework, and adds a joint attack-and-defense track that measures whether ownership signals survive extraction onto the surrogate model (Zhao et al., 12 May 2026).

1. Naming, definition, and scope

GraphIP-Bench is the title used by the paper "GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?" (Zhao et al., 12 May 2026). In that work, the benchmark is defined around GNN theft via model-extraction attacks and the empirical evaluation of ownership and endpoint defenses under a query-only, black-box setting. Its central contribution is a standardized protocol with public splits, shared query sets, fixed query budgets, explicit endpoint assumptions, and matched reporting of fidelity, task utility, ownership verification, and computational cost.

The benchmark was motivated by fragmented experimental practice. Prior studies were described as using private splits, incompatible budgets, inconsistent metrics, and varying endpoint assumptions such as labels-only versus confidence scores. The paper also argues that existing graph-security benchmarks emphasize robustness or privacy rather than model extraction together with watermarking and fingerprinting, and that they omit computational cost and joint evaluation of attacks against defended targets (Zhao et al., 12 May 2026).

Two nearby names in the literature are distinct. The benchmark suite introduced in "Graph topology inference benchmarks for machine learning" does not use the alias GraphIP-Bench in the paper itself; it is a task-driven benchmark for graph topology inference rather than model extraction (Lassance et al., 2020). Likewise, the framework "GraIP: A Benchmarking Framework For Neural Graph Inverse Problems" consistently uses the acronym GraIP, and the provided text explicitly states that the term "GraphIP-Bench" does not appear there (Cantürk et al., 26 Jan 2026). This suggests that GraphIP-Bench should be understood specifically as the GNN extraction-and-defense benchmark of 2026, not as a generic label for graph inference benchmarks.

2. Threat model and black-box protocol

GraphIP-Bench adopts a query-only, black-box setting in which a deployed target GNN $f$ serves predictions through an inference API, and an adversary trains a surrogate $g$ using only returned outputs (Zhao et al., 12 May 2026). The attacker is constrained by a query budget $B$, and the benchmark standardizes both endpoint settings in which the service returns hard labels or probability vectors/logits. The formalization uses an attributed graph $G = (V, E, X, A)$ with node set $V$, edge set $E$, node-feature matrix $X$, and adjacency $A$, and fixed dataset splits $D = (X_t, X_v, X_{\text{test}}, X_q)$ for train, validation, test, and query subsets.

Extraction is defined by drawing queries $x \in X_q$ subject to $g$0, receiving either $g$1 or $g$2, and training the surrogate to minimize discrepancy with the target on collected query-response pairs. The benchmark reports budgets at $g$3, $g$4, $g$5, $g$6, and $g$7 times the test-set size. This fixed budget schedule is central to the benchmark’s comparability claims because it enforces identical resource constraints across attacks and defended targets (Zhao et al., 12 May 2026).

The benchmark also defines four graph-access regimes through attack_x_ratio and attack_a_ratio: features only, structure only, both, and data-free. In the features-only regime, the attacker has real features but synthetic or unknown adjacency; in the structure-only regime, real adjacency but synthetic or unknown features; in the both regime, access to both real features and adjacency; and in the data-free regime, no real input, with queries synthesized instead. Structural queries are permitted only insofar as the regime allows them. A plausible implication is that GraphIP-Bench does not treat model extraction as a single monolithic threat model, but as a family of access assumptions that can materially affect empirical conclusions.

3. Benchmark composition: attacks, defenses, datasets, backbones, and tasks

GraphIP-Bench integrates twelve extraction attacks and twelve defenses spanning multiple methodological families (Zhao et al., 12 May 2026). The attack side includes the six-baseline MEA family, AdvMEA, CEGA, Realistic, and three data-free variants DFEA_I, DFEA_II, and DFEA_III. The defense side includes ownership-tracing methods such as RandomWM, BackdoorWM, SurviveWM, ImperceptibleWM, and Integrity; output perturbation methods OP_low and OP_high; prediction rounding methods PR_2bit and PR_top1; and query-pattern detection methods PRADA, AdaptMisinfo, and GradRedir.

Several attack objectives are formalized explicitly. Hard-label distillation uses

$g$8

while soft-label distillation uses

$g$9

For DFEA_III, the consistency term is

$B$0

with total loss $B$1. On the defense side, output perturbation is written as $B$2, $B$3, followed by $B$4, and ownership verification is standardized through a verification rate on a watermark set.

The dataset coverage spans ten public graphs for the node-classification core benchmark: Cora, CiteSeer, PubMed, Computers, Photo, CoauthorCS, CoauthorPhysics, OGBN-Arxiv, RomanEmpire, and AmazonRatings. These are chosen to represent homophilic, heterophilic, and large-scale regimes. The benchmark further uses three backbones—GCN, GAT, and GraphSAGE—and three tasks, with node classification as the primary task and link prediction and graph classification included in the appendices (Zhao et al., 12 May 2026).

The following summary captures the benchmark’s main components.

Component	Coverage	Notes
Extraction attacks	12	Includes MEA0–MEA5, AdvMEA, CEGA, Realistic, DFEA_I–III
Defenses	12	Includes watermarking, output perturbation, prediction rounding, query-pattern detection
Public graphs	10	Homophilic, heterophilic, and large-scale regimes
Backbones	3	GCN, GAT, GraphSAGE
Tasks	3	Node classification, plus link prediction and graph classification extensions

This breadth is one of the benchmark’s defining properties. The paper’s formulation makes clear that GraphIP-Bench is not only a leaderboard of attacks or a catalog of defenses; it is an attempt to standardize the entire evaluation surface on which theft and protection are studied.

4. Metrics, reporting conventions, and the joint evaluation track

GraphIP-Bench reports four metric families: fidelity, task utility, ownership verification, and computational cost (Zhao et al., 12 May 2026). Fidelity is the agreement rate between target and surrogate on the test set,

$B$5

with KL-based fidelity also available when soft labels are exposed. Task utility is measured by accuracy and macro F1 on $B$6. Ownership verification is measured on a standardized watermark set $B$7 through

$B$8

where $B$9 is either the protected target or the extracted surrogate. Computational cost includes queries, wall-clock time, peak GPU memory, and GPU hours.

A distinctive feature is the joint attack-and-defense track. The protocol first trains and deploys a defended target $G = (V, E, X, A)$0, then runs each attack against that defended model at a standardized budget, trains the surrogate against its responses, measures surrogate fidelity on $G = (V, E, X, A)$1, and finally measures watermark survival on the surrogate via $G = (V, E, X, A)$2. This procedure is designed to expose whether protection survives extraction rather than merely verifying on the protected model itself (Zhao et al., 12 May 2026).

This joint track addresses a specific limitation in prior evaluation practice. A defense may achieve high verification on the defended target while leaving the surrogate both accurate and effectively untraceable by the owner’s chosen watermark. The benchmark therefore distinguishes between protected-model verification and post-extraction verification survival. This suggests that GraphIP-Bench treats ownership tracing as a transfer property under adversarial distillation, not merely as a static property of the original model.

The reporting conventions are also standardized at the infrastructure level. The benchmark uses shared splits, identical query sets, standardized budgets, three seeds $G = (V, E, X, A)$3, unified loaders and configuration defaults, and identical hardware and software, with code released for reproducibility (Zhao et al., 12 May 2026). In a benchmarking context, these controls are part of the benchmark’s methodological content rather than ancillary engineering details.

5. Empirical findings

The paper’s main empirical conclusion is concise: stealing a GNN is easy at medium query budgets, and most defenses do not materially change this outcome (Zhao et al., 12 May 2026). The strongest improvements in extraction are reported between budgets of $G = (V, E, X, A)$4 and $G = (V, E, X, A)$5, with saturation near $G = (V, E, X, A)$6. On homophilic graphs such as Cora, the MEA family reaches high fidelity quickly: MEA0 at $G = (V, E, X, A)$7 reaches $G = (V, E, X, A)$8, MEA2 at $G = (V, E, X, A)$9 reaches $V$0, and MEA2 at $V$1 reaches $V$2. On Computers, MEA3 at $V$3 reaches $V$4, MEA4 at $V$5 reaches $V$6, and DFEA_III reaches $V$7 at $V$8 and $V$9 at $E$0.

The large-scale and heterophilic settings are more resistant, but not uniformly so. On OGBN-Arxiv, the strongest attacks reach approximately $E$1–$E$2 fidelity at $E$3 budget, while AdvMEA drops to approximately $E$4, which the paper interprets as evidence that class-space complexity rather than scale is the main bottleneck. On RomanEmpire, whose edge homophily is reported as $E$5, the strongest data-driven attacks reach approximately $E$6 fidelity at medium budgets, while AdvMEA collapses to approximately $E$7. On AmazonRatings, the strongest attacks still reach at least $E$8 fidelity. The benchmark therefore reports that heterophilic graphs are systematically harder to steal, while also showing that heterophily does not eliminate extraction success (Zhao et al., 12 May 2026).

Defenses have sharply differentiated effects. Watermarking methods generally do not materially reduce surrogate fidelity, with strong attacks reaching within a few points of undefended baselines. Output-side defenses and prediction rounding are more effective operationally: PR_top1 consistently yields perfect verification on protected models and reduces extraction fidelity without heavy accuracy loss, while OP_low preserves accuracy with verification close to noise-free and OP_high reduces verification on heterophilic and high-class graphs. Query-pattern detection methods such as PRADA and AdaptMisinfo reduce surrogate fidelity but can incur substantial clean-accuracy costs; on Cora, the paper reports accuracy drops to $E$9 and $X$0, respectively. GradRedir often achieves perfect verification with modest accuracy changes, though accuracy is reduced on Photo (Zhao et al., 12 May 2026).

The benchmark’s most pointed finding concerns watermark survival after extraction. On protected models, BackdoorWM and ImperceptibleWM reach $X$1 verification; SurviveWM and RandomWM vary across datasets; Integrity is high but bimodal due to proxy design. On extracted surrogates for Computers at $X$2 budget, however, BackdoorWM drops from $X$3 verification on the target to $X$4 on the surrogate; SurviveWM from $X$5 to $X$6; RandomWM from $X$7 to $X$8; ImperceptibleWM from $X$9 to $A$0; while Integrity remains at $A$1 because its verification is query-time rather than parameter-side. This result grounds the paper’s central claim that single-model evaluations can miss a gap between verifying on the protected model and verifying on the extracted one (Zhao et al., 12 May 2026).

6. Reproducibility, efficiency, and operational guidance

GraphIP-Bench was released with code at https://github.com/LabRAI/GraphIP-Bench, fixed seeds, a unified Conda environment, and explicit hardware and software specifications: NVIDIA A100 80GB, PyTorch 2.2.1+cu121, DGL 2.1.0, and PyG 2.7.0 (Zhao et al., 12 May 2026). Output locations are documented for the benchmark’s research questions, including attack results, defense results, and the joint attack-defense track. Default defense hyperparameters are also listed, such as watermark-node ratio $A$2 for RandomWM, trigger density $A$3 for BackdoorWM, strength $A$4 for SurviveWM, epsilon $A$5 for ImperceptibleWM, and $A$6 and $A$7 for OP_low and OP_high.

Efficiency measurements are part of the benchmark rather than a secondary appendix. Under the both regime and at budget $A$8, the MEA family and CEGA typically take $A$9–$D = (X_t, X_v, X_{\text{test}}, X_q)$0 minutes, AdvMEA is slower and more variable, and Realistic is described as prohibitively expensive, requiring hundreds to thousands of minutes. On the defense side, BackdoorWM trains in approximately $D = (X_t, X_v, X_{\text{test}}, X_q)$1 seconds, SurviveWM in approximately $D = (X_t, X_v, X_{\text{test}}, X_q)$2–$D = (X_t, X_v, X_{\text{test}}, X_q)$3 seconds, Integrity in approximately $D = (X_t, X_v, X_{\text{test}}, X_q)$4–$D = (X_t, X_v, X_{\text{test}}, X_q)$5 seconds, while ImperceptibleWM incurs much heavier training cost, for example $D = (X_t, X_v, X_{\text{test}}, X_q)$6 seconds on Cora, with long memory tails (Zhao et al., 12 May 2026).

The benchmark also provides practical recommendations for cloud providers. It recommends returning only top-1 labels through PR_top1 to limit information leakage while maintaining utility, adding small Gaussian noise to logits with OP_low, enforcing query budgets and rate limits to deter medium-budget extraction, and deploying query-pattern detectors such as PRADA or GradRedir with careful threshold calibration because of their benign-accuracy trade-offs. For ownership tracing, it recommends query-time verification in the style of Integrity and argues that parameter-side watermarks should be evaluated primarily on extracted surrogates rather than only on protected models (Zhao et al., 12 May 2026).

These recommendations are operational rather than normative. They follow directly from the benchmark’s empirical observation that output-side restrictions and query-time mechanisms retain utility or survival properties more reliably than most trigger-based or parameter-side watermarks when the threat model is actual extraction.

7. Relation to adjacent benchmark traditions

GraphIP-Bench belongs to a broader landscape of graph benchmarking, but its scope is distinct. The benchmark "Graph topology inference benchmarks for machine learning" evaluates graph inference methods through downstream tasks such as unsupervised clustering of vertices, semi-supervised classification of vertices, and denoising of graph signals; it is publicly released and task-driven, but it is not a model-extraction benchmark and does not use GraphIP-Bench as its official name (Lassance et al., 2020). GraIP, by contrast, unifies neural graph inverse problems such as causal discovery, neural relational inference, rewiring, combinatorial optimization, and gene regulatory network inference under an inverse-problem formulation, again with datasets, metrics, and baselines, but the provided text explicitly notes that "GraphIP-Bench" does not appear in that work (Cantürk et al., 26 Jan 2026).

A different adjacent tradition is performance benchmarking for GNN systems. The framework-independent suite gSuite focuses on GPU inference performance, kernel behavior, computational models such as message passing versus SpMM, and simulator-friendly implementation, rather than on adversarial extraction or ownership defenses (Tekdoğan et al., 2022). Its relevance is architectural: it illustrates how benchmark design can standardize kernels, workloads, and metrics for reproducible systems analysis.

Against these neighboring efforts, GraphIP-Bench is novel in centering the security question of stealing and stopping GNNs under one standardized black-box protocol (Zhao et al., 12 May 2026). Its principal methodological move is to benchmark attacks, defenses, and their composition jointly. This suggests a broader shift in graph ML evaluation: from isolated, task-specific claims toward protocol-level comparability in which the unit of evaluation is not merely a model, but an interaction between access assumptions, query budgets, endpoint policies, and ownership criteria.