GraphNetz: Statistical Benchmarking of Graph Neural Networks with Paired Tests and Rank Aggregation

Abstract: Graph Neural Networks (GNNs) benchmarks often report single point estimates, even when performance differences are small relative to variation across random seeds, train/test splits, and datasets. Confidence intervals, paired comparisons, multiple-comparison correction, and rank-based aggregation are standard statistical tools, but they are rarely the default output of graph-learning benchmark suites. We introduce GraphNetz, a benchmarking framework whose default output is a structured statistical report rather than a raw accuracy table. GraphNetz currently includes 63 dataset loaders, four task types, and five canonical GNN architectures, while also supporting custom datasets and models. The framework standardizes multi-seed evaluation and automatically returns per-cell confidence intervals, Holm-corrected paired tests, and Friedman-Nemenyi critical-difference diagrams across tasks. In a cross-category benchmark over ten heterogeneous tasks, apparent rank differences among four canonical node-level encoders fall within a single Nemenyi clique, indicating that none is significantly better than the others at $α= 0.05$. GraphNetz therefore provides researchers with a reproducible computational and statistical pipeline to benchmark new graph-learning methods against standard architectures, over different tasks and a wide set of applications, while reporting principled statistical evidence for benchmarking which accounts for seed uncertainty. This framework is set to serve the graph-learning community with a reproducible and honest model comparison ready to be added to papers.

• The paper introduces GraphNetz, a framework that employs seed-paired confidence intervals and multiple comparison corrections to assess GNN performance.
• The paper demonstrates that canonical GNN models show no statistically significant differences when evaluated with robust parametric and non-parametric tests.
• The paper outlines a reproducible protocol with deterministic seeding and rank aggregation, ensuring fair and calibrated model comparisons.

Statistical Benchmarking of GNNs: The GraphNetz Framework

Motivation and Context

Benchmarking practices for Graph Neural Networks (GNNs) have serious limitations due to the pervasive reporting of point statistics without sufficient attention to variance induced by random seeds, dataset splits, and multiple hypothesis testing. This lack of statistical rigor leads to the presentation of architecture rankings and performance claims that do not reflect actual statistical significance. Prior work has demonstrated that when appropriate statistical controls are applied, reported differences between canonical GNN architectures often vanish, suggesting that much of the literature’s “progress” is attributable to evaluation artifacts rather than genuine modeling advances. The "GraphNetz: Statistical Benchmarking of Graph Neural Networks with Paired Tests and Rank Aggregation" (2605.09099) paper introduces a comprehensive benchmarking suite, GraphNetz, designed to address these deficiencies by embedding standardized statistical analysis directly into the benchmarking pipeline.

Framework and Methodology

GraphNetz provides a fundamentally different default interface: instead of yielding raw accuracy tables, it emits structured statistical reports including seed-paired confidence intervals (CIs), paired statistical significance testing with multiple comparison correction, and rank-based aggregation diagrams. Its framework covers:

• Datasets: 63 prepackaged loaders mapped to a ten-category taxonomy including combinatorics, biology, social networks, infrastructure, finance, computing, vision, physics, security, and knowledge graphs. The framework exposes a task-type taxonomy (node classification, graph classification, graph regression, link prediction) and offers formal documentation of coverage gaps.
• Models: Five established GNN encoders—GCN, GAT, GIN, GraphSAGE, and Graph Transformer—with adapters for all task types. The system supports custom architectures via an extensible API.
• Protocol: For each defined benchmark run, every compatible (model, task) pair is evaluated over multiple seeds (default S = 10), and all random number generators are deterministically reseeded per trial. Results (test/validation metrics) are aggregated along three axes: per-cell (mean ± Student’s t CI), per-task (Holm–Bonferroni corrected pairwise tests, both parametric and robust non-parametric alternatives), and across tasks (Friedman–Nemenyi rank aggregation with critical-difference diagrams).
• Statistical Layer: Statistical artifacts include per-cell CIs, pairwise significance tables for all model comparisons within each task, and critical-difference (CD) diagrams visualizing aggregate rank relationships across heterogeneous tasks.

Empirical Results

A cross-category evaluation comprising ten representative datasets (one per research area) and five GNN families demonstrates the practical implications of principled benchmarking. The experiment employs ten random seeds per (model, task), yielding reliable estimation of within-cell metric distribution. Key findings include:

• Absence of Significant Differences: Aggregate analysis via Friedman and Nemenyi procedures reveals that none of the four canonical node-level GNN encoders (GCN, GAT, GraphSAGE, Graph Transformer) are significantly better than the others at standard significance levels ($\alpha = 0.05$). The maximum mean-rank separation among these encoders is 0.8, well below the calculated Nemenyi critical difference ($\mathrm{CD}_{0.05} = 1.48$), resulting in all encoders falling in a single indistinguishable clique.
• Pairwise Tests Robustness: Parametric (paired t-test) and non-parametric (Wilcoxon signed-rank) tests were both applied (with Holm correction) within each task and yielded total agreement, confirming the stability of findings with $S = 10$ seeds. The framework reports both, facilitating inspection of statistical reliability and normality assumptions.
• Reproducibility and Determinism: The protocol’s deterministic seed threading extends to user-supplied datasets and synthetic data generation, ensuring that results are fully reproducible under a fixed software environment.

Framework Extensions and Recommendations

GraphNetz is explicitly designed for extensibility—supporting one-line registration of custom datasets and GNN models. The essential recommendation for the field is that honest benchmarking must report:

• Per-cell multi-seed confidence intervals (preferably percentile-bootstrap for non-Gaussian metrics)
• Multiple-comparison-corrected, seed-paired pairwise significance tests
• Rank aggregation with critical-difference diagrams across all benchmarked tasks

Moreover, the statistical pipeline should be integrated into the default reporting surface in order to avoid cherry-picking and overconfident claims, making calibrated assessment accessible to typical users. The framework can be immediately utilized for benchmarking future GNNs or new application domains with no modifications to its internals.

Limitations and Implications

The experiment’s main limitation is the breadth of tested datasets (N = 10), which, while spanning all major application domains, constrains the power of the Friedman–Nemenyi step to detect small but consistent architectural differences. As the CD threshold shrinks with increased $N$, future evaluations should scale the number of datasets and tasks to increase sensitivity. Additionally, all evaluations were performed using fixed (but standard) hyperparameters and at fixed training epochs; richer hyperparameter sweeps and model selection protocols can be incorporated by extending the inner benchmarking loop.

Contradictory to the common narrative in the GNN literature, GraphNetz empirically demonstrates that past headline performance gains among standard GNN architectures do not reach statistical significance under principled controls. This finding invalidates many prior claims of architectural superiority that did not control for seed and split variance or false discovery rates.

Broader Impact and Future Trajectories

GraphNetz neither proposes new architectures nor datasets but rather a pragmatic methodological intervention: reducing friction for the adoption of statistically-valid model comparison. The framework’s statistical apparatus and deterministic protocol can be generalized to any ML benchmarking context. Its future development trajectory will likely emphasize integration of emerging graph foundation models, further expansion of the catalog to close noted coverage gaps, and support for real-world and large-scale environments.

Conclusion

GraphNetz sets a new standard for methodological rigor in GNN benchmarking. By making statistical validation, multi-seed evaluation, and principled rank aggregation the default, it provides calibrated evidence for the actual significance of architectural differences. Its findings sharply underline the necessity for the community to revise its benchmarking norms and curb the propagation of unsupported claims arising from unreliable evaluation pipelines. The framework simultaneously enhances reproducibility and accelerates more accurate empirical science in graph-based machine learning (2605.09099).