GraphBench: Unified Graph Benchmark Suites

• GraphBench is a collection of standardized benchmarking suites designed for graph machine learning, optimization, embedding, and inference tasks.
• It enforces rigorous evaluation protocols with consistent data splits, specialized metrics, and automated experimental pipelines.
• GraphBench benchmarks both classic and state-of-the-art models across diverse domains, fostering fair model comparison and research transparency.

GraphBench refers to a family of comprehensive, standardized benchmarking suites that serve distinct purposes across graph machine learning, graph optimization, graph embedding, graph system, and graph inference tasks. Diverse research groups have developed independent iterations of GraphBench to address critical needs for reproducibility, fair comparison, and rigorous evaluation in their respective graph-research subfields. Below, key design paradigms, benchmarked domains, methodological advances, and common themes are articulated, referencing the authoritative literature.

1. Definitions and Variants

The designation "GraphBench" is attributed to multiple publicly released benchmarking infrastructures, each targeting a specific class of graph computing problems:

• GraphBench for Graph Learning: Unified benchmark for node-, edge-, graph-level, and generative learning on graphs, emphasizing consistent splits and evaluation across diverse domains such as social networks, hardware design, algorithmic reasoning, and combinatorial optimization (Stoll et al., 4 Dec 2025).
• GraphBench for Graph Foundation Models (GFM): Rigorous suite for evaluating the generalization, scalability, and efficiency of self-supervised GNN pre-training, encompassing node classification, link prediction, and clustering, and analyzing hardware metrics across small and very large graphs (Xu et al., 2024).
• GraphBench for Combinatorial Optimization: Standardized platform for empirical/theoretical comparison of algorithms (evolutionary, black-box, learning-based) on NP-hard graph problems (MIS and MC), with well-characterized random and structured graph generators (Benito et al., 27 May 2026).
• GraphBench for Graph Embedding: Benchmarking corpus and metrics (GFS-score) for low-dimensional node embedding and link prediction techniques, emphasizing sensitivity to graph structural features (Goyal et al., 2019).
• GraphBench for Topology Inference: Task-driven framework for evaluating algorithms that infer graphs from raw feature data, with downstream protocols for clustering, semi-supervised classification, and signal denoising (Lassance et al., 2020).
• GraphBench for Graph System Evaluation: Blueprint for reproducible, hardware-aware comparison of graph processing systems, focusing on kernel workloads, preprocessing artifacts, and data statistical properties (Mehrotra et al., 2024).

2. Benchmark Design Principles

Major iterations of GraphBench converge on several rigorous design objectives:

• Standardization: All variants enforce strict protocols for train/validation/test splits relevant to the task. Examples include temporal splits for social forecasting, cross-size splits for combinatorial optimization, and scaffold/project/species splits for property prediction and generalization (Stoll et al., 4 Dec 2025, Hu et al., 2020).
• Domain and Scale Breadth: Datasets are curated to span multiple real-world domains (social, biological, technological, hardware, etc.) and scales (thousands to billions of nodes/edges), capturing the statistical idiosyncrasies pertinent to graph-structured phenomena (Stoll et al., 4 Dec 2025, Goyal et al., 2019).
• Evaluation Metrics: Tasks are paired with metrics purpose-built to probe performance and generalization. Common metrics include ROC-AUC, mean Average Precision (AP), accuracy, $R^2$, relative squared error (RSE), adjusted mutual information (AMI), normalized mutual information (NMI), closed gap (CG), and population diversity statistics (Benito et al., 27 May 2026, Stoll et al., 4 Dec 2025).

3. Methodologies and Experimental Protocols

GraphBench frameworks deploy reproducible, automated end-to-end pipelines that systematize experimental comparison:

• Core Pipeline Phases:
  • Data loading (often with PyTorch Geometric/DGL interfaces).
  • Algorithm/model evaluation under standardized splits and metrics.
  • Hyperparameter optimization (e.g., Bayesian optimization via SMAC3, grid search), sometimes with multi-fidelity scheduling (Stoll et al., 4 Dec 2025, Benito et al., 27 May 2026).
  • Artifact reporting, including solution quality, convergence curves, wall-clock runtime, and memory footprint.
• Baseline Algorithms: Suites benchmark both classic (e.g., MLP, GIN, GraphConv, Matrix Factorization, Node2Vec, spectral heuristics) and state-of-the-art models (Graph Transformers, Deep Autoencoders, GSSL, evolutionary computation, etc.), establishing reference bounds (Hu et al., 2020, Benito et al., 27 May 2026, Stoll et al., 4 Dec 2025).
• System Evaluation: For hardware benchmarking, protocols include comprehensive cache locality analysis, scaling curves, vertex orderings, and isolation effects to expose artifact-driven performance variability (Mehrotra et al., 2024).

4. Dataset Curation and Statistical Protocols

GraphBench suites emphasize explicit documentation and analysis of graph dataset properties crucial for fair benchmarking:

Domain	Example GraphBench Datasets	Structural/Challenge Aspect
Social	BlueSky, Twitter2010, LiveJournal	Scale, temporal splits, degree skew
Biological	Proteins (STRING), MolHIV	Heterogeneity, clustering, OOD
Hardware/CAD	Circuit AIGs, Analog circuits	Size OOD, input schemas, efficiency
Technology	SNAP, CommunityFitNet	Sparse tech networks, link prediction
Optimization	ER/BA/RB graphs for MIS/MC	Generative structure, size scaling

Detailed statistical controls include:

• Recording/controlling for degree distribution, density, clustering, isolations ($f$), and diameter.
• Using synthetic data only when structural realism and parameterization can be justified (e.g., Smooth Kronecker over RMAT) (Mehrotra et al., 2024).
• Publishing dataset versioning and hashes to ensure repeatability.

5. Task Coverage and Evaluation Regimes

GraphBench spans a wide taxonomic breadth of tasks:

• Prediction Tasks: Node/edge/graph classification and regression, property prediction, node clustering, algorithmic reasoning (topological sort, MST), generative design (circuit AIG synthesis), and runtime/solver performance forecasting (Stoll et al., 4 Dec 2025, Hu et al., 2020).
• Combinatorial Optimization: Maximum independent set and cut problems with penalty-based fitness for infeasible solutions, supporting direct comparison of evolutionary algorithms, ML, and exact solvers (Benito et al., 27 May 2026).
• Graph Inference and Embedding: Evaluation of graph construction from data via similarity, sparsity, or smoothness models, plus embedding quality assessed by downstream link prediction and clustering (Lassance et al., 2020, Goyal et al., 2019).

6. Key Empirical Findings and Baseline Performance

Cross-framework findings highlight both progress and persistent challenges:

• Superiority of Structure-Aware Algorithms: GNNs, edge-aware transformers, and deep structural embeddings (SDNE) typically outperform feature-only or shallow baselines in node and link tasks, although classic matrix factorization and evolutionary heuristics sometimes rival or surpass GNNs in certain OOD or unsupervised regimes (Hu et al., 2020, Benito et al., 27 May 2026, Goyal et al., 2019).
• OOD Generalization Gaps: Realistic splitting strategies consistently reveal significant degradation relative to random splits—commonly 10–30 points in accuracy or AUC—suggesting that graph ML models remain brittle when generalized beyond in-distribution data (Stoll et al., 4 Dec 2025, Hu et al., 2020).
• Combinatorial Optimization: Local-search–augmented evolutionary algorithms (Baldwinian, Lamarckian) dominate generic Darwinian EAs and deep learning baselines for both MIS and MC over diverse random/structural graph ensembles. Baldwinian EA frequently outperforms specialized heuristics (e.g., KaMIS) on large ER MIS instances (Benito et al., 27 May 2026).
• Self-Supervised Foundation Models: No existing graph self-supervised pre-training method achieves uniformly strong performance on all downstream tasks (classification, link prediction, clustering) under a single early-stopping criterion; batch efficiency and memory trade-offs highlight the need for more scalable GNN self-supervision (Xu et al., 2024).
• Graph Embedding: SDNE demonstrates superior macro-GFS-score overall, especially on sparse or high-diameter graphs (economic/technological domains); traditional heuristics (CN, AA) are competitive for global link prediction in social graphs (Goyal et al., 2019).

7. Reproducibility, Best Practices, and Limitations

Across the various domains, GraphBench implementations address reproducibility and artifact evaluation as first-class concerns:

• Public codebases ship with all loaders, splits, metrics, and evaluation protocols (Stoll et al., 4 Dec 2025, Xu et al., 2024). Artifact availability lags, with only ~22% of graph systems publications offering full reproducibility (Mehrotra et al., 2024).
• All preprocessing (vertex ordering, isolate removal), hyperparameter search, random seed usage, and computational resources must be fully documented.
• Limitations include incomplete coverage of dynamic graphs, text-attributed nodes, or mega-scale synthetic graphs, and persistent challenges in automating robust hyperparameter optimization and multi-task self-supervision (Xu et al., 2024, Goyal et al., 2019).

A plausible implication is that robust, broad-scope progress in graph machine learning and combinatorial reasoning will require community-wide adherence to GraphBench-like methodologies, with continuous benchmarking updates, transparent reporting, and principled coverage of emerging graph modalities.