DHG-Bench: Benchmark for Hypergraph Learning

• DHG-Bench is a comprehensive benchmark for deep hypergraph learning, evaluating hypergraph neural networks across diverse tasks.
• It supports 16 state-of-the-art HNN models and 20 real-world datasets, covering node, hyperedge, and graph-level prediction scenarios.
• The platform standardizes preprocessing, experimental protocols, and metrics to ensure reproducible, efficient, robust, and fair comparisons.

DHG-Bench is a comprehensive benchmark designed to address the lack of standardized evaluation in deep hypergraph learning (DHGL), where higher-order relations are modeled using hypergraph neural networks (HNNs). Unlike traditional graph learning benchmarks limited to pairwise GNNs, DHG-Bench systematically assesses HNNs across a broad suite of tasks, datasets, and evaluation axes, focusing on effectiveness, efficiency, robustness, and fairness. The benchmark standardizes preprocessing, experimental protocol, and metrics and supports 16 state-of-the-art HNNs and 20 real-world datasets, constituting the first large-scale, reproducible comparison platform for DHGL (Li et al., 17 Aug 2025).

1. Motivation and Context

Relational learning in complex systems frequently involves interactions that are not restricted to pairs but include higher-order associations—such as co-authorship, biological modules, or multi-member social groups—that are more naturally expressed as hyperedges than as simple edges. Hypergraph neural networks (HNNs) are the prevailing DHGL architectures addressing these generalizations of standard GNNs, operating on incidence matrices or higher-order tensor representations.

Prior to DHG-Bench, the field lacked coverage in several key areas:

• Benchmarks for HNN methods underrepresented the diversity of available datasets, particularly ignoring graph- and hyperedge-level tasks, heterophilic settings, and scalability/fairness aspects.
• Direct comparisons between spectral, spatial, and tensor-based HNNs were rarely possible due to inconsistent preprocessing, hyperparameter tuning, and data splits.
• Most published work narrowly focused on effectivity (accuracy), sidelining aspects like runtime, robustness to noise, and group fairness. DHG-Bench was introduced to resolve these limitations, enabling precise and fair benchmarking of DHGL approaches and fostering broader research on generalizable, efficient, and equitable hypergraph methods (Li et al., 17 Aug 2025).

2. Dataset Suite and Preprocessing

DHG-Bench integrates 20 datasets, categorized by three task domains: node-level classification, hyperedge-level prediction, and hypergraph-level classification. Each domain incorporates both homophilic and heterophilic interaction regimes, as well as fairness-sensitive scenarios. All datasets are processed under unified, standardized protocols, with features normalized and splits fixed to reproducible proportions.

Node-Level Classification:

• Homophilic: Cora, Pubmed, Cora-CA, DBLP-CA, Walmart, Trivago
• Heterophilic: Actor, Amazon-ratings, Twitch-gamers, Pokec, Yelp
• Fairness-sensitive: German, Bail, Credit Defaulter

Hyperedge-Level Prediction:

• Cora, Pubmed, Cora-CA, DBLP-CA (homophilic), Actor, Pokec (heterophilic)
• Negative hyperedge sampling combines size-based (SNS), motif-based (MNS), and clique-based (CNS) negatives to ensure diversity.

Hypergraph-Level Classification:

• Synthetic: RHG-10, RHG-3
• Real-world: IMDB-Dir-Form, IMDB-Dir-Genre, Steam-Player, Twitter-Friend

A hypergraph is defined by its incidence matrix $\mathbf{H}\in\{0,1\}^{n\times m}$, with the vertex-degree and edge-degree matrices constructed as $D_v = \mathrm{diag}(d_i)$, $d_i=\sum_e H_{i,e}$ and $D_e = \mathrm{diag}(\delta_e)$, $\delta_e=\sum_i H_{i,e}$. All datasets adopt this representation, with edge weights $w_e=1$. Splits are: node classification (50/25/25), hyperedge prediction (60/20/20), hypergraph classification (80/10/10).

3. Models and Algorithmic Formalisms

DHG-Bench implements 16 HNN architectures under APIs guaranteeing comparability. These encompass three principal modeling frameworks:

Spectral-based (10): e.g., HGNN, HyperGCN, HCHA, SheafHyperGNN, TF-HNN.

• Hypergraph Laplacian:

$L = I - D_v^{-1/2} H D_e^{-1} H^\top D_v^{-1/2}$

• HGNN updates:

$$X^{(\ell+1)} = \sigma\left(D_v^{-1/2} H D_e^{-1} H^\top D_v^{-1/2} X^{(\ell)} W^{(\ell)} \right)$$

Spatial-based (4): e.g., HNHN, UniGNN (UniGCNII), AllSetTransformer, ED-HNN.

• UniGNN: two-stage aggregation for hyperedge and vertex embeddings, using permutation-invariant functions over sets.

Tensor-based (2): EHNN, T-HyperGNN.

• Hypergraph as sparse tensor $$\mathcal{T} \in \{0,1\}^{n \times n \times \cdots \times n}$$ with $M$-order.
• Layers are equivariant linear maps, maintaining symmetry under vertex permutations.

Each HNN method is benchmarked using consistent code, initialization, and splitting, allowing matched hyperparameter search grids for learning rate, epochs, size, and regularization.

4. Evaluation Protocol and Metrics

All evaluations are conducted using PyTorch 2.2.2 (CUDA 12.1), and benchmark results are aggregated over five random seeds. Evaluation proceeds along four axes:

1. Effectiveness: Accuracy and macro-F1 for node/hypergraph-level classification; AUROC and AP for hyperedge prediction.
2. Efficiency: Training wall-clock time (to best validation score), peak GPU memory.
3. Robustness: Systematic noise injections—
   • Structure: Random deletion/addition of entries in the incidence matrix (up to 90%),
   • Feature: Gaussian feature noise, random feature masking,
   • Supervision: Training label flips and label sparsity.
4. Fairness: Group fairness using Demographic Parity

$D_v = \mathrm{diag}(d_i)$0

and Equalized Odds,

$D_v = \mathrm{diag}(d_i)$1

Hyperparameter grids are matched across models: learning rate ∈ $D_v = \mathrm{diag}(d_i)$2; epochs ∈ [100, 1000]; hidden size ∈ $D_v = \mathrm{diag}(d_i)$3.

5. Empirical Findings

5.1 Effectiveness

TF-HNN achieves top or runner-up node classification accuracy on 8 of 11 benchmarks (e.g., 79.47% Cora, 87.90% Pubmed, 90.79% Trivago). On homophilic graphs, most HNNs attain high accuracy (≥74%), while heterophilic graphs produce notably lower performance (<36% on Yelp/Amazon-ratings). Trivago and Yelp induce OOM or degraded performance in many methods. In hyperedge prediction, spectral HNNs (HGNN, HyperGCN, EHNN) outperform recent approaches (e.g., DPHGNN, TF-HNN) by up to 10 AUROC points. Real-world graph-level datasets (IMDB, Steam, Twitter) witness large accuracy variance, with best methods reaching only ∼64% (IMDB-Form, macro-F1 <52% Twitter), and no HNN dominates universally.

5.2 Efficiency and Scalability

Tensor-based HNNs (EHNN, T-HyperGNN) are computationally prohibitive, exhibiting 10–400× slower runtimes than HGNN and frequent OOM failures on large graphs. Spatial and spectral approaches scale to medium graphs but carry steep resource costs on graphs like Yelp (ED-HNN/EHNN: 9×/23× slower than HGNN). TF-HNN demonstrates the best trade-off, combining high accuracy, <1.6 GB memory, and <30 s training time on the largest benchmarks.

5.3 Robustness

Spectral HNNs (HGNN, SheafHyperGNN, DPHGNN) experience >30% accuracy losses under 90% hyperedge deletions, while spatial/tensor methods sustain <10% drops. Feature masking (90%) causes 20–30% accuracy loss, whereas analogous Gaussian noise has milder effects (<5% loss). Label noise (15%) results in over 15% accuracy decrease in all HNNs.

5.4 Fairness

Across fairness-sensitive datasets, HNNs amplify demographic bias relative to non-message-passing MLPs. On Credit, MLP achieves $D_v = \mathrm{diag}(d_i)$4, $D_v = \mathrm{diag}(d_i)$5; the fairest HNN (UniGNN) yields $D_v = \mathrm{diag}(d_i)$6, $D_v = \mathrm{diag}(d_i)$7. No HNN attains uniformly low group fairness gap across all sensitive benchmarks; observed bias patterns are dataset- and metric-dependent.

6. Recommendations and Research Directions

DHG-Bench identifies several key challenges for DHGL:

• Development of unified, task-agnostic HNN architectures that generalize across all task types (node, edge, graph), homophily regimes, and class distributions—current models are specialized and brittle.
• Advancement in scalability and efficiency, with a focus on training-free aggregation, higher-order sampling, and memory-efficient sparse tensor computation for real-world, large-scale hypergraphs.
• Robustness-aware design to mitigate accuracy loss from hyperedge structure perturbations and label noise, including adversarial training and robust diffusion kernels.
• Design of fairness-conscious HNN architectures via counterfactual reasoning, fairness-regularized objectives, and attribute-disentangled message passing.

A plausible implication is that fair and robust DHGL progress will require novel algorithmic primitives, not just incremental variants of GNN/HNN message passing (Li et al., 17 Aug 2025).

7. Open-Source Library and Reproducibility

All code for DHG-Bench is available under the MIT license (https://github.com/Coco-Hut/DHG-Bench), providing a unified library for dataset preprocessing, model training/evaluation, and automated hyperparameter searches. Standardized APIs facilitate integration of new HNN variants and datasets, supporting sustained reproducibility and extension by the research community (Li et al., 17 Aug 2025).