HUG-Bench: Multi-Domain Benchmark Suite

• HUG-Bench is a set of structured benchmarks evaluating zero-shot robotic grasping, ML knowledge graphs IR, and human–robot hugging interactions.
• It employs detailed protocols, diverse datasets, and rigorous splits to ensure reproducibility in both simulation and real-world tests.
• The benchmarks drive advances in physical interaction, resource recommendation, task classification, and autonomous grasping performance.

HUG-Bench refers to three distinct, rigorously designed benchmarks in the domains of dexterous robotic grasping, information retrieval on open ML resources, and human–robot physical interaction. Each incarnation of HUG-Bench is characterized by comprehensive data collection, task definitions, and evaluation protocols tailored to its research community. The following sections clarify and systematize HUG-Bench in its major forms as documented in peer-reviewed literature.

1. Zero-Shot Dexterous Grasping Evaluation Suite

HUG-Bench, as introduced in "Human Universal Grasping" (Wu et al., 15 Jun 2026), is a simulated and real-world benchmarking framework for evaluating zero-shot dexterous grasping policies. It was constructed to assess the capability of models, such as the HUG flow-matching model, to generalize to entirely unseen everyday objects.

Dataset Composition

HUG-Bench comprises 90 objects spanning five geometric categories—cylindrical, spheroidal, prismatic, appendaged (objects with handles or protrusions), and amorphous (irregular)—and three size bins per category (small ≲ 6 cm, medium ≈ 6–15 cm, large up to ≈ 30 cm). For each of the 15 category–size bins, there are six objects: four reserved for simulation validation and two for real-world testing, yielding 30 objects for hardware evaluation. Mesh assets are obtained via a multi-view stereo pipeline ("aria2mesh" built on SAM3D), which reconstructs accurate, metric-scale 3D models based on egocentric RGB-D image capture and post-processes for watertightness, convex decomposition, and physics compatibility.

Simulation and Benchmark Protocol

Grasping is evaluated in MuJoCo using a fixed-shape MANO hand model, with each grasp comprising: (1) pre-grasp configuration (fingers open, wrist offset from target pose by Δ_pre ≃ [−0.03, 0, −0.03] m), (2) a linear approach and closure phase, and (3) a vertical lift along the camera z-axis (+0.5 m). Success is defined as the object no longer contacting the table after the lift. Evaluation is performed via 10 independent trials per object, for a total of 300 test trials.

Table: HUG-Bench Grasping Object Categories and Evaluation Splits

Category	Size Bin	Num. Objects (val)	Num. Objects (test)
Cylindrical	Small/Med/Lg	4	2
Spheroidal	Small/Med/Lg	4	2
Prismatic	Small/Med/Lg	4	2
Appendaged	Small/Med/Lg	4	2
Amorphous	Small/Med/Lg	4	2

Metrics

Two principal metrics are used:

• Success Rate (SR):

$$SR = \frac{1}{N_\text{objects} \times N_\text{trials}} \sum_{i=1}^{N_\text{objects}} \sum_{j=1}^{N_\text{trials}} 1[\text{success}_{i,j}]$$

• Fingertip Contact Error (FC): $$FC = \frac{1}{2} (|d_\text{thumb}| + \min_{f \in F} |d_f|)$$ where $d_i$ is the signed distance of fingertip $i$ to the object surface, $F$ is the set of non-thumb fingers.

Baseline Comparisons and Results

In real-world tabletop testing, HUG achieves a 66.7% SR on the 30-object set with Ability hand hardware, compared to 43.7% for Dex1B (sim-trained policy) and 32.7% for CAP (contact-anchored parallel-jaw policy), offering improvements of +23 and +34 percentage points, respectively. In in-the-wild settings (YOR robot + WUJI hand), HUG yields 62.0% SR. Simulation results at the best-validation checkpoint reach 73.0% SR (±2.6 SE), with a human grasp oracle at 94.0% and an average FC error of 14.6 mm.

Implementation

Evaluation is fully reproducible, with a supplied script automating MuJoCo rollouts, and key hyperparameters controlling pre-grasp offset, force close offset (+5°), lift height (0.5 m), and physical simulation parameters (2 ms timestep, palm compliance) (Wu et al., 15 Jun 2026).

2. Multi-task Information Retrieval Benchmark on ML Knowledge Graphs

In "Benchmarking Recommendation, Classification, and Tracing Based on Hugging Face Knowledge Graph" (Chen et al., 23 May 2025), HUG-Bench is a multi-task suite for IR on the large-scale HuggingKG, the first heterogeneous KG built from the Hugging Face community.

Construction of the Underlying Knowledge Graph

HuggingKG encodes 2.6M nodes and 6.2M edges, modeling entities such as Models (46%), Users (27.9%), Spaces, Datasets, and Tasks. Thirty semantic relations include user–model "Like", user "Publish", model–model "Finetune"/"Adapter"/"Merge"/"Quantize", and more. The graph is generated via API crawls, textual attribute extraction, and relation synthesis; it is verified for de-duplication and referential integrity.

HUG-Bench Tasks

HUG-Bench comprises three main IR tasks:

• Resource Recommendation: For each user, recommend new models/datasets/spaces based on a 5-core user–item "Like" subgraph (38,624 users, 25,080 items, 1,065,220 interactions), user–user social "Follow" graph, and multiple KG neighborhoods. CF, knowledge graph-based, and social baselines are evaluated. Key metrics: Recall@K and NDCG@K.
• Task Classification: Multi-label prediction of tasks served by each Model/Dataset (K=52 tasks), using graph structure, textual features, and various GNNs. Metric: micro-F1.
• Model Tracing: For a given checkpoint $t$ and relation $r$ (Finetune, Adapter, Merge, Quantize), predict the source $h$ (model tracing), posed as a link prediction problem. Tasks are evaluated via MRR and Hit@K.

Table: HUG-Bench (HuggingKG) IR Tasks and Evaluation Metrics

Task	Data	Metric(s)
Resource Rec	5-core Like graph, KG views	Recall@K, NDCG@K
Task Classification	Labeled Models/Datasets	Micro-F1
Model Tracing	Model evolution subgraph	MRR, Hit@K

Empirical Findings

• KG-based methods (e.g., KGCL) have the highest recommendation scores, but variance depends strongly on KG subgraph selection.
• High-quality text representations (finetuned BERT, BGE) are critical in classification, outperforming pure GNN depth.
• Translational KGE models (TransE) excel in model tracing owing to the predominantly one-to-one nature of HuggingKG's evolution edges.
• Data sparsity (e.g., 57.2% of models lack descriptions, social/follow edges are sparse) impairs some methods, highlighting the need for robust hybrid approaches.

3. Human-Robot Hugging Interaction Benchmark

As presented in (Bagewadi et al., 2019), a third form of HUG-Bench is the multimodal dataset and evaluation suite for human–robot hugging, developed to advance physical human–robot interaction research.

Dataset Characteristics

The benchmark comprises 353 hug episodes between 33 adult subjects and a Baxter robot, instrumented with Myo armbands, custom force-sensor shoes, and motion capture. Principal features include kinematics (joint angles), contact forces, full-body motion, and multimodal synchronization.

Core Benchmark Tasks

• Hug detection and segmentation: Binary classification of time steps into “in-hug” vs “out-hug”.
• Anticipatory motion prediction: Next 1 sec of robot joint trajectories from preceding human pose/IMU data.
• Intensity estimation: Classification into intensity tiers (none, light, firm) via FSR curves.

Standardized cross-validation (subject-wise 5-fold CV, Leave-One-Subject-Out), normalization, and signal preprocessing routines are provided. Baselines include threshold models, linear regression, and SVM/RF classifiers.

Significance

This benchmark supports reproducible evaluation of detection, anticipation, and safe force control in physical human–robot interaction, with clear metrics (precision, recall, F1, RMSE, DTW, MAE, Pearson $r$) and open-source tooling and splits (Bagewadi et al., 2019).

4. Standardization Practices and Reproducibility

All three HUG-Bench instances are characterized by:

• Rigid dataset splits ensuring train–test disjunction, temporal splits or object holds-outs.
• Precisely defined input/output formats for each task.
• Formal mathematical definitions of success criteria and metrics, ensuring comparability and statistical rigor.
• Explicit protocols, including open-source pseudocode or launch files, for full experimental reproducibility.
• Where relevant, physical assets (3D meshes, URDFs), feature matrices, and textual resources are released publicly.

5. Limitations and Future Directions

Limitations include the domain restriction of HUG-Bench (HuggingKG's ML focus), sparsity in certain graph modalities, and fully synthetic or automatically generated nature of some test collections, omitting manually annotated or richer semantic tasks. Proposed expansions involve connecting cross-platform resources (e.g., GitHub, Kaggle), improved methods for dealing with graph/data sparsity, and new QA/retrieval challenges using LLMs or curated annotation (Chen et al., 23 May 2025).

6. Comparative Insights

HUG-Bench, in its various forms, represents a trend toward highly structured, standardized, and multi-modal benchmarking efforts. Across domains, it enforces a zero-shot or holdout evaluation paradigm and combines diverse data types (semantic graphs, physics assets, sensor streams) with formal task definitions, promoting methodological transparency and robust cross-study comparison. Its design has surfaced methodological lessons: e.g., viability of simple translational KGE for model tracing, dependence of IR accuracy on high-quality embeddings, and the importance of data-rich, sensor-diverse datasets for physical human–robot interaction (Wu et al., 15 Jun 2026, Chen et al., 23 May 2025, Bagewadi et al., 2019).