1M-HUGS: Large-Scale Human Grasp Dataset

• 1M-HUGS dataset is a large-scale egocentric collection of over 1 million annotated human grasp frames captured with smart-glasses in diverse indoor settings.
• It employs detailed MANO hand pose parameters and synchronized RGB-D data, enabling robust mapping of natural human grasps to arbitrary robotic hands.
• The dataset underpins the HUG model, achieving 66.7% tabletop success on HUG-Bench and showcasing strong zero-shot sim-to-real transfer capabilities.

The 1M-HUGS dataset is a large-scale egocentric collection of human grasp demonstrations intended to advance multi-fingered robotic grasping by leveraging the natural dexterity and generality of human hand movements. Collected using smart glasses, 1M-HUGS spans roughly 1 million frames (27.8 hours), encompassing 6,707 object instances in diverse real-world indoor environments across 41 buildings. The dataset, introduced in the context of the HUG ("Human Universal Grasping") project, serves as the foundational training data for flow-matching models that predict natural human grasps from RGB-D images and can be retargeted to arbitrary robot hands for zero-shot grasping in novel scenes (Wu et al., 15 Jun 2026).

1. Data Acquisition and Scope

The 1M-HUGS dataset is acquired through smart-glasses-mounted RGB-D sensors, capturing egocentric views of human grasp interactions. Each recorded frame contains the human hand in the process of grasping an object, with rich annotation of hand pose. The acquisition protocol yields:

• 1,000,000 frames (27.8 hours) of video data focused on genuine human-object interaction.
• 6,707 distinct object instances, ensuring substantial diversity.
• Environments span 41 unique buildings, increasing exposure to real-world variability.
• All grasps are recorded in uncontrolled settings, capturing natural variation in lighting, scene composition, and hand approach.

A plausible implication is that this scale and diversity enable generalizable models that better predict grasping strategies representative of unstructured environments.

2. Grasp Representation and Annotation

Within 1M-HUGS, each grasp is parameterized by:

• Wrist translation (3 DoF)
• Wrist rotation (6D representation, for minimal intrinsic ambiguity)
• Full MANO hand pose (an articulated MANO model: 15 ball joints, six DoF wrist)

This parameterization enables unified mapping to both human and robotic hands, supporting downstream retargeting and simulation. The MANO hand model is chosen due to its widespread adoption for hand pose estimation and retargeting fidelity.

All grasp annotations are extracted from stereo RGB-D imagery, with fused depth and color data supporting accurate 3D localization and back-projection. Camera intrinsics and extrinsics are provided per scene to support consistent frame registration.

3. Dataset Structure and Distribution

The 1M-HUGS dataset constitutes the training backbone for the HUG grasping model, whereas evaluation is standardized on the disjoint HUG-Bench suite. The dataset includes:

• Per-frame egocentric RGB and depth images.
• Per-frame MANO pose parameters and associated object ID.
• Camera calibration files (intrinsics, extrinsics) and per-object metadata, including category and mass/volume where available.

While the specific directory arrangement for 1M-HUGS is not detailed in the summary, HUG-Bench—the evaluation benchmark—adheres to a strictly hierarchical asset layout, with assets, scenes, cameras, and scripts directories to facilitate efficient usage (Wu et al., 15 Jun 2026).

4. Usage in Grasp Learning and Retargeting

The primary application of 1M-HUGS is to train grasp prediction models that map a single RGB-D observation and a user-specified object (by segmentation click) to a plausible grasp parameterized in the MANO space. The HUG model architecture—trained solely on 1M-HUGS—incorporates:

• RGB ViT (Vision Transformer) + DINOv2 for feature extraction
• PointNeXt for depth point-cloud encoding
• Flow-matching diffusion to predict $[t, R_{6d}, \theta_{6d}]$

Resulting grasps are directly retargetable to various robot hands via the MANO-to-robot kinematic mapping, supporting zero-shot deployment in novel scenes without additional robot or synthetic data.

5. Role in Benchmarking and Evaluation (HUG-Bench)

HUG-Bench is the companion evaluation suite to 1M-HUGS, consisting of 90 category- and size-stratified objects with metric 3D meshes—hence, training occurs exclusively on 1M-HUGS, and all protocol-driven evaluation uses HUG-Bench. The benchmark specifies simulation and real-world protocols, with the following features:

• No overlap between 1M-HUGS training objects and evaluation objects—test generalization is strictly enforced.
• Evaluation metrics include Success Rate (SR), Fingertip Contact Error (FC), intersection percentage, and physical penetration diagnostics:

$$\mathrm{SR} = \frac{1}{N}\sum_{i=1}^{N}1(\text{object not touching table after lift in trial }i)$$

$$\mathrm{FC} = \frac{1}{2}\left(|d_{\mathrm{thumb}}| + \min_{f\in\mathcal{F}}|d_f|\right)$$

where $d_{i}$ denotes fingertip-to-surface distance.

As reported, the HUG model, trained on 1M-HUGS, achieves 66.7% tabletop success rate on the 30-object HUG-Bench test set, outperforming Dex1B and CAP baselines by +23 and +34 points, respectively. In-the-wild success rate reaches 62.0%, demonstrating substantial zero-shot sim-to-real and device transfer (Wu et al., 15 Jun 2026).

6. Extensibility and Integration

HUG models trained on 1M-HUGS can be integrated into new robotics pipelines by retargeting to alternative end effector kinematics via the provided retargeting function interface. HUG-Bench enables object and camera extension without retraining, as camera intrinsics are used only for 3D back-projection. Adding objects to the evaluation pipeline mirrors the standard asset and metadata conventions.

This suggests that 1M-HUGS, placed alongside flexible evaluation protocols and strong zero-shot performance, provides a scalable foundation for research in dexterous, human-derived grasp planning in arbitrary environments.