1M-HUGs: Egocentric Grasping Dataset

• 1M-HUGs is a large-scale egocentric dataset consisting of ~1M human grasp demonstrations captured using wearable smart glasses.
• It provides synchronized RGB, grayscale, depth, and 6-DoF hand pose data across diverse indoor settings and object types.
• The dataset underpins state-of-the-art zero-shot grasp generation and evaluation, driving progress in imitation learning and robotic manipulation.

1M-HUGs is a large-scale, egocentric dataset of human grasp demonstrations designed to support research in dexterous robotic grasping and imitation learning. It comprises approximately one million frames capturing natural human grasps across a wide variety of objects and indoor settings, enabling data-driven modeling of human grasp distributions in metric 3D space. The dataset introduces a highly structured annotation scheme robust enough to inform zero-shot grasp generation and evaluation in both simulated and real-world robotic platforms (Wu et al., 15 Jun 2026).

1. Data Acquisition Setup

1M-HUGs was collected using wearable Meta Project Aria Gen 2 smart glasses, which provide synchronized, calibrated RGB (1920×1920) and stereo grayscale (1920×1920) video streams at ≈30 Hz, along with 6-DoF camera poses and a 21-landmark hand skeleton in the device reference frame. Depth estimation for each frame was performed via stereo matching (S²M²), delivering per-pixel metric depth and confidence.

Recordings were acquired in naturalistic conditions across 41 buildings, spanning diverse environments such as apartments, offices, kitchens, and bedrooms. Each of the 6,707 recordings follows a protocol wherein the wearer first surveys a static object for 15–30 seconds (without the hand in view), then reaches in with their right hand and grasps the object. Wearers exercised unrestricted choice of objects, and no laboratory constraints or teleoperation systems were imposed.

A distinctive feature of the capture method is the exploitation of SLAM-based temporal pose propagation: each “grasp event” yields not only the grasp instance itself but, via backward propagation of the final wrist+hand pose through preceding frames, produces hundreds of “object-only image, grasp” pairs from variable perspectives at no additional cost. Participant demographics were not catalogued; instead, post-hoc normalization addresses hand-size variation.

2. Dataset Statistics and Ontology

After filtering, 1M-HUGs contains approximately one million RGB frames and one million corresponding grayscale frames, totaling 27.8 hours of video. The dataset encompasses 6,707 recordings corresponding to approximately 1,500 unique object instances.

Objects are sorted into five geometric categories:

• Cylindrical
• Spheroidal
• Prismatic
• Appendaged (e.g., handles, levers)
• Amorphous

Additionally, for downstream evaluation, objects are binned into small, medium, and large sizes. The object set includes everyday household artifacts with diversity in shape, size, and material, covering plastic, wood, metal, fabric, deformable, and articulated items. The acquisition protocol ensures a long-tailed object distribution, reflecting real-world statistics. The dataset itself serves as a training corpus; separate validation and test splits are defined in a distinct benchmark (HUG-Bench).

3. Modalities, Annotation, and File Structure

Raw Modalities

Each frame in 1M-HUGs provides:

• 224×224 RGB crop (overlapping stereo FoV)
• 224×224 grayscale crop (left stereo)
• 224×224 metric depth map (meters, with per-pixel confidence)
• Per-pixel object mask
• Cropped camera intrinsics $K$ (focal length, principal point)

Grasp Annotation Format

Each datum is annotated with a 99-dimensional grasp vector:

$$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$

where:

• $t \in \mathbb{R}^3$: wrist translation in the camera (OpenCV) frame, meters
• $R_{6d}$: wrist orientation in 6-D continuous representation (Zhou et al., 2019)
• $\theta_{6d}$: MANO hand pose (15 joints × 6D each)

Underlying this, a full 45-DoF MANO hand model is fit to the 21 sparse landmarks via anatomically-constrained optimization; $\beta$ (hand shape) is fixed post-optimization and only joint pose $\theta$ is retained.

All position and depth quantities are expressed in the camera’s SLAM frame and use metric units.

Directory Structure

Data are organized as follows: $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$8

4. Access, Licensing, and API

The complete dataset, including images, depth maps, masks, grasp parameters, and camera intrinsics, is publicly released at https://grasping.io. Off-the-shelf Python APIs and loader scripts are included to streamline access and integration. The licensing model is research-use only (exact terms as specified on the project website); no explicit commercial restriction is listed, but users are directed to consult the licensing documentation for legal specifics (Wu et al., 15 Jun 2026).

5. Downstream Usage: HUG Model and HUG-Bench

Model Training

1M-HUGs underpins the training of the HUG grasp model, which leverages a point-conditioned flow-matching transformer to map $$(\text{RGB-D patch},\ 3\text{D click point}) \rightarrow \mathbf{x}$$. The model is trained with a composite loss:

$$L = \lambda_v \|f_\phi(x_t,t,s) - v_t\|^2 + \lambda_{3D}(1 - t) \cdot \|\text{landmarks}(\text{MANO}(x_0)) - \text{landmarks}_{gt}\|_1$$

where $L_v$ is the flow velocity MSE, $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$0 is $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$1 loss on reprojected 3D hand landmarks, and $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$2 is the flow timestep.

Benchmark: HUG-Bench

Evaluation uses HUG-Bench, a simulated and real-world benchmark of 90 previously unseen objects across five geometric categories and three size bins. Objects are defined as metric-scale watertight meshes (via multi-view SAM3D and manual alignment). Metrics include:

• Success Rate (SR %): % of trials where the object is lifted off the table in an open-loop pre-grasp → grasp → lift sequence
• Fingertip contact error: $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$3, with $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$4 as the signed distance from fingertip $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$5 to object surface

Real robot evaluation employs Ability hand + ZED camera (tabletop) and YOR mobile manipulator + WUJI hand + Aria (in-the-wild).

Empirical results demonstrate state-of-the-art performance: on the 30-object tabletop test set, HUG attains 66.7% SR (vs. Dex1B’s 43.7% and CAP’s 32.7%). In an in-the-wild home setting, HUG achieves 62.0% SR without per-robot or per-camera fine-tuning. The framework approaches the “human-grasp oracle” in simulation, with HUG ≈20% SR below the estimated human upper bound of ≈94%.

6. Limitations and Suggested Extensions

Several limitations are acknowledged:

• Exclusively right-handed, single-hand grasps; no data on left or bimanual grasps
• MANO hand shape $$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$6 is held fixed; individual hand size/shape variation is not modeled
• Data collection and execution are open-loop (no force feedback or closed-loop vision), resulting in grip failures during slippage or table contact
• Sparse representation for small/large objects (beyond 0.3 m crop radius) and distant objects
• Hand-tracking under occlusions introduces annotation noise

Future work is proposed in the following directions:

• Inclusion of left-hand and bimanual grasp events for richer manipulation contexts
• Learning per-user or canonical hand shapes ($$\mathbf{x} = [t_x, t_y, t_z \parallel R^\text{wrist}_{6d}, \theta^\text{finger}_{6d}] \in \mathbb{R}^{99}$$7 calibration)
• Addition of force/torque and tactile sensing for closed-loop grasping
• Generation of multiple grasp candidates for fallback in error-prone scenarios
• Broadening capture settings (e.g., outdoor, in-use tools, moving objects)
• Scaling dataset beyond one million frames, as scaling experiments indicate the model remains data-bound at current size (Wu et al., 15 Jun 2026).

7. Significance and Research Impact

1M-HUGs constitutes the first large-scale, in-the-wild egocentric dataset providing metric depth and full 6-DoF hand pose annotations for natural human grasps. Its structure and compositional diversity facilitate modeling of human grasp distributions and transfer to robotic domains via retargeting. The closed-loop benchmarks and strong zero-shot grasping results highlight its value as a foundation for progress in imitation learning and generalist robot manipulation (Wu et al., 15 Jun 2026).