HOT3D: Egocentric 3D Hand & Object Tracking

• HOT3D is a large-scale egocentric dataset that provides synchronized RGB, monochrome, eye-gaze, and point cloud data for precise hand-object tracking in settings like kitchens and offices.
• It integrates high-resolution imagery and calibrated motion-capture ground truth to enable rigorous evaluation of 3D hand tracking and 6DoF object pose estimation, yielding significant accuracy improvements.
• The dataset supports diverse benchmarks including hand tracking, object pose estimation, and trajectory generation, making it essential for AR/VR, robotics, and world modeling research.

HOT3D is a large-scale, publicly released dataset for egocentric 3D hand and object tracking, acquired with head-mounted cameras in controlled multi-view settings. HOT3D provides synchronized RGB and monochrome images, eye gaze, scene point clouds, and comprehensive motion-capture ground truth for hands, objects, and cameras. The dataset emphasizes the benchmarking of hand–object interaction in challenging, real-world tasks typical of kitchen, office, and living room environments, and serves as a central testbed for computer vision, robotics, and augmented/virtual reality (AR/VR) research involving fine-grained hand manipulation and pose estimation (Banerjee et al., 2024, Banerjee et al., 2024).

1. Dataset Structure and Recording Modalities

HOT3D contains 833 minutes of egocentric video, comprising 1.5 million synchronized multi-view frames (over 3.7 million images at 30 fps), captured from 19 participants performing diverse interactions with 33 rigid household and office objects. The data is collected with two Meta headsets: Project Aria (AI research AR glasses) and Quest 3 (consumer VR headset). Project Aria’s sensor suite includes a 1408×1408 RGB camera, dual 640×480 monochrome fisheye cameras, dual eye-tracking cameras, and IMUs; Quest 3 provides front-facing 1280×1024 global-shutter monochrome cameras.

All video streams are hardware-synchronized and calibrated. For every frame, HOT3D provides per-camera intrinsics and extrinsics aligned to a common, metric gravity-aligned world frame. Additionally, motion-capture data (OptiTrack system, with small 3 mm markers) delivers sub-millimeter 3D ground-truth for hand and object poses (Banerjee et al., 2024, Banerjee et al., 2024).

<table> <tr> <th>Device</th> <th>Cameras (#, Type, Resolution)</th> <th>Other Modalities</th> </tr> <tr> <td>Project Aria</td> <td\>1 RGB (1408×1408), 2 mono (640×480), 2 eye (320×240)</td> <td>IMUs, SLAM point clouds, calibrated gaze rays</td> </tr> <tr> <td>Quest 3</td> <td\>2 mono (1280×1024)</td> <td>N/A (no IMU/audio)</td> </tr> </table>

2. Ground Truth Annotation and Formats

HOT3D provides precise 3D annotations for hands, objects, and cameras using motion-capture with infrared markers. Hand meshes are parametrized both by UmeTrack (user-specific high-accuracy hand models) and the 45-parameter MANO model (θ∈ℝ³⁷ for pose, β∈ℝ¹⁰ for shape). Object meshes (33 total, watertight) feature PBR (Physically-Based Rendering) materials and are aligned with their tracked 6 DoF poses (rotation, translation).

Annotations are given in standardized formats, e.g., for MANO: $$V_{\text{hand}}(\theta, \beta) = \bar{V} + B_{S}\beta + B_{P}\theta$$, where $\bar{V}$ is the mean hand shape and $B_S$, $B_P$ are blend shape matrices (Banerjee et al., 2024, Banerjee et al., 2024).

For multi-view calibration, all camera poses are mapped to the OptiTrack-based world frame; the dataset includes per-frame SLAM point clouds (from Project Aria), and synchronized eye-gaze data for gaze-task benchmarks.

3. Evaluation Benchmarks and Task Protocols

HOT3D supports benchmarks for 3D hand tracking, 6DoF object pose estimation, and 3D lifting of in-hand objects. Task-specific metrics and representative results are as follows:

• 3D Hand Tracking: Evaluated via Mean Per-Joint Position Error (MPJPE),

$$\mathrm{MPJPE} = \frac{1}{N} \sum_{i=1}^N \|\hat{J}_i - J_i\|_2,$$

where $N$ is the number of joints. Multi-view input yields significant accuracy gains: e.g., 2-view evaluation improves MPJPE by ∼41% relative to single-view (from 15.4 mm to 10.9 mm on HOT3D test) (Banerjee et al., 2024).

• 6DoF Object Pose Estimation: Uses ADD and ADI metrics (as in BOP challenge),

$$\mathrm{ADD} = \frac{1}{|M|} \sum_{x\in M} \| R x + t - (\hat{R} x + \hat{t}) \|_2.$$

Multi-view yields 8–12 percentage points absolute recall gain at (5 cm, 5°) thresholds (e.g., Aria 1-view: 25.2%; 3-view: 33.8%) (Banerjee et al., 2024).

• 3D Lifting of In-Hand Objects: Given per-view 2D segmentation, the task estimates the 3D object centroid. With three synchronized views, the StereoMatch baseline achieves 64.4% recall at 5 cm error, vastly outperforming single-view and hand-proxy methods (Banerjee et al., 2024).
• Gaze Estimation: Used in follow-up work, such as HOIGaze (Hu et al., 28 Apr 2025), for mean angular error benchmarks exploiting eye–head–hand coordination.
• Trajectory Generation and Object Manipulation: HOT3D supports evaluation of models generating 6DoF trajectories from action descriptions, as in manipulation forecasting and language–vision tasks (Yoshida et al., 4 Jun 2025).

4. Benchmark Subsets and Derived Annotations

Several works define canonical HOT3D test splits:

• Curated Clips: 4,117 five-second segments (2,969 train / 1,148 test) selected for guaranteed, fully annotated hand–object interactions (Banerjee et al., 2024).
• HOT3D-HIT: Used for evaluating reconstruction under Hand Interaction Timelines (HITs): 1,239 stable grasp clips, 113 full timelines, each annotated to partition static, stable, and unstable hand–object phases completely automatically from mesh correspondences (Zhu et al., 8 Dec 2025).

Subsets are further utilized for world-model evaluation (EgoHOI: 1,516 clips of 150 frames @16 Hz; comprehensive 3D hand, object, and camera annotations per frame) (Li et al., 13 Mar 2026). Test splits are publicly held-out for challenge evaluation (e.g., ECCV BOP, Hand Tracking Challenge).

5. Downstream Impact and Related Work

HOT3D directly supports advancements in learning-based multi-view hand and object pose estimation, AR/VR perception, robotic skill transfer, and egocentric world modeling. Its scale, annotation fidelity, and variety (4 scenarios; 33 objects; per-object onboarding for reconstruction; eye gaze and SLAM point clouds) have led to active usage in:

• Gaze estimation research, surpassing previous datasets through dense multi-modal cues (Hu et al., 28 Apr 2025).
• 6DoF trajectory generation conditioned on language or vision, providing a high-precision benchmark for testbed comparison (Yoshida et al., 4 Jun 2025).
• Physically-structured world models, in which HOT3D enables training and evaluation of contact-consistent prediction and simulation systems under real egocentric control (Li et al., 13 Mar 2026).

Comparison to prior datasets such as HO-3D reveals that HOT3D features greater scale, diversity, and uniquely synchronized multi-modal streams—including eye gaze and marker-based accurate ground truth—enabling more reliable hand–object–scene interpretation and learning (Hampali et al., 2021).

6. Data Accessibility, Organization, and Limitations

HOT3D is available for non-commercial research from https://facebookresearch.github.io/hot3d/ under a research-only license. The dataset layout includes:

• /objects/: 33 watertight PBR-scanned meshes with textures.
• /subjects/: 19 unique hand models (UmeTrack, MANO).
• /recordings/: Synchronized, multi-modal egocentric videos (Aria, Quest 3); per-frame annotations and metadata.
• /clips/: Benchmarks and challenge splits.
• /onboarding/: Model-free object tracking and 3D reconstruction support.

Key limitations include laboratory environment constraints, limited number of deformable objects (only rigid), absence of per-pixel depth in raw captures (though pseudo-depth can be synthesized), and limited number of subjects compared to very large-scale in-the-wild datasets (Yoshida et al., 4 Jun 2025, Zhu et al., 8 Dec 2025).

7. Significance and Ongoing Research Directions

HOT3D provides an unprecedented combination of scale, annotation accuracy, synchronized multi-modal egocentric data, and comprehensive ground truth. It serves as a standard for developing and benchmarking 3D joint pose tracking, contact reasoning, hand–object interaction understanding, and egocentric skill transfer. The public release, challenge organization, and adoption in a range of world modeling, trajectory generation, and gaze estimation research demonstrate its central role in the field. Ongoing studies address further expansion of object categories, increased diversity in hand appearances, extension to unstructured environments, and integration of physical property annotation for broader learning of hand–object–scene interaction dynamics (Banerjee et al., 2024, Banerjee et al., 2024, Hu et al., 28 Apr 2025, Yoshida et al., 4 Jun 2025, Li et al., 13 Mar 2026, Zhu et al., 8 Dec 2025).