HandX: Bimanual Hand Interaction Dataset

• HandX is a comprehensive, high-fidelity bimanual hand motion dataset that integrates precise motion capture, detailed textual annotations, and standardized evaluation protocols.
• It employs advanced kinematic extraction and hierarchical LLM-based captioning to generate multi-level, semantically rich descriptions of hand interactions.
• The dataset offers over 54 hours of motion data and supports rigorous benchmarking for diffusion and autoregressive generative models in hand motion synthesis.

HandX is a large-scale, high-fidelity dataset designed to address the deficiency of realistic bimanual hand motion and interaction data in human motion synthesis research. It provides unified resources spanning motion capture, automated annotation, and evaluation standards, with a focus on dexterous behavior, nuanced finger articulation, contact timing, and inter-hand coordination in both object-centric and collaborative manipulations. HandX consolidates multiple public and newly captured datasets, offers scalable, semantically rich textual annotations, and presents standardized protocols for benchmarking generative models under diverse conditioning modes (Zhang et al., 30 Mar 2026).

1. Dataset Structure and Composition

HandX comprises 54.2 hours of high-quality motion data (approx. 5.9 million frames at 30 FPS), representing 485,700 text-motion pairs and about 98,000 unique 60-frame motion clips. Newly captured data involves 12 professional actors (balanced gender, ages 22–35) using a 36-camera OptiTrack optical system with 25 reflective markers per hand and calibrated to 0.5 mm spatial accuracy, downsampled from 120 Hz to 30 FPS. Additional sources include GigaHands, HOT3D, ARCTIC, H2O, and HoloAssist. All data adopts a unified 21-joint skeleton format for compatibility.

The dataset is exclusively bimanual, treating unimanual interactions as bimanual cases with one hand static. It features a broad variety of object categories (e.g., household objects, tools, sports equipment, utensils, everyday items), with precise breakdowns listed in the accompanying data catalog (e.g., 25,000 “cup”, 18,000 “door handle” clips). Collaborative hand–hand events (e.g., clapping, finger gestures) constitute about 12% of all clips. Official splits are stratified by object type and contact ratio: 80% train, 10% validation, 10% test. Scaling subsets (5%, 20%, 100% of train) are standardized for modeling studies.

2. Annotation Pipeline and Semantics

HandX employs a decoupled two-stage annotation process:

1. Kinematic feature extraction (rule-based):
   • Six descriptors are computed per frame: Finger Flexing (signed joint angle), Finger Spacing (adjacent-finger angle), Finger–finger Distance, Palm–palm Relation (cloud mean), Finger–palm Distance, and Wrist Trajectory.
   • Events are segmented by threshold crossings or temporal continuity (≥1 s), and encoded as JSON with event type, frame range, values, and state.
2. Hierarchical LLM-based captioning:
   • The extracted JSON events feed into a fixed-prompt T5-XXL model (OpenAI API, temperature 0.7).
   • LLM outputs hierarchical descriptions at five detail levels (from one-sentence summary to ≈60-token fine-grained narrative).
   • Caption structure explicitly distinguishes left (T_L), right (T_R), and inter-hand (T_I) actions, with strong temporal alignment and explicit reference to contact and separation events.

Example Level 3 annotation:

{ "left_summary": "The left thumb closes onto the index finger, holds for 0.8 s, then releases.", "right_summary": "...", "interaction": "...finger-finger contact then separation at frame 45." }

3. Motion Features and Kinematics

All data are retargeted to a consistent 21-joint skeleton compatible with MANO. Motion features include per-finger MCP–PIP–DIP joint angles, a 1-DoF scalar for in-plane flexion, direct measurements of fingertip-to-fingertip and palm-center–to–palm-center inter-hand distances, and explicit contact events at a 2 cm distance threshold for finger–finger and finger–palm interactions. Canonical coordinates are defined at frame 0: X (wrist–wrist), Y (wrists→fingertips), and Z (upward). Per-subject bone-lengths and wrist-joint locations are refined through constrained least-squares optimization.

Statistical properties:

• Mean contact ratio: 0.24 (σ = 0.12).
• Contact duration: median 0.45 s; 10–90% interval [0.12, 1.28] s.
• Finger flexion: MCP ∼ Uniform(5°, 100°), PIP ∼ Uniform(10°, 140°).
• Motion intensity distributions are documented in supplementary figures.

4. Model Benchmarks and Conditioning Protocols

HandX establishes rigorous benchmarks for generative modeling, evaluating both diffusion and autoregressive (AR) models:

• Diffusion models: Transformer decoder; layers D∈{4,8,12,16}; hidden dimensions {256,512,768}; FFN sizes {1024,3072}. Parameter counts (M): 4.63 (D=4), 26.33 (D=8), 38.95 (D=12), 260.97 (D=16).
• AR models: T5-based, text-prefix AR on FSQ tokens; layers ∈{8,12,16}; codebook sizes ∈{512, 1024, 2048, 4096}; parameter counts: 29.63–215.31 M.

Conditioning modes:

• Free-form text (including T_L, T_R, T_I fields)
• Partial skeleton (masked partial-denoising)
  • Motion in-betweening (first/last 5 frames fixed)
  • Keyframe-specification (arbitrary frames fixed)
  • Wrist-trajectory constraint (fix J_wrist)
  • Hand-reaction (one hand fixed)
  • Auto-regressive long-horizon chaining

All models are trained on the same held-out validation/test sets, with “few-data” studies facilitated by official smaller subsets.

5. Evaluation Metrics

Standard generative metrics:

• FID (Fréchet Inception Distance) over motion encoder features.
• Diversity (mean pairwise feature distance).
• R-Precision: top-k retrieval accuracy of text–motion matches.
• Multimodal Distance (MM Dist): $L_2$ distance between text and motion embeddings.

Hand-focused contact metrics (per-event, frame-level, 2 cm threshold):

• Contact precision:

$$C_\mathrm{prec} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}$$

• Contact recall:

$$C_\mathrm{rec} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}$$

• Contact $F_1$:

$$C_\mathrm{F1} = \frac{2\,C_\mathrm{prec}\,C_\mathrm{rec}}{C_\mathrm{prec} + C_\mathrm{rec}}$$

• Per-joint position error:

$$L_2 = \|\mathbf{p}_\mathrm{pred} - \mathbf{p}_\mathrm{gt}\|_2$$

All scores are averaged framewise, then over clips.

6. Scaling Trends and Modeling Results

HandX enables systematic scaling studies. For diffusion models (D = 12), Top-1 R-Precision improves with data/model scale: 0.343 (5% data), 0.357 (20%), 0.427 (100%). Contact $F_1$ rises to 0.641 at full scale, and FID drops as low as 1.14. Regression analysis for R-Precision vs. compute (FLOPs) gives

$$\mathrm{Rprec} = 0.4391 \times \log_{10}(\mathrm{FLOPS}) - 3.8707,$$

with Pearson $r=0.96$. For AR models (16 layers, codebook 4096), FID ≈ 1.72, R-Precision ≈ 0.281–0.481, $C_{F_1} ≈ 0.605$.

Qualitative analysis reveals that larger models and more data yield crisper finger articulation, more accurate contact timing (pinches, slides), and enhanced bimanual synchrony.

7. Access, Format, and Licensing

HandX is distributed in preprocessed TFRecord format, including 21-joint skeleton, per-frame rotation scalar, and multi-level JSON annotations. The dataset is available at https://handx-dataset.org.

Licensing is governed by the underlying sources:

• HandX package: CC BY-NC 4.0 (academic/research use only).
• Component datasets feature a mix of terms: GigaHands (CC BY-NC 4.0); HOT3D, ARCTIC, H2O (custom academic terms); HoloAssist (CDLA v2). All users should review sub-dataset licenses in /licenses.

Citation format: Z. Zhang et al., “HandX: Scaling Bimanual Motion and Interaction Generation,” CVPR 2024 (Zhang et al., 30 Mar 2026).