HandX: Bimanual Hand Motion Synthesis

• HandX is a framework for high-fidelity bimanual hand motion synthesis, integrating large-scale motion capture data and fine-grained LLM-generated annotations.
• It employs a two-stage annotation pipeline and benchmarks diffusion and autoregressive models with specialized hand and contact metrics.
• Empirical results showcase scalable improvements in semantic alignment and motion realism, highlighting the impact of data and model depth.

HandX denotes a data, annotation, and generative modeling framework for high-fidelity bimanual hand motion and dexterous interaction synthesis. It is designed to overcome the limitations of existing human motion resources and models, which typically lack sufficient coverage or resolution of bimanual, finger-level interaction and inter-hand coordination. The HandX foundation delivers a unified motion capture corpus, a scalable annotation pipeline, hand-focused evaluation metrics, and benchmarking of state-of-the-art generative models for semantically conditioned, text-to-motion synthesis and interactive completion tasks (Zhang et al., 30 Mar 2026).

1. Data Collection, Integration, and Processing

HandX integrates and standardizes bimanual interaction data from two sources: (1) a comprehensive consolidation of public egocentric and hand-centric datasets (such as GigaHands, ARCTIC, HOT3D, H2O, HoloAssist), and (2) a new, marker-based motion capture dataset recording underrepresented, dexterous two-hand activities. Four preprocessing stages unify diverse sources: skeleton mapping to a 21-joint topology (wrist + MCP, PIP, DIP, tip for all fingers), global alignment of the coordinate frame (x: wrist-to-wrist, y: wrist-to-fingertip, z: up), frame-rate resampling to 30 Hz, and removal of artifacts (gaps, jitter) and static segments using a clip-level bimanual intensity filter based on angular velocity:

$$\bar\omega = \tfrac12\bigl(\bar\omega_{\mathrm{left}} + \bar\omega_{\mathrm{right}}\bigr)$$

where

$$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$

Clips are retained only if $\bar\omega_{\mathrm{left}}\ge25$, $\bar\omega_{\mathrm{right}}\ge25$, and $\bar\omega\ge30$ (in degrees/sec). Public dataset consolidation yields approximately 44 hours and 4.8 million frames; the new 36-camera OptiTrack mocap corpus contributes 10.2 hours and 1.1 million frames of rare, contact-rich bimanual activities. Post-processing produces a unified corpus of 54.2 hours (5.9 million frames), partitioned into 2-second clips for downstream annotation (Zhang et al., 30 Mar 2026).

2. Scalable Annotation Pipeline

Manual captioning at finger-level granularity is computationally prohibitive for the scale of HandX. Instead, a two-stage decoupled annotation pipeline is introduced combining automatic kinematic event extraction and LLM based description generation.

A. Kinematic Feature Extraction:

For each 60-frame clip, six descriptors are computed per hand:

• Finger Flexion $\theta^f_t$, state-categorized (Hyper-extend, Fully-extend, Partially-bend, Fully-bend)
• Finger Spacing (inter-finger angle)
• Finger–Finger Distance (binary contact <2 cm)
• Finger–Palm Distance (states: contact, near, far)
• Palm–Palm Relation (vector between cross-hand palm samples, decomposed by axis)
• Wrist Trajectory (3D path)

These descriptors are temporally segmented into a compact list of "events" (e.g., frame/time ranges with categorical state changes or contact onset/offset).

B. LLM-Based Captioning:

Extracted kinematic events are formatted as JSON and provided to an LLM (e.g., GPT-4, Gemini) using a prompt structured for five semantic detail levels, enabling concise to fine-grained, multi-granular text descriptions. This process produces 486,000 captions and supports nuanced, semantically aligned supervision for generative modeling (Zhang et al., 30 Mar 2026).

3. Generative Modeling Architectures for Bimanual Motion

HandX benchmarks two families of semantically conditioned motion generation models:

A. Diffusion Model:

Each frame is represented as $x^i = [p^i; s^i]$, $p^i \in \mathbb{R}^{2J \times 3}$ (joint 3D), $s^i \in \mathbb{R}^{2J}$ (1-DoF rotation scalars). Denoising diffusion is performed as:

$$x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon, \quad \bar\alpha_t = \prod_{t'=1}^t (1-\beta_{t'})$$

Model training minimizes: $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$0

Text conditioning is achieved by cross-attention to T5-encoded embeddings of left/right/interaction-level annotations. Inference supports partial conditioning, including keyframe in-betweening and per-joint soft clamping.

B. Autoregressive Model with Finite Scalar Quantization (FSQ):

Local frame state is discretized: $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$1 A FSQ encoder $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$2 produces latent tokens (codebook size $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$3), and a decoder $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$4 reconstructs frames, supervised by $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$5.

A text-prefix causal Transformer autoregressively models the token sequence, optimizing: $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$6 where $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$7, $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$8, $$\bar\omega_h = \frac{\sum_{t,j\in\mathcal J_h}\lambda_j\,\omega_{t}^{(j)} } {\sum_{t,j\in\mathcal J_h}\lambda_j}$$9 are left, right, and interaction text descriptors (Zhang et al., 30 Mar 2026).

4. Evaluation Metrics and Quantitative Benchmarks

HandX defines both standard and novel hand- or contact-focused metrics:

Metric Type	Description	Notes
FID (Fréchet Inception Dist.)	Distributional realism of generated motion	Embedding-based
R-Precision, MM-Dist	Text–motion alignment	Cross-modal
Diversity	Inter-sample variance for given prompt
Contact-F1/Precision/Recall	Correct detection of thumb–fingertip or hand–hand contact (2 cm threshold)	Intra-/inter-hand split
Contact ratio/duration/freq.	Corpus statistics: duration, frequency, proportion of frames in contact

Scaling ablations show strong positive effects for both data and model size on text-matching (Top-3 R-Precision), contact F1 (up to 0.64 for diffusion, 0.60 for AR), and FID (as low as ~1.35 for diffusion, ~1.7 for AR with 100% data). Perceptual studies report that HandX-generated motions are rated more natural than GigaHands or HoloAssist (mean 4.2/5 vs. 3.1/5), and captions are judged more accurate (4.4/5 vs. 2.8/5) (Zhang et al., 30 Mar 2026).

5. Empirical Results and Scaling Laws

Experimental results establish clear empirical scaling laws: as both data quantity (5%–100% of corpus) and model depth (4–16 diffusion layers, 8–16 AR layers, codebooks 512–4096) increase, motion quality, semantic alignment, and fidelity improve log-linearly up to saturation. Specifically, for diffusion models at limited data, Top-3 R-Precision scales as

$\bar\omega_{\mathrm{left}}\ge25$0

with Pearson $\bar\omega_{\mathrm{left}}\ge25$1. Qualitative samples highlight the synthesis of natural, semantically controlled bimanual behaviors, including pinch, twist, slide, and hand-object transfers, and demonstrate flexible conditioning such as partial keyframing and trajectory specification for realistic, plausible hand completion and interpolation (Zhang et al., 30 Mar 2026).

6. Limitations, Significance, and Future Directions

HandX represents the first large-scale (~54 h, ~6 million frames) bimanual, finger-dense hand motion corpus with nearly half a million fine-grained LLM-based annotations, directly enabling systematic study of dexterous coordinated behavior. Despite this scale, HandX cannot span the full diversity of human social and cultural dexterity (e.g., musical performance, rich sign language). The aggregation process may admit minor reconstruction errors, and LLM-generated captions may hallucinate some details.

Future research directions include integrating HandX with full-body motion datasets for unified synthesis, incorporating explicit physical or contact simulation priors, developing unsupervised or weakly supervised learning with raw RGB/D data, and adapting HandX-trained models for telepresence, robotics, and sign language generation under non-commercial research licenses (Zhang et al., 30 Mar 2026).

7. Context within the 3D Hand Modeling Ecosystem

HandX differs fundamentally from approaches like XHand (Gan et al., 2024), which focuses on high-fidelity, real-time hand mesh generation and rendering (geometry, appearance, and pose-dependent detail), or body-centric mesh recovery frameworks such as DanceHMR (Shen et al., 18 May 2026), which model hand pose within a whole-body temporal architecture. Whereas XHand and DanceHMR target mesh generation and tracking, HandX specifically addresses the generative modeling and semantic conditioning of fine-grained, bimanual hand motion, providing the necessary data and evaluation infrastructure for algorithmic advances in dexterous behavior synthesis.