UniDex-Cap: Portable Human-Data Capture

• UniDex-Cap is a portable human-data capture system that integrates commodity sensors and systematic calibration to convert egocentric human motions into precise robot control sequences.
• It employs a dual-sensor setup with synchronized RGB-D streams and hand pose estimates, supported by GUI-assisted intrinsic and extrinsic calibration to ensure subframe temporal alignment.
• Performance benchmarks indicate sub-33 ms latency, centimeter-level calibration and retargeting accuracy, and significant co-training gains for universal dexterous manipulation.

UniDex-Cap is a portable human-data capture subsystem within the UniDex suite, designed to acquire, synchronize, and retarget egocentric human demonstration data as robot-executable trajectories for dexterous manipulation research. It combines commodity hardware, systematic calibration, and interactive software for transforming multimodal human demonstrations—collected from synchronized RGB-D streams and detailed hand pose estimates—into high-fidelity robot control sequences suitable for universal, cross-hand dexterous manipulation tasks (Zhang et al., 23 Mar 2026).

1. Hardware Configuration and Calibration

UniDex-Cap utilizes a dual-sensor setup: the Apple Vision Pro (VP) headset and an Intel RealSense L515 depth camera, mechanically mounted with a custom 3D-printed rigid fixture. The RealSense is positioned immediately below the Visor of the VP for overlapping first-person workspace coverage, approximately ±60° horizontally and ±40° vertically. The VP delivers 3D hand and head pose estimates (with per-finger joint positions and root transforms) at 30 Hz in its intrinsic “VP frame.” The RealSense provides synchronized RGB (184 × 224 or 640 × 480 @30 Hz) and depth data (“RS frame”).

Both devices are hardware-synchronized at 30 Hz, with embedded millisecond-precision timestamps in each packet or frame. A host-side clock (e.g., ROS time) tags all incoming streams. For cross-device association, each VP pose is matched to the nearest RealSense RGB-D frame within ±10 ms, ensuring subframe-level temporal alignment.

Calibration consists of two stages:

• Intrinsic (RealSense): Manufacturer’s intrinsics are used, $$K = \left[\begin{array}{ccc} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{array}\right]$$, for mapping depth pixels to 3D points.
• Extrinsic (VP→RS): A GUI-assisted process estimates the rigid transform $T^{VP\to RS} \in SE(3)$ by displaying the VP hand skeleton overlayed with the RealSense pointcloud and allowing manual tuning (six sliders for three rotations and three translations) until visual alignment is attained. This calibration enables transformation of any VP joint $P_{VP}$ into the RS/world frame via $P_{RS} = T^{VP\to RS} \cdot P_{VP}$.

2. Data Acquisition and Preprocessing Pipeline

The data acquisition pipeline consists of the following key steps:

• Raw Stream Capture: RealSense streams color frames $(\text{RGB}_t)$ and depths $(\text{Depth}_t)$; Vision Pro outputs per-frame hand joint positions $\{\mathbf{J}_i \in \mathbb{R}^3\}$ and a root transform $T_{\text{hand}}^{VP}$.
• Temporal Synchronization: Each captured sample is associated with a timestamp $\tau$; RGB-D frames are paired to the closest VP joint pose sample within a 10 ms window.
• Preprocessing:
  • Depth-to-pointcloud conversion: For each image pixel at depth $d$ and coordinates $T^{VP\to RS} \in SE(3)$0, compute 3D position:

  $T^{VP\to RS} \in SE(3)$1

  Color from $T^{VP\to RS} \in SE(3)$2 is assigned; downsampling uses a 5 mm voxel grid. - Hand-masking: WiLoR and SAM2 segment human hands in RGB. Corresponding points are excised from the pointcloud, producing hand-free object data. - Temporal smoothing: Each $T^{VP\to RS} \in SE(3)$3 is low-pass filtered with $T^{VP\to RS} \in SE(3)$4:

  $T^{VP\to RS} \in SE(3)$5

All downstream processing is performed in the RS (“world”) frame after converting via $T^{VP\to RS} \in SE(3)$6.

3. Human-to-Robot Retargeting

Human-to-robot retargeting converts human joint pose sequences into temporally synchronized robot joint commands $T^{VP\to RS} \in SE(3)$7. The procedure is defined as follows:

• Fingertip Alignment via Inverse Kinematics (IK): Human fingertip positions $T^{VP\to RS} \in SE(3)$8 in the RS/world frame and the global human hand root $T^{VP\to RS} \in SE(3)$9 are extracted. The robot hand kinematic model accepts $P_{VP}$0 and a global base offset $P_{VP}$1, introduced between world and URDF base frames.

• IK Formulation:

  • Defining robot fingertip forward kinematics $P_{VP}$2, the concatenated residuals

  $P_{VP}$3 - The optimization problem:

  $P_{VP}$4

  with joint damping.

• Optimization Stages:

  • Automatic: Fix $P_{VP}$5, solve for $P_{VP}$6 via PyBullet multi-end-effector IK solver.
  • Interactive: $P_{VP}$7 (3D translation + 3D rotation) is exposed in a GUI; user adjusts offsets for visual contact improvement, re-running IK after each step (typically 3–5).
• Mimic Joint Corrections: For robot hands with linked joints, enforce $P_{VP}$8 constraints iteratively (2–3 passes to converge).

4. Software Architecture and Data Handling

UniDex-Cap is organized into four software modules:

Module Number	Functionality	Outputs/Formalisms
1	Capture: device drivers, timestamped logging	$P_{VP}$9, $P_{RS} = T^{VP\to RS} \cdot P_{VP}$0, $P_{RS} = T^{VP\to RS} \cdot P_{VP}$1
2	Preprocessing & Synchronization: pairing, conversion, hand-masking, smoothing	Paired frames, pointclouds, hand-masked data
3	Retargeting: compute $P_{RS} = T^{VP\to RS} \cdot P_{VP}$2, run automatic & GUI IK, mimic corrections	Robot trajectory $P_{RS} = T^{VP\to RS} \cdot P_{VP}$3, wrist pose $P_{RS} = T^{VP\to RS} \cdot P_{VP}$4
4	Packaging & Export: collate per-timestep records	ROS-bag or HDF5 with $P_{RS} = T^{VP\to RS} \cdot P_{VP}$5, action vectors, images

• Data Formats: Each timestep record stores:
  • Pointcloud $P_{RS} = T^{VP\to RS} \cdot P_{VP}$6
  • FAAS action vector $P_{RS} = T^{VP\to RS} \cdot P_{VP}$7 (18 wrist, 64 per-finger actuators)
  • Robot joint sequences $P_{RS} = T^{VP\to RS} \cdot P_{VP}$8
• Export Options: ROS-bag and HDF5, supporting downstream robot co-training and replay.

5. Performance and Validation

Empirical performance benchmarks for UniDex-Cap are as follows:

• Latency and Synchronization:
  • End-to-end acquisition lag from sensors to storage is approximately 33 ms per frame.
  • Timestamp jitter between the VP and RealSense streams is less than 5 ms (95th percentile).
• Calibration Precision:
  • Post-GUI extrinsic calibration achieves L2 error less than or equal to 1.5 cm on fingertip markers, orientation error less than or equal to 2°.
  • Joint smoothing reduces pose jitter to less than 2 mm RMS.
• Retargeting Quality:
  • Automatic IK produces average fingertip error on the order of 3–4 cm.
  • With three interactive $P_{RS} = T^{VP\to RS} \cdot P_{VP}$9 corrections, contact error reduces to about 1–2 cm for 90% of frames.
  • Mimic joint corrections typically converge in two iterations.
• Human-Robot Co-training Gains:
  • Qualitative visualization: previewed retargeting aligns human grasps to plausible robot contacts on objects (e.g., kettle handle and spout).
  • Quantitative findings: two retargeted human demonstrations are equivalent to one robot demonstration for fine-tuning on a "Make Coffee" manipulation task.
  • With 20 human and 10 robot demonstrations, downstream policy reaches 80% task progress, matching that of 50 robot-only demonstrations.

6. Significance and Extensibility

UniDex-Cap provides a reproducible, lightweight method for rapid, scalable capture of robot-ready dexterous demonstrations from human egocentric motion. Its pipeline—comprising a front-mounted sensor rig, GUI-supported calibration and retargeting, synchronized multimodal capture, and standardized FAAS-based action export—enables large-scale data collection necessary for universal dexterous manipulation policy development. The clear modular delineation between calibration, data association, finger-centric IK retargeting, and export makes the system straightforward to adapt for new robot hands or sensor configurations. This approach underlies demonstrated gains in co-training for dexterous manipulation and facilitates cross-hand policy generalization in the broader UniDex framework (Zhang et al., 23 Mar 2026).