Portable TWIST2: Humanoid Teleoperation Suite

• Portable TWIST2 is a fully-portable, mocap-free teleoperation and data-collection suite designed for whole-body humanoid robot control using commodity VR hardware and 3D-printed robotic neck.
• It employs a two-stage optimization process for accurate motion retargeting and kinematic mapping, ensuring precise lower-body placement and effective neck articulation.
• A Redis-backed, real-time control pipeline integrates low-level PD control and high-level diffusion policies, enabling sub-100ms latency and scalable, vision-based autonomous control.

Portable TWIST2 is a fully-portable, mocap-free teleoperation and data-collection suite for whole-body humanoid robotic control, specifically designed to facilitate scalable, cost-effective human-to-humanoid transfer with egocentric vision. The system enables single-operator, holistic demonstration collection using commodity VR hardware and a 3D-printed robot neck with stereo cameras, alongside an extensible hierarchical visuomotor learning pipeline. By removing the need for external motion capture (mocap) setups and providing an integrated, reproducible stack, TWIST2 advances the scalability and accessibility of humanoid data collection and learning (Ze et al., 4 Nov 2025).

1. Hardware System Design

TWIST2 comprises three primary hardware subsystems: (1) a portable human motion source, (2) a humanoid platform with a modular add-on neck, and (3) a simulation environment replicating the physical stack.

• Portable Human Motion Source: Utilizes a PICO 4U VR headset and two PICO ankle trackers to capture whole-body human poses at 100 Hz with approximately 1 ms latency. All sensing is achieved without external cameras or markers, with a hardware cost near \$1,000. Data streaming to downstream processes occurs via XRoboToolkit over USB or Wi-Fi.
• Humanoid Platform and Modular Neck: Based on the Unitree G1 robot (29 DoF: 3 DoF waist, 2×6 DoF legs, 2×7 DoF arms) with two 7-DoF Dex31 hands. A custom, 2-DoF “TWIST2 Neck” is attached—3D-printed, servo-actuated (Dynamixel XC330-T288; U2D2 bus), powered directly from the robot’s 12 V/5 A line. The neck enables yaw (±90°) and pitch (±40°) with approximately 600 g additional weight. A Zed Mini stereo RGB camera (≈\$400) is affixed to the neck, facilitating dynamically adjustable egocentric vision.
• Simulation Counterpart: MuJoCo XML models for both the base robot and the TWIST2 neck subsystem enable rapid in-simulation development, control algorithm prototyping, and policy training.

2. Motion Retargeting and Kinematic Mapping

TWIST2 employs a real-time two-stage optimization process for mapping human joint angles ($\mathbf{q}_{\text{human}}$) to robot joint commands ($\mathbf{q}_{\text{humanoid}}$): an adaptation of GMR [Ze et al. 2025].

• Stage 1: Solves a local inverse kinematics (IK) problem (specifics not detailed).
• Stage 2 (Global Alignment): Finds robot joint targets that minimize both rotational and positional discrepancies between human and robot in a pelvis-centered frame:

$$\mathbf{q}^* = \arg\min_\mathbf{q} \bigg[ \sum_{i \in \mathcal{L}_{\text{up}} \cup \mathcal{L}_{\text{low}}} w_i^R \| R_i^{\text{human}} - R_i^{\text{robot}}(\mathbf{q}) \|_F^2 + \lambda_{\text{pos}} \sum_{k \in \mathcal{P}_{\text{low}}} w_k^p \| p_k^{\text{human,pelvis}} - p_k^{\text{robot}}(\mathbf{q}) \|^2 \bigg]$$

where $R_i \in SO(3)$ are link orientations, $p_k$ are positions, and $w, \lambda$ weight the cost terms. This structure enforces accurate lower-body (foot) placement while permitting liberal upper-body mapping.

• Neck Mapping: Derived directly from human spine and head orientation:

$$R_{\text{rel}} = R_{\text{spine}}^\top R_{\text{head}} \implies \begin{cases} q_{\text{yaw}} = \arctan2(r_{21}, r_{11}) \ q_{\text{pitch}} = \arcsin(-r_{31}) \end{cases}$$

for $R_{\text{rel}} = [r_{ij}]$.

• Forward Kinematics: Optionally, a classical product-of-exponentials model computes robot link transformations.

3. Software Architecture and Real-Time Control Pipeline

A Redis-backed distributed control stack orchestrates real-time data flow, motion retargeting, robot actuation, and egocentric vision:

• Data and Control Flow (50–100 Hz):
  • PICO → XRoboToolkit → Redis “/human_pose” channel.
  • Retargeter (C++/Python) processes pose, solves for command $\mathbf{p}_{\text{cmd}}$, publishes to “/cmd”.
  • Low-level controller $\pi_{\text{low}}$ consumes command and robot state from Redis, outputs joint targets $\mathbf{q}_{\text{humanoid}}$0, which are converted to torque via PD control:

  $\mathbf{q}_{\text{humanoid}}$1 - Stereo images (H.265) streamed via GStreamer for operator VR display; JPEG-compressed images sent to ZMQ for logging.

• Controllers:

  • Low-Level: Inputs include task-space pose references and robot state; outputs target joint positions. Trained via PPO on a 20,000-clip dataset (AMASS, OMOMO, TWIST2 MoCap) with composite rewards.
  • High-Level: In teleoperation, direct $\mathbf{q}_{\text{humanoid}}$2 is passed through. In autonomous mode, a Diffusion policy ($\mathbf{q}_{\text{humanoid}}$3) predicts future command trajectories from egocentric RGB (224×224) and past command history.
• Safety and Synchronization: Joystick input supports pause/resume; control signals are smoothed to eliminate discontinuities. End-to-end control latency is under 100 ms (compared to 500 ms for the prior TWIST system).

4. Data Collection Efficiency and Evaluation

TWIST2 enables efficient, scalable collection of whole-body humanoid demonstration data with detailed modality logging:

Modality	Sampling Rate (Hz)	Description
Stereo RGB Video	30	Egocentric stereo camera feeds
Robot Proprioception	50	Joint state, velocities, torques
Human Pose	100	Full-body 3D pose from VR setup
$\mathbf{q}_{\text{humanoid}}$4 (Retargeted)	-	Logged with robot and episode segmentation

• Throughput and Latency: All data streams operate ≥50 Hz; end-to-end loop time is <0.1s.
• Demonstration Throughput: An expert operator can collect ~100 bimanual pick-and-place demonstrations in 18 minutes (approximately 11 s/demo, 100% success rate), and ~46 mobile manipulation demonstrations in 20 minutes.
• Ablation Study (User Evaluation):
  • With full TWIST2: novice 75.6 s, expert 59.9 s per 10-demo run (avg 67.8 s), 20/23 successes.
  • -Stereo: 98.3 s, 20/27.
  • -No neck: 112.3 s, 16/29.
  • -No egocentric: 68.7 s, 20/23 (but requires operator to stand next to the robot).

This demonstrates the importance of the neck and stereo egocentric vision for operator performance and generalizability.

5. Hierarchical Visuomotor Learning Framework

TWIST2 supports a hierarchical, vision-based policy learning pipeline:

• Low-Level Controller ($\mathbf{q}_{\text{humanoid}}$5): Reused in both teleoperation and autonomous control.
• High-Level Diffusion Policy ($\mathbf{q}_{\text{humanoid}}$6):
  • Observation: Egocentric RGB image + command history.
  • Architecture: Visual encoder (ResNet-18 pretrained with R3M), temporal stacking via 1D convolutions.
  • Action: Predicts 64-step command chunks (2 s horizon) at 20 Hz on RTX 4090.
  • Objective: Simplified diffusion objective:

  $\mathbf{q}_{\text{humanoid}}$7

  where $\mathbf{q}_{\text{humanoid}}$8 is a noise-corrupted future action and $\mathbf{q}_{\text{humanoid}}$9. - Data Augmentation: Proprioception noise (10%), visual jitter (crop, rotate, color).

Empirical Tasks:

• WB-Dex (bimanual cup pick-&-place): 170 demonstrations; qualitative success 80–90%, with light-object drift identified as a failure case.
• Kick-T (T-shaped box kicking): 50 demonstrations; 6/7 real-world success, demonstrating robust left-foot kicking and step recovery.

6. Reproducibility and Open Source Artifacts

TWIST2 is open-sourced and designed for rapid adoption and extendability, with full documentation, hardware designs, datasets, and simulation resources publicly released:

• Code & Documentation: https://github.com/YanjieZe/TWIST2
• Datasets & Visualizations: https://twist-data.github.io
• Setup Workflow:

1. Assemble and mount TWIST2 neck on Unitree G1; attach Zed Mini camera.
2. Install XRoboToolkit, Redis, ZMQ, and GStreamer.
3. Equip operator with PICO 4U headset and ankle bands.
4. Launch control servers, retargeter, low-level controller, VR display.
5. Calibrate human pelvis to robot base frame (1 minute).
6. Operate and record; post-process recorded data with a provided GUI.

• Extendability: Includes MuJoCo models, tutorial training notebooks for both low-level and high-level controllers, and data conversion scripts.

Altogether, TWIST2 constitutes a low-cost (approximately \$1,650), scalable, and easily reproducible workflow for richly annotated, whole-body humanoid data generation and vision-based control, facilitating future data-driven humanoid robotics research (Ze et al., 4 Nov 2025).