XRoboToolkit: XR Teleoperation Framework

• XRoboToolkit is a modular, cross-platform framework for real-time robot teleoperation using XR headsets, integrating diverse devices and simulators.
• It employs multi-modal tracking and optimization-based inverse kinematics to deliver low-latency stereoscopic feedback and precise control, achieving sub-millimeter accuracy in tasks.
• The system provides robust data collection and seamless integration with various simulators and robotic platforms, supporting advanced Vision-Language-Action research.

XRoboToolkit is a modular, cross-platform software framework designed for real-time robot teleoperation using Extended Reality (XR) headsets. Built upon the OpenXR standard, it integrates heterogeneous XR devices, robot hardware, and simulators with a unified, extensible architecture. XRoboToolkit addresses the scalability, setup complexity, and data quality limitations of prior teleoperation systems by delivering low-latency stereoscopic feedback, multi-modal tracking, real-time optimization-based control, and robust data collection facilities suitable for Vision-Language-Action (VLA) research (Zhao et al., 31 Jul 2025).

1. Architecture and Layered System Design

XRoboToolkit is organized as a set of modular software layers bridging user XR input, middleware services, and robot-side control. The primary components are:

• XR-Side Unity Client: Operates as an OpenXR application on platforms such as PICO 4 Ultra and Meta Quest 3. It captures head, controller, hand, and auxiliary tracker poses (up to 90 Hz), streams stereo video from onboard (PICO) or external (ZED Mini) cameras, and provides a GUI for network setup, tracking options, and data logging.
• PC Service (C++): An asynchronous daemon that ingests JSON-encoded 6-DOF pose data ([x, y, z, qx, qy, qz, qw], right-handed system) at 90 Hz, parses it, and republishes data using TCP/UDP or shared memory for consumption by robot-side modules.
• Pybind Interface: Exposes all XR pose streams directly in Python, abstracting device and data handling for user-level scripts.
• Robot-Side Modules: Include XRoboToolkit-Robot-Vision (stereo video streaming), XRoboToolkit-Teleop-Sample-Python (implementing inverse kinematics, dexterous hand retargeting, and mobile base control), and adapters for simulators (MuJoCo) and real robots (UR5, ARX R5, Galaxea R1-Lite) including gripper devices like the Shadow Hand.

The system achieves conformant stereoscopic vision delivery based on OpenXR conventions. On PICO 4 Ultra, video is streamed directly with approximately 100 ms round-trip latency. When using the ZED Mini with a host PC relaying to PICO 4 Ultra, mean end-to-end latency is reduced to approximately 82 ms by leveraging the laptop GPU. Custom Unity shaders set the interpupillary distance and focal plane (approximately 1 m) for optimized depth perception.

2. Multi-Modal Tracking and Task-Space Teleoperation

XRoboToolkit supports simultaneous, synchronized tracking across five sensor modalities:

• Head: 6-DOF pose, status, current input mode.
• Controllers: 6-DOF pose, analog triggers/grips, button states, joystick axes.
• Hand Gestures: 26 joint poses per hand, sampling at 60 Hz.
• Whole-Body: 24 joint positions, velocities, accelerations via PICO’s full-body model.
• Auxiliary Motion Trackers: Pose, velocity, acceleration, serial ID.

All streams are timestamped and unified into a single JSON object at 90 Hz, then fused into task-space robot commands. For example, pressing the controller grip causes the controller’s motion (Δx, Δy, Δz, Δroll, Δpitch, Δyaw) to specify a desired end-effector velocity for the inverse kinematics module. Auxiliary elbow trackers provide additional positional constraints, facilitating redundancy resolution and anthropomorphic posture generation in 7-DOF arms.

3. Optimization-Based Inverse Kinematics and Retargeting

Manipulator and dexterous hand control leverage real-time Quadratic Program (QP) optimization at ≥100 Hz (PlaCo/Pinocchio backend). For a joint configuration $q$, joint velocity $\dot{q}$ is found by minimizing:

$$\min_{\dot{q}} \sum_{i=1}^N w_i \| J_i(q)\dot{q} + e_i(q) \|^2 + \epsilon \|\dot{q}\|^2$$

subject to

$\dot{q}_{min} \leq C(q) \dot{q} \leq \dot{q}_{max}$

where $J_i(q)$ is the task Jacobian, $e_i(q)$ the residual error, $w_i$ task priority weight, $\epsilon$ a regularization term, and $C(q)$ encodes joint velocity, collision, and workspace constraints.

To prevent singularities, the manipulability measure $m(q) = \sqrt{\det [J(q) J(q)^T]}$ is monitored and optionally used to adjust regularization or as an optimization criterion.

Dexterous hand retargeting is cast as, at each time step $\dot{q}$0:

$\dot{q}$1

subject to

$\dot{q}$2

where $\dot{q}$3 is the $\dot{q}$4-th human hand keypoint, $\dot{q}$5 maps robot hand joints to task-space points, $\dot{q}$6 compensates for scale differences, and $\dot{q}$7 smooths the solution. The dex_retargeting module executes this at 60 Hz.

4. Communication Latency and Benchmark Performance

Empirical evaluation of XRoboToolkit demonstrates substantial improvements over legacy teleoperation pipelines:

Video Pipeline	Mean Latency (ms)	Std Dev (ms)
Open-TeleVision (ZED→Quest 3)	121.5	6.0
XRoboToolkit (ZED→PICO 4 Ultra)	82.0	6.3
XRoboToolkit (PICO→PICO 4 Ultra)	100.5	3.1

In precision manipulation tasks—such as dual UR5 arms inserting a 3 mm screwdriver into a 4 mm hole—operators achieved sub-millimeter accuracy under stereoscopic control. For mobile manipulation and redundancy resolution, elbow-tracker teleoperation enabled natural arm postures during prolonged bimanual tasks, such as carpet folding, with 95%+ success rates.

Subjective operator feedback indicates reduced motion sickness due to synchronous visual-vestibular cues (shader focal tuning, omission of roll), and the neck-worn configuration supported fatigue-free operation for more than 30 minutes.

5. Dataset Collection for Vision-Language-Action (VLA) Research

The Data Collection panel enables simultaneous recording of:

• High-rate tracker pose streams at 90 Hz.
• Robot joint state and control command streams at 50 Hz.
• Multiple RGB camera streams (424×240, 50 FPS).

In a bimanual carpet folding dataset, 100 demonstrations (20 s each, ~1,000 frames per demo) were collected on ARX R5 dual arms with wrist and overhead cameras, each frame capturing 14-dimensional joint states, 14-dimensional commands, and three RGB images.

VLA models were trained on this data, with fine-tuning of the $\dot{q}$8 policy using LoRA for 80,000 steps (batch size 16, horizon 50), achieving 100% success in a 30-minute continuous test and exhibiting autonomous regrasping and repositioning as emergent behaviors.

6. Extensibility and Future Directions

Planned and ongoing extensions include:

• Hand retargeting support for underactuated grippers (e.g., INSPIRE) via explicit modeling of joint couplings.
• Direct integration with additional simulators beyond MuJoCo (including Roboverse, Gazebo, Isaac Gym).
• Whole-body teleoperation by standardizing OpenXR full-body schemas and validation on humanoid platforms.
• Incorporation of learned control primitives from VLA models to implement shared autonomy for teleoperators.
• Standardization efforts to contribute XRoboToolkit’s conventions to the OpenXR standard for greater cross-vendor interoperability.

In aggregate, XRoboToolkit constitutes a unified pipeline supporting low-latency stereoscopic XR feedback, multi-modal tracking, QP-based control, and large-scale, high-quality demonstration data acquisition, catalyzing both high-fidelity teleoperation and VLA-centered research (Zhao et al., 31 Jul 2025).