LeFranX: VR Teleoperation for Imitation Learning

• LeFranX is an open-source implementation of the LeVR framework that enables VR-based expert demonstration collection and addresses bottlenecks in dexterous imitation learning.
• It integrates a Meta Quest VR interface with real-time arm teleoperation using differential inverse kinematics and constrained optimization for hand retargeting.
• The system provides a complete pipeline—from data logging and policy finetuning to deployment—facilitating reproducible benchmarking in robotic dexterous manipulation.

Searching arXiv for the specified paper to ground the article. tool call: arxiv_search({"query":"id:(Weng et al., 17 Sep 2025)", "max_results": 5, "sort_by": "submittedDate", "sort_order": "descending"}) LeFranX is the open-source, embodiment-specific implementation of the broader LeVR framework for VR teleoperation and imitation learning. It instantiates LeVR on a Franka Emika Research (FER) arm paired with a RobotEra XHand dexterous hand, and provides an end-to-end pipeline that goes from VR expert demonstration collection to policy finetuning and real-world rollout in the LeRobot ecosystem. In the underlying work, LeFranX is presented as a concrete system for addressing two practical bottlenecks in dexterous imitation learning: inadequate VR teleoperation support for multi-finger hands, and the absence of native support in LeRobot for VR-collected demonstrations (Weng et al., 17 Sep 2025).

1. Conceptual position within LeVR

LeVR is the modular framework, whereas LeFranX is its concrete realization for one robot embodiment. The system is explicitly designed to solve two bottlenecks in imitation learning for dexterous manipulation. First, VR teleoperation for dexterous hands is often poor or missing; many prior systems only support simple grippers or direct joint mappings that do not work well for multi-finger hands. Second, LeRobot does not natively accept VR-collected demonstrations, so researchers still need custom integration code before the data can be used for policy training (Weng et al., 17 Sep 2025).

LeFranX addresses these issues by combining a Meta Quest-based hand-tracking VR interface, retargeting for both arm and hand, a LeRobot extension that turns VR streams into standard observation/action logs, a Franka control server for real-time robot execution, and a dataset/policy pipeline that directly supports LeRobot training and deployment. In this formulation, LeFranX is neither a generic dexterous teleoperation abstraction nor merely a data logger; it is a complete pipeline spanning collection, representation, training, and deployment.

This positioning is central to its research utility. The paper characterizes LeFranX as a complete, reproducible, open-source pipeline for dexterous imitation learning, with a public dataset and a direct path to benchmarking visuomotor policies. The intended use cases include imitation learning research, benchmarking dexterous manipulation policies, studying teleoperation systems, evaluating arm-hand retargeting methods, and developing and comparing visuomotor policies in realistic manipulation settings.

2. VR interface and LeRobot integration

Expert demonstration collection is performed through a Meta Quest headset using the headset’s built-in hand tracking through the OpenXR Hand API. The OpenXR hand API produces a 27-landmark skeletal model, which the VR application converts to a 21-joint MediaPipe-style topology. At 30 Hz, the interface outputs the operator wrist pose $T_{\text{wrist},t} \in SE(3)$ and hand keypoints $$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$. The interface also shows a HUD and a shadow overlay that visually mirrors the tracked hand so the operator can verify tracking quality in real time (Weng et al., 17 Sep 2025).

Connectivity is implemented through a wired TCP connection over Android Debug Bridge (ADB) rather than wireless transport, with the stated purpose of reducing jitter. The 30 Hz stream is synchronized with camera frames, policy control, and robot actuation to maintain temporal consistency. This synchronization is important because LeFranX uses the same interface for both demonstration collection and policy rollout.

The bridge into LeRobot is implemented as a LeRobot extension. This adapter layer translates VR wrist pose and hand skeleton into robot-specific action commands and standardized logged episodes. It exposes two parallel branches: an arm branch, which maps wrist motion to end-effector motion, and a hand branch, which maps hand skeleton to dexterous finger joint targets. The resulting action format is unified as arm joint targets $\mathbf{q}^{\text{arm}}_t$ and hand joint targets $\mathbf{q}^{\text{hand}}_t$, while observations $\mathbf{o}_t$ contain synchronized robot proprioception and visual streams. The significance of this design is that VR-collected demonstrations become directly consumable by LeRobot training and deployment workflows without an additional custom conversion stage.

3. Arm teleoperation on the Franka FER

For the Franka FER arm, LeFranX uses a differential intent inverse kinematics formulation. At teleoperation start, it records the operator wrist pose $T_{\text{wrist},0}$ and the robot end-effector pose $T_{\text{ee},0}$. At each time step, it computes the operator’s motion relative to the start pose, maps that motion to the robot base frame, and solves inverse kinematics for the target end-effector pose (Weng et al., 17 Sep 2025).

The operator-space increments are defined as

$$\Delta \mathbf{p}_{\text{op}} = \mathbf{p}(T_{\text{wrist},t}) - \mathbf{p}(T_{\text{wrist},0}),$$

$$\Delta \mathbf{q}_{\text{op}} = \mathbf{q}_{\text{wrist},t} \otimes \mathbf{q}_{\text{wrist},0}^{-1}.$$

A calibrated transform maps from operator coordinates to robot coordinates, yielding

$$T^{\text{target}}_{\text{ee}} = \big(T^{\text{base}}_{\text{op}}[\Delta \mathbf{p}_{\text{op}}, \Delta \mathbf{q}_{\text{op}}](T^{\text{base}}_{\text{op}})^{-1}\big)\,T_{\text{ee},0}.$$

In the appendix, the coordinate mapping is written explicitly as

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$0

with translational and rotational mappings

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$1

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$2

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$3

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$4

For the Franka-specific inverse kinematics solver, LeFranX uses GeoFIK, a closed-form analytical IK solver for the Franka. Scaling is fixed at $$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$5, since the Franka workspace is similar to a human arm. The Franka’s 7-DOF redundancy is handled by optimizing the null space, especially the 7th joint. The redundancy objective balances proximity to a neutral pose, continuity with the previous pose, and manipulability:

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$6

where

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$7

is Yoshikawa manipulability. The solver uses Brent’s method with tolerance $$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$8 and a maximum of 100 iterations. The paper attributes smooth and stable real-time arm motion to this formulation.

4. Hand retargeting for the RobotEra XHand

The hand branch retargets the 21-keypoint human skeleton into the RobotEra XHand’s 12-DOF joint space. The paper describes this as a DexPilot-style optimization using the robot URDF, joint limits, and mimic constraints (Weng et al., 17 Sep 2025). The main optimization is

$$\mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21}$$9

The appendix gives a version with temporal smoothing:

$\mathbf{q}^{\text{arm}}_t$0

subject to joint bounds.

The XHand-specific configuration sets hand scaling to $\mathbf{q}^{\text{arm}}_t$1 and applies EMA smoothing,

$\mathbf{q}^{\text{arm}}_t$2

The system also introduces an adaptive pinky scaling heuristic to compensate for the XHand pinky being proportionally longer than the human pinky. Final commands are 12-DOF joint targets.

The solver is gradient-based and solved with SLSQP, using tolerance $\mathbf{q}^{\text{arm}}_t$3 and a maximum of 200 iterations. The paper also specifies adaptive reference handling with projected distances $\mathbf{q}^{\text{arm}}_t$4 m and saliency weights of 400 for projected finger-finger, 200 for projected wrist-finger, and 1 for free motion. These details show that LeFranX does not rely on direct joint mapping for dexterous control; instead, it uses a constrained retargeting procedure designed for embodiment mismatch and temporal consistency.

5. Real-time execution, dataset construction, and policy deployment

Because the Franka Control Interface (FCI) and libfranka require a real-time Linux kernel, LeFranX uses a separate computer for robot control. The Franka Robot I/O server receives high-level actions over TCP, sends proprioceptive feedback back to the client, and bridges the 30 Hz command stream to Franka’s 1 kHz control loop (Weng et al., 17 Sep 2025). For trajectory generation, the server uses Ruckig as an online jerk-limited trajectory generator. Each new 30 Hz action becomes a target for Ruckig, Ruckig generates a smooth feasible trajectory, and the robot is commanded at 1 kHz. The paper states that this is crucial for stable execution and makes the teleoperation feel smoother than naive direct command streaming.

LeFranX records data in LeRobot-compatible format and uploads it to Hugging Face. Proprioception and actions are stored in Apache Parquet, while visual data are stored as compressed MP4 with synchronized timestamps. The action space has 19 dimensions: 7 for the Franka arm and 12 for the XHand. Observations include robot proprioception and videos from three Intel RealSense D435 cameras: overhead, third-person, and wrist-mounted. Everything is synchronized at 30 Hz, with image resolution $\mathbf{q}^{\text{arm}}_t$5 and camera frame rate 30 fps.

The released public dataset contains 100 expert demonstrations per task across three tasks: Orange Cube Task, defined as picking and placing an orange cube into a blue bin; Boxed Pie Task, defined as opening a box lid, retrieving a pie, and placing it into a brown bin; and Bread Toaster Task, defined as inserting bread into a toaster and pressing the lever. The paper states that the dataset is available on Hugging Face.

LeFranX also supports the full imitation learning loop. Using LeRobot, the authors finetuned ACT and Diffusion Policy (DP), with each policy finetuned on the 100 demonstrations per task for 100k steps and the best validation checkpoint selected. During rollout, the robot receives synchronized multimodal observations and the policy outputs 19-DOF actions at 30 Hz. Inference was run on a remote GPU server using a gRPC-based policy server provided by LeRobot. The hardware setup consisted of the Franka FER plus RobotEra XHand, an NVIDIA RTX 6000 Ada policy inference server, and a System76 workstation for robot control and data transfer.

6. Empirical results, significance, and limitations

The evaluation covers both teleoperation efficiency and policy performance. Teleoperation efficiency was measured in a single-operator study in which one experienced operator performed each task directly by hand and via LeFranX teleoperation, with 10 repetitions per condition (Weng et al., 17 Sep 2025). Average completion times were 3.2 s for direct manipulation and 6.3 s for LeFranX teleoperation on Orange Cube, 5.7 s and 11.3 s on Boxed Pie, and 5.5 s and 12.0 s on Bread Toaster. The comparison reported Open-Teach teleoperation times of 11.9 s on Orange Cube and 20.5 s on Bread Toaster, with no Boxed Pie value reported. The paper’s takeaway is that LeFranX was faster than Open-Teach in this comparison, while teleoperation remained roughly $\mathbf{q}^{\text{arm}}_t$6 slower than direct human execution.

Policy evaluation used 10 autonomous rollouts for each task-policy pair, with randomized initial states and success defined as completing the full task. ACT achieved success rates of 8/10 on Orange Cube, 5/10 on Boxed Pie, and 4/10 on Bread Toaster. DP achieved 6/10, 3/10, and 1/10 respectively. The paper interprets these results as showing that both methods did well on the easiest task, performance degraded as task contact complexity increased, ACT outperformed DP in this setup, and DP was more sensitive to inference latency.

These results frame the significance of LeFranX in practical rather than purely algorithmic terms. The system provides VR-based expert demonstrations with low-cost consumer hardware, dexterous retargeting rather than gripper-only control, native LeRobot compatibility, a real-world deployment path, and a public dataset to benchmark policies. Its core contribution is therefore infrastructural: it lowers the engineering barrier between collecting teleoperated demonstrations, formatting them for learning, finetuning state-of-the-art visuomotor policies, and deploying them on a real robot.

The paper also states several limitations. Consumer VR tracking adds latency; teleoperation still lags behind direct manipulation; there is no haptic feedback; there is no egocentric vision; operator burden increases for harder tasks; the teleoperation evaluation is a single-operator case study rather than a large user study; DP is especially sensitive to inference latency; and more complex tasks expose policy weaknesses. These constraints are important for interpreting the reported gains: LeFranX is a practical, research-grade dexterous imitation learning system for the Franka FER and RobotEra XHand, but it does not eliminate the residual gaps between teleoperation and direct human manipulation, nor the performance degradation associated with increasing contact complexity.