LeVR: Modular VR Teleoperation for Imitation Learning

• LeVR is a modular VR teleoperation framework that enables efficient collection of robot demonstrations for imitation learning in dexterous manipulation by decoupling arm and hand control.
• It employs a dual-branch retargeting methodology using differential inverse kinematics for the arm and geometric optimization for the hand to enhance performance.
• LeVR integrates natively with the LeRobot framework, providing a seamless pipeline from VR data collection to policy training and deployment on real-world robots.

LeVR is a modular virtual-reality teleoperation framework for collecting robot demonstrations and using them directly for imitation learning, especially for robot systems that combine an arm with a multi-finger dexterous hand. It was introduced together with LeFranX, an open-source implementation for the Franka FER arm and RobotEra XHand, with the explicit aim of addressing two practical gaps: VR teleoperation that works for dexterous hands, and native integration with the LeRobot imitation learning framework so that VR-collected demonstrations can be used directly for training and deployment (Weng et al., 17 Sep 2025).

1. Definition, scope, and nomenclature

In its primary contemporary usage, LeVR denotes the framework presented in “LeVR: A Modular VR Teleoperation Framework for Imitation Learning in Dexterous Manipulation” (Weng et al., 17 Sep 2025). The framework is described as modular because it separates the system into reusable components: a VR hand-tracking interface, a LeRobot extension that converts tracked human motion into robot actions and LeRobot-compatible data, and an optional robot I/O server when hardware APIs require a special runtime environment. This decomposition is the paper’s main architectural idea.

The framework is specifically motivated by dexterous manipulation. The paper argues that many existing teleoperation systems are not well suited to dexterous hands: they often support only simple grippers, or rely on direct joint-angle mappings from the human hand to the robot hand, which become awkward when the robot hand’s kinematics differ from the human hand. LeVR instead uses a two-branch retargeting design, with separate arm and hand pathways, and treats VR teleoperation, data logging, and policy rollout as parts of one software stack (Weng et al., 17 Sep 2025).

The term is potentially ambiguous in nearby literature. A 2020 paper on the “Levitation Simulator” describes “LeVR” as the use of that simulator as a VR-based prototyping environment for ultrasonic levitation interfaces (Paneva et al., 2020). By contrast, “LaVR” is a scene-latent-conditioned video re-rendering method and is explicitly not a separate method called “LeVR” (Xie et al., 21 Jan 2026), while “LVR” in “Latent Visual Reasoning” denotes a multimodal latent-reasoning paradigm rather than the teleoperation framework (Li et al., 29 Sep 2025). This suggests that, in robotics and imitation-learning contexts, “LeVR” most precisely refers to the VR teleoperation framework of (Weng et al., 17 Sep 2025).

2. Modular system architecture

LeVR begins with commodity VR hardware. The reference system uses Meta Quest hand tracking through the OpenXR Hand API. The headset provides a 27-landmark skeletal hand model, which the application converts into a 21-joint MediaPipe-style hand topology. At each time step, the VR interface outputs a wrist pose and hand landmarks,

$$T_{\text{wrist},t} \in SE(3), \qquad \mathcal{K}_t = \{\mathbf{k}_i \in \mathbb{R}^3\}_{i=1}^{21},$$

and streams them at 30 Hz over a wired TCP connection via Android Debug Bridge (ADB) to reduce jitter (Weng et al., 17 Sep 2025).

Within the LeRobot extension, the tracked human motion is processed by two retargeting branches in parallel. The arm branch maps operator wrist motion to robot end-effector motion. The hand branch maps human hand geometry to robot finger joint targets. Because both branches use robot kinematic descriptions such as URDFs, the framework is presented as embodiment-flexible: porting to new hardware should primarily require robot-specific URDFs, controllers, and retargeting settings rather than rewriting the entire pipeline.

A standardized observation-action interface is central to the design. Observations $\mathbf{o}_t$ contain synchronized proprioception and vision, while actions $\mathbf{a}_t$ contain a unified joint-target structure including $\mathbf{q}^{\text{arm}}_t$ and $\mathbf{q}^{\text{hand}}_t$. The paper emphasizes that the same APIs are used during expert teleoperation and during autonomous rollout; the only difference is whether actions come from the VR operator or from a trained policy checkpoint. This interface is what makes VR-collected data natively usable inside LeRobot (Weng et al., 17 Sep 2025).

3. Retargeting methodology for arm and hand control

The arm branch uses what the paper calls differential intent inverse kinematics. Teleoperation starts by recording the operator’s initial wrist pose $T_{\text{wrist},0}$ and the robot’s initial end-effector pose $T_{\text{ee},0}$. At time $t$, operator motion is interpreted as a displacement relative to the start configuration:

$$\Delta \mathbf{p}_{\text{op}} = \mathbf{p}(T_{\text{wrist},t}) - \mathbf{p}(T_{\text{wrist},0}),$$

$$\Delta \mathbf{q}_{\text{op}} = \mathbf{q}_{\text{wrist},t} \otimes \mathbf{q}_{\text{wrist},0}^{-1}.$$

A calibrated transform maps this relative intent into the robot base frame, yielding the desired end-effector pose:

$\mathbf{o}_t$0

An inverse-kinematics solver then converts this target end-effector pose into arm joint commands $\mathbf{o}_t$1, respecting joint limits (Weng et al., 17 Sep 2025).

For the Franka-specific implementation, the appendix gives the explicit coordinate transform from VR/world frame to robot base frame,

$\mathbf{o}_t$2

followed by corresponding translational and rotational mappings. This shows that the system is not replaying tracked poses directly; it is constructing a relative human-intent signal, calibrating it into robot coordinates, and solving for robot-compatible arm configurations.

The hand branch addresses dexterous retargeting through geometric optimization rather than direct joint-angle copying. In the paper’s main formulation, robot hand joint angles are obtained by solving

$\mathbf{o}_t$3

The appendix gives a more detailed XHand-specific objective,

$\mathbf{o}_t$4

again with joint-limit constraints. The vectors include wrist-to-finger and fingertip-to-fingertip relations, and the optimization uses SLSQP with Jacobian-based gradients, tolerance $\mathbf{o}_t$5, and a maximum of 200 iterations. The same appendix specifies state-dependent reference-vector adaptation, weights of 400 for finger-finger projected constraints, 200 for wrist-finger projected constraints, and 1 in free motion, as well as an exponential moving average

$\mathbf{o}_t$6

and an adaptive pinky scaling heuristic

$\mathbf{o}_t$7

with $\mathbf{o}_t$8 (Weng et al., 17 Sep 2025).

4. LeFranX as the reference implementation

LeFranX instantiates LeVR for a Franka FER 7-DOF robot arm and a RobotEra XHand dexterous hand with 12 DOF. The hardware setup includes Meta Quest VR hardware for hand tracking, three Intel RealSense D435 cameras, a local workstation for robot control and data transfer, and a remote policy inference server with an NVIDIA RTX 6000 Ada Generation GPU. The paper also notes that local policy inference on an RTX 4060 was more problematic for latency-sensitive policies, especially Diffusion Policy (Weng et al., 17 Sep 2025).

The arm teleoperation implementation uses the GeoFIK analytical IK solver, chosen because it is closed-form, fast, and deterministic for the Franka. Since the Franka workspace is similar to that of a human arm, the arm scaling factor is set to 1. Redundancy in the Franka’s 7-DOF arm is handled by searching over the seventh joint and maximizing

$\mathbf{o}_t$9

where $\mathbf{a}_t$0 is Yoshikawa manipulability. The appendix states that, for fixed $\mathbf{a}_t$1, the remaining 6-DOF geometric IK can yield up to eight candidate solutions,

$\mathbf{a}_t$2

Because Franka’s FCI and libfranka require a real-time Linux kernel, LeFranX includes an embodiment-specific Franka server. The main PC runs VR and policy-inference client logic, while a separate real-time Linux machine runs the low-level server and communicates over TCP. Another important rate-mismatch bridge is Ruckig: teleoperation and policy pipelines operate at 30 Hz, but the Franka control loop runs at 1 kHz, so each incoming target is converted by Ruckig into a smooth jerk-limited 1 kHz trajectory. This is a central implementation detail of the framework rather than a peripheral optimization (Weng et al., 17 Sep 2025).

5. Demonstration collection and integration with imitation learning

The data-collection workflow is designed to match LeRobot’s expectations. During demonstration collection, synchronized observations and actions are recorded at 30 Hz. The action space is explicitly 19-dimensional: 7 arm joint targets and 12 hand joint targets. The observation space contains robot proprioception and video from three external Intel RealSense D435 cameras: one overhead, one third-person, and one wrist-mounted. The paper states the image resolution and frame rate as 320×240 at 30 fps. For storage, proprioceptive states and actions are saved in Apache Parquet, and video is stored as compressed MP4 with synchronized timestamps, following LeRobot conventions (Weng et al., 17 Sep 2025).

The released manipulation tasks increase in dexterous and sequential complexity. They are: Orange Cube Task, in which the robot picks up an orange cube and places it in a blue bin; Boxed Pie Task, in which it opens a box lid, picks up the pie, and places it into a brown bin; and Bread Toaster Task, in which it picks up a slice of bread, inserts it into a toaster, and pushes down the toaster lever. The paper identifies these tasks as moving from a basic pick-and-place to a multi-step manipulation task and then to a contact-rich insertion-and-actuation task.

One point is internally inconsistent. The abstract says the authors collected “a public dataset of 100 expert demonstrations,” but the later data-collection and evaluation sections say they collected 100 demonstration episodes for each of three tasks, implying 300 demonstrations total. The later sections are more specific. A plausible implication is that the dataset should be interpreted as 100 demonstrations per task, but the inconsistency remains part of the published description (Weng et al., 17 Sep 2025).

For policy learning, the authors fine-tuned two state-of-the-art visuomotor imitation-learning policies available in LeRobot: ACT (Action Chunking with Transformers) and DP (Diffusion Policy). Each model was fine-tuned separately on the 100 demonstrations for each task, for 100k training steps, with the best validation checkpoint selected for deployment. The paper does not specify additional hyperparameters such as learning rate, batch size, chunk horizon, or diffusion settings.

6. Empirical results

Evaluation consisted of 10 autonomous rollouts per task-policy pair from randomized initial states, using a binary success metric in which a rollout counted as successful only if the full task was completed. The reported success rates show that the demonstrations collected through LeFranX were sufficient to fine-tune competent visuomotor policies, but also that performance dropped as tasks became more contact-rich and sequential. ACT consistently outperformed Diffusion Policy in this setup (Weng et al., 17 Sep 2025).

Task	ACT	DP
Orange Cube Task	8/10	6/10
Boxed Pie Task	5/10	3/10
Bread Toaster Task	4/10	1/10

The paper also reports a teleoperation efficiency case study with a single experienced operator, one of the authors. Each task was performed 10 times under three conditions when available: direct human manipulation, LeFranX teleoperation, and Open-Teach teleoperation. The authors are explicit that this is a single-operator anecdotal case study rather than a definitive benchmark. Even so, they use it to argue that LeFranX supports reasonably efficient teleoperation and may outperform a direct-joint-mapping baseline for dexterous tasks.

Task	Direct / LeFranX / Open-Teach (s)
Orange Cube	3.2 / 6.3 / 11.9
Boxed Pie	5.7 / 11.3 / not reported
Bread Toaster	5.5 / 12.0 / 20.5

Qualitatively, the paper attributes the advantage over Open-Teach to preserving geometric intent through dexterous retargeting rather than using direct joint-angle mapping. It specifically notes that the baseline could not perform the prying motion required in the Boxed Pie Task. At the same time, LeFranX teleoperation remained slower than direct human manipulation, often about 2× slower (Weng et al., 17 Sep 2025).

7. Limitations, significance, and broader context

The paper does not include formal ablation studies isolating EMA filtering, pinky scaling, GeoFIK, camera views, or wired ADB transport, and it does not report statistical significance testing. Failure analysis is mainly qualitative. On the teleoperation side, the reported gap relative to direct human manipulation is attributed to kinematic mismatch between human and robot, robot inertia and motion limits, missing haptic feedback, and lack of egocentric vision. On the policy side, failures increase with task complexity. The authors observed that Diffusion Policy was particularly sensitive to inference latency, and that temporal misalignment between observation and action remained a recurring source of failure even when inference was offloaded to a remote GPU server (Weng et al., 17 Sep 2025).

Within dexterous imitation learning, LeVR’s significance lies in infrastructure rather than in a new policy architecture. Its technical core is a two-branch retargeting architecture—differential IK for the arm and geometric optimization-based retargeting for the hand—wrapped in a modular software design that makes VR-collected data directly useful inside LeRobot. The open-source components explicitly released are the LeVR/LeFranX codebase and the public demonstration dataset on Hugging Face. In the paper’s framing, the framework is intended for labs that want to experiment quickly with arm-hand systems without designing custom exoskeletons, leader-follower rigs, or one-off data formats.

In that sense, LeVR is best understood as infrastructure for dexterous imitation learning: a VR-to-robot-to-dataset-to-policy pipeline. Its claimed novelty is not only VR teleoperation, but out-of-the-box support for robot arms paired with dexterous hands and native integration with LeRobot. The LeFranX implementation demonstrates that this pipeline can work in practice on a Franka arm with an XHand, while the reported ACT and Diffusion Policy results indicate that the collected demonstrations are sufficient for real-world policy deployment, albeit with clear degradation on harder, latency-sensitive, contact-rich tasks (Weng et al., 17 Sep 2025).