CaFe-TeleVision: VR Teleoperation Framework

• CaFe-TeleVision is a VR-based teleoperation framework that integrates coarse-to-fine control with immersive, situated visualization for remote bimanual robotic tasks.
• The system maps human IMU data and VR controller inputs into precise, ergonomic end-effector poses using a multi-objective optimization approach and real-time impedance control.
• Controlled trials demonstrate significantly faster task completion, improved success rates, and reduced physical and cognitive load compared to conventional teleoperation baselines.

CaFe-TeleVision is a VR-based teleoperation framework that integrates a coarse-to-fine control paradigm with immersive, situated visualization, targeting enhanced efficiency and ergonomics in remote robot manipulation tasks. Designed around a bimanual humanoid platform, the system addresses common challenges in teleoperation—particularly physical strain and cognitive load—through a multi-objective retargeting approach and an on-demand visual feedback interface. CaFe-TeleVision is validated through controlled trials on bimanual tasks, demonstrating statistically significant improvements over comparative baselines in success rate, task completion time, and ergonomic metrics (Tang et al., 16 Dec 2025).

1. System Architecture

CaFe-TeleVision is implemented on a robot platform comprising two Franka Emika Panda arms mounted on a humanoid torso "CURI" equipped with a neck pan–tilt mechanism. Operator motion is captured using eleven Xsens MVN inertial measurement units (IMUs) distributed over the upper body, sampling 6-DoF hand poses at 60 Hz. Visual data streams originate from a ZED 2i stereo global "eye" camera (1080p@15 Hz), complemented by two Intel RealSense D435i "wrist" cameras rigidly attached to each arm's flange.

A VR interface based on the Meta Quest 2 HMD, together with a Unity-based GUI, supports stereo video feedback and operator interaction. Communication utilizes standard ROS drivers for arm command transmission and ZeroMQ for low-latency gripper-anchor updates.

The system software pipeline is composed of:

1. Perception module for real-time ingest and rendering of stereo and wrist camera streams, as well as VR controller state;
2. Retargeting node, which maps human IMU data and controller commands into end-effector poses through a coarse-to-fine logic;
3. Control subsystem, which relays target poses via ROS at 60 Hz to a model predictive impedance controller running joint-level QPs at 500 Hz;
4. Visualization layer in Unity, which maps the robot gripper location into the VR HMD space and dynamically manages wrist-view overlays.

The end-to-end data flow can be summarized as: Perception modules → VR HMD ← Controller inputs ↑ retargeting ← Human IMUs + VR controllers ↓ control → Robot arms → Cameras → Perception

2. Coarse-to-Fine Retargeting and Control Methodology

The central retargeting mechanism in CaFe-TeleVision employs a multi-objective optimization strategy, concurrently minimizing end-effector pose error and deviation from neutral, ergonomic robot postures. For each arm, the system solves:

$$\min_{q} \; J_{\rm eff}(q) + \lambda\,J_{\rm ergo}(q) \quad\text{s.t.}\; q_{\rm min}\!\le q\le q_{\rm max},\;\dot q_{\rm max},\;\tau_{\rm max}\,.$$

where

• $J_{\rm eff}(q)$ quantifies pose error in SE(3),
• $J_{\rm ergo}(q)$ penalizes postural strain,
• $\lambda>0$ controls the efficiency-ergonomics trade-off.

Coarse (Natural) Mode maps the human hand pose to the robot frame via scaling ($s$) of workspace dimensions and alignment of reference frames; orientation mapping composes the human hand orientation with robot-specific transforms. Fine (Joystick) Mode permits incremental end-effector rotations via thumbstick inputs $(u_1,u_2)$, applying small-angle updates about $z$ and $y$ axes. Upon release, spherical linear interpolation (SLERP) restores the manipulator to the natural reference when the angular deviation falls below a set threshold $\epsilon_{\rm ori}$.

The underlying QP for target joint positions is solved at 500 Hz within an inner impedance control loop, ensuring real-time execution. Coarse mode leverages closed-form scaling and alignment; fine mode injects incremental rotations directly into the pose tracking objective.

3. Immersive Situated Visualization

To reduce cognitive load from multi-view management, CaFe-TeleVision incorporates an on-demand situated visualization technique. Wrist camera views are displayed only when requested by the operator (via VR buttons R4/R5 polled at 50 Hz), and are anchored contextually to the robot gripper within the HMD’s virtual space.

The system computes the gripper position in the eye camera frame, transforms it into Unity world space, and projects it onto a focal plane to define a 2D anchor for overlay placement. This ensures the wrist-view quads—overlaid with fixed screen size and deliberate offset—do not occlude the main eye view, efficiently supporting spatial task awareness without increasing visual clutter.

4. Experimental Evaluation and Quantitative Results

A user study involving 24 participants (16 male, 8 female; ages 19–33; mean teleoperation skill 3.2/5; mean VR skill 3.4/5) assessed CaFe-TeleVision’s performance across six representative bimanual manipulation tasks:

1. Insert torus (coarse alignment)
2. Grasp fruits (occlusion management)
3. Pour tea (fine orientation)
4. Twist cap (large wrist rotation)
5. Hang towel (flat placement)
6. Pack bag (complex coordination)

Quantitative metrics included:

• Success rate ($\text{SR} = \#$successes/$\#$trials)
• Task completion time
• NASA-TLX (Mental, Physical, Temporal Demand, Performance, Effort, Frustration)
• SUS ("System Usability Scale," 10-item)

Comparisons were made against three baselines: N-S (natural mode + stereo only), CaFe-SL (coarse-to-fine + static multi-view), R-TeleVision (relative mode + situated).

Statistical analysis was conducted using Friedman tests for repeated measures, post-hoc Nemenyi tests, and paired t-tests. Key results:

• CaFe-TeleVision achieved up to 26.81% faster task completion vs. N-S ($p<0.05$) and 29.3% faster vs. R-TeleVision ($p<0.001$).
• In expert trials, success rate gains reached 28.89% (e.g., pour tea: 46.7% to 100%), with completion time reductions up to 26.81%.
• NASA-TLX indicated statistically lower scores in physical demand (PD↓ by ∼15%), mental demand (MD↓ by ∼12%), and temporal demand (TD↓ by ∼14%) for CaFe-TeleVision ($p<0.01$).
• SUS favored CaFe-TeleVision in both retargeting ($p=0.0034$) and perception ($p=0.0000$) modes.

5. Ergonomic Impacts and Cognitive Considerations

The adoption of coarse-to-fine mapping within CaFe-TeleVision mitigates the need for operators to assume extreme wrist postures, reducing musculo-skeletal strain. Immersive, on-demand visualization via gripper-anchored wrist views minimizes the necessity for gaze shifts and lessens the impact of visual occlusion, thereby addressing both cognitive and temporal demands.

Physical and cognitive ergonomic improvements are substantiated by the measured reductions in task load (NASA-TLX) and the increased operator acceptance (SUS scores), as established through statistically robust user studies.

6. Limitations and Potential Extensions

Documented limitations of CaFe-TeleVision include:

• The joystick fine-control mapping presumes approximate alignment between gripper approach and operator viewpoint; future iterations are expected to implement viewpoint-adaptive mapping.
• The current system does not provide haptic feedback; extension towards tactile cue integration is anticipated, especially for tasks with significant force interaction requirements.
• Collision avoidance is presently operator-mediated, with planned enhancements involving explicit constraint enforcement within the QP control framework for autonomous safety.

A plausible implication is that these augmentations could further generalize CaFe-TeleVision’s applicability and improve both safety and task fidelity in human-robot teleoperation scenarios.

7. Comparative Position and Future Directions

CaFe-TeleVision establishes a practical, plug-and-play solution for VR-based teleoperation, advancing state-of-the-art approaches by jointly optimizing task efficiency and operator ergonomics. The demonstrated improvements in success rates, completion times, and ergonomic metrics position the framework as a reference point for future research in immersive teleoperation systems. Continued development along the identified limitations—such as adaptive viewpoint mapping, haptic augmentation, and autonomous safety constraints—is projected to reinforce the generalizability and robustness of the methodology in complex manipulation environments (Tang et al., 16 Dec 2025).