MR-UBi: Mixed Reality-Based Underwater Robot Arm Teleoperation System with Reaction Torque Indicator via Bilateral Control (2510.20407v1)

Abstract: We present a mixed reality-based underwater robot arm teleoperation system with a reaction torque indicator via bilateral control (MR-UBi). The reaction torque indicator (RTI) overlays a color and length-coded torque bar in the MR-HMD, enabling seamless integration of visual and haptic feedback during underwater robot arm teleoperation. User studies with sixteen participants compared MR-UBi against a bilateral-control baseline. MR-UBi significantly improved grasping-torque control accuracy, increasing the time within the optimal torque range and reducing both low and high grasping torque range during lift and pick-and-place tasks with objects of different stiffness. Subjective evaluations further showed higher usability (SUS) and lower workload (NASA--TLX). Overall, the results confirm that \textit{MR-UBi} enables more stable, accurate, and user-friendly underwater robot-arm teleoperation through the integration of visual and haptic feedback. For additional material, please check: https://mertcookimg.github.io/mr-ubi

• The paper introduces MR-UBi, a mixed reality teleoperation system that integrates bilateral control with a novel Reaction Torque Indicator to enhance underwater manipulation.
• The paper demonstrates quantitative improvements in torque regulation, with optimal torque events increasing from ~30% to ~70–75% and a mean absolute error reduction of over 50%.
• The paper shows enhanced usability and reduced operator workload, as validated by significant improvements in SUS scores and NASA-TLX metrics.

Mixed Reality-Based Underwater Robot Arm Teleoperation with Reaction Torque Indicator via Bilateral Control

Introduction and Motivation

The paper introduces MR-UBi, a mixed reality (MR) teleoperation system for underwater robot arms that integrates bilateral control with a novel Reaction Torque Indicator (RTI) visualized in a head-mounted display (MR-HMD). The system addresses the challenge of ambiguous haptic feedback in underwater environments, where hydrodynamic disturbances and limited operator training impede precise force regulation. By overlaying real-time torque information directly in the operator’s field of view, MR-UBi aims to enhance situational awareness, reduce cognitive load, and improve manipulation accuracy.

Figure 1: Overview of Proposed MR-UBi, highlighting the integration of underwater robot arm, bilateral control, and MR-HMD with RTI.

System Architecture

MR-UBi comprises three main modules:

1. Underwater Robot Arm: A cost-efficient, 3-DoF manipulator with a gripper, actuated by waterproof Dynamixel XW540-T260 motors. The mechanical design leverages aluminum profiles and 3D-printed components for durability and modularity.

   Figure 2: MR-UBi Arm, showing the 3-DoF structure and gripper mechanism.

2. Bilateral Control: Four-channel bilateral control synchronizes leader (operator) and follower (underwater) arms, exchanging both position and torque at 1 kHz. Sensorless torque estimation is performed via a Reaction Torque Observer (RTOB), ensuring real-time feedback and dynamic stability.

   Figure 3: Example of bilateral control data for the gripper joint (J4), demonstrating action-reaction torque tracking.

3. Reaction Torque Indicator (RTI): The RTI overlays a bar at the bottom of the MR-HMD view, encoding torque magnitude by bar length and torque state by a hybrid continuous-discrete color scheme. The color transitions smoothly between blue (low), green (optimal), and red (high), with linear interpolation near thresholds to facilitate fine modulation.

   Figure 4: Design of the RTI, illustrating bar length and color encoding for torque feedback.

   

   Figure 5: RTI overlay in MR-HMD passthrough, providing real-time visual feedback during teleoperation.

The Unity engine handles communication, visualization, and MR rendering, while the bilateral control loop operates independently at high frequency.

Figure 6: System overview, showing the underwater follower, user-side leader, and Unity-based MR-HMD integration.

Experimental Design

Sixteen participants (mostly novices) performed two manipulation tasks—Lift and Pick & Place—on objects of varying stiffness (rigid block and compliant sponge) in a controlled underwater environment. The paper compared MR-UBi against a baseline bilateral-control system without RTI, using a within-subject design.

Figure 7: Experimental environments and conditions, including water tank setup and operator station.

Figure 8: Experimental flow, detailing preparation, task execution, and post-task evaluation.

Figure 9: Task flow for Lift and Pick & Place operations.

Quantitative and Subjective Evaluation

Grasping Torque Regulation

MR-UBi demonstrated substantial improvements in torque control:

• Optimal Torque Proportion: Increased from ~30% (baseline) to ~70–75% (MR-UBi) across both tasks and objects.
• Low/High Torque Proportion: Both under- and over-grasping events were significantly reduced, with high torque events suppressed to <5% under MR-UBi.
• Mean Absolute Error (MAE): Decreased by more than 50%, indicating tighter regulation around the optimal torque.

  Figure 10: Comparison of gripper torque distributions, showing concentration within the optimal band for MR-UBi.

Usability and Workload

• System Usability Scale (SUS): Median score increased from 66.25 (baseline) to 83.75 (MR-UBi), $p<0.01$.
• NASA-TLX: Median workload score decreased from 49.50 to 41.17, $p<0.05$.
• Participants reported higher confidence, reduced effort, and lower frustration with MR-UBi.

  Figure 11: Results of SUS and NASA-TLX, indicating significant improvements in usability and workload.

Implementation Considerations

• Hardware Cost and Scalability: The underwater arm can be fabricated for approximately USD 3,400, enabling scalable deployment in research and field applications.
• Control Loop Frequency: Bilateral control at 1 kHz is critical for stable haptic feedback; lower frequencies may degrade performance, especially in dynamic underwater environments.
• RTI Visualization: The hybrid color scheme and continuous bar length provide both coarse and fine feedback, supporting novice and expert operators. The MR-HMD passthrough ensures minimal gaze shift and seamless integration with the real workspace.
• Sensorless Torque Estimation: RTOB-based estimation is robust for low-DoF arms but may require calibration for higher-DoF or more complex manipulators.

Implications and Future Directions

MR-UBi demonstrates that integrating haptic-based visual feedback with bilateral control can significantly enhance underwater teleoperation, especially for non-expert users. The approach is generalizable to other domains where haptic feedback is ambiguous or unreliable, such as space robotics or hazardous material handling.

Potential future developments include:

• Extension to Higher-DoF Arms: Scaling the RTI and bilateral control to more complex manipulators.
• Adaptive Feedback Schemes: Dynamic adjustment of RTI parameters based on task context or operator proficiency.
• Integration with Autonomous Assistance: Combining MR-UBi with shared autonomy or AI-based intent prediction for semi-autonomous manipulation.
• Field Deployment: Validation in real-world underwater scenarios, including deep-sea exploration and infrastructure maintenance.

Conclusion

MR-UBi provides a robust, cost-effective solution for underwater robot arm teleoperation, leveraging mixed reality and bilateral control to deliver intuitive, accurate, and user-friendly manipulation. The RTI’s visual-haptic integration enables operators to maintain optimal grasping torque, reducing both under- and over-force events. Experimental results confirm significant gains in manipulation accuracy, usability, and workload reduction. The system’s architecture and feedback design offer a promising foundation for future research and practical deployment in underwater robotics and beyond.