An Augmented Reality Interface for Teleoperating Robot Manipulators

Abstract: Effective real-time robot control is essential as we increasingly integrate robots into various societal contexts. Moreover, obtaining high-quality demonstration data is critical for the success of data-driven approaches, such as imitation learning. Existing platforms for robot control and data collection in manipulation tasks often place significant physical and mental demands on the user, require additional hardware, or necessitate specialized knowledge. In this work, we introduce a novel augmented reality interface for teleoperating robotic manipulators, focusing on the user experience, particularly when performing complex, precise tasks. Designed for the Microsoft HoloLens 2, this interface leverages the adaptability of mixed reality, allowing users to control a physical robot via a digital end effector surrogate. We evaluate the effectiveness of our approach across four complex manipulation tasks and compare its performance with the 3D SpaceMouse, a traditional teleoperation method in robotics, and kinesthetic teaching, the assumed performance upperbound in robotic control. Our findings reveal that, quantitatively, our method addresses a key limitation of the SpaceMouse: its unintuitive mapping of rotations. Additionally, a user study demonstrates that our AR-based system achieves higher usability scores and recommendation likelihood, and lower task load compared to the SpaceMouse.