OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation (2403.07870v1)
Abstract: Open-sourced, user-friendly tools form the bedrock of scientific advancement across disciplines. The widespread adoption of data-driven learning has led to remarkable progress in multi-fingered dexterity, bimanual manipulation, and applications ranging from logistics to home robotics. However, existing data collection platforms are often proprietary, costly, or tailored to specific robotic morphologies. We present OPEN TEACH, a new teleoperation system leveraging VR headsets to immerse users in mixed reality for intuitive robot control. Built on the affordable Meta Quest 3, which costs $500, OPEN TEACH enables real-time control of various robots, including multi-fingered hands and bimanual arms, through an easy-to-use app. Using natural hand gestures and movements, users can manipulate robots at up to 90Hz with smooth visual feedback and interface widgets offering closeup environment views. We demonstrate the versatility of OPEN TEACH across 38 tasks on different robots. A comprehensive user study indicates significant improvement in teleoperation capability over the AnyTeleop framework. Further experiments exhibit that the collected data is compatible with policy learning on 10 dexterous and contact-rich manipulation tasks. Currently supporting Franka, xArm, Jaco, and Allegro platforms, OPEN TEACH is fully open-sourced to promote broader adoption. Videos are available at https://open-teach.github.io/.
- Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
- Task-oriented hand motion retargeting for dexterous manipulation imitation. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
- Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation. arXiv preprint arXiv:2203.13251, 2022.
- Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962–5969. IEEE, 2023.
- Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.01918, 2023.
- Discriminative and adaptive imitation in uni-manual and bi-manual tasks. Robotics and Autonomous Systems, 54(5):370–384, 2006.
- Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
- A systematic review of commercial smart gloves: Current status and applications. Sensors, 2021. ISSN 1424-8220. doi: 10.3390/s21082667.
- Extreme parkour with legged robots. arXiv preprint arXiv:2309.14341, 2023.
- Iterative residual policy: for goal-conditioned dynamic manipulation of deformable objects. arXiv preprint arXiv:2203.00663, 2022.
- Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
- Imitation learning from pixel observations for continuous control. 2021.
- From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047, 2022.
- Low-cost exoskeletons for learning whole-arm manipulation in the wild. arXiv preprint arXiv:2309.14975, 2023.
- Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024.
- Learning to fly by crashing. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3948–3955. IEEE, 2017.
- Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control. IEEE Transactions on Robotics, 2022.
- Openvr: Teleoperation for manipulation. arXiv preprint arXiv:2305.09765, 2023.
- Telerobotic control in virtual reality. In OCEANS 2019 MTS/IEEE SEATTLE, pages 1–8, 2019. doi: 10.23919/OCEANS40490.2019.8962616.
- See to touch: Learning tactile dexterity through visual incentives. arXiv preprint arXiv:2309.12300, 2023.
- Watch and match: Supercharging imitation with regularized optimal transport. arXiv preprint arXiv:2206.15469, 2022.
- Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497, 2023.
- Megatrack: Monochrome egocentric articulated hand-tracking for virtual reality. 2020.
- Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164–9170, 2020. doi: 10.1109/ICRA40945.2020.9197124.
- The dlr bimanual haptic device with optimized workspace. In 2011 IEEE International Conference on Robotics and Automation, pages 3441–3442. IEEE, 2011.
- Control of a quadrotor with reinforcement learning. IEEE Robotics and Automation Letters, 2(4):2096–2103, 2017. doi: 10.1109/LRA.2017.2720851.
- Benjamin G Katz. A low cost modular actuator for dynamic robots. PhD thesis, Massachusetts Institute of Technology, 2018.
- A walking motion imitation framework of a humanoid robot by human walking recognition from imu motion data. In 2009 9th IEEE-RAS International Conference on Humanoid Robots, pages 343–348. IEEE, 2009.
- Mujoco haptix: A virtual reality system for hand manipulation. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 657–663, 2015. doi: 10.1109/HUMANOIDS.2015.7363441.
- Shared-autonomy control for intuitive bimanual tele-manipulation. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pages 1–9. IEEE, 2018.
- Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In 2019 International Conference on Robotics and Automation (ICRA), pages 416–422. IEEE, 2019.
- A mobile robot hand-arm teleoperation system by vision and imu. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10900–10906. IEEE, 2020.
- A dexterous hand-arm teleoperation system based on hand pose estimation and active vision. IEEE Transactions on Cybernetics, 2022.
- Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.
- Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14687–14697, 2021.
- Combining learning-based locomotion policy with model-based manipulation for legged mobile manipulators. IEEE Robotics and Automation Letters, 7(2):2377–2384, 2022. doi: 10.1109/LRA.2022.3143567.
- Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879–893. PMLR, 2018.
- A continuous teleoperation subspace with empirical and algorithmic mapping algorithms for nonanthropomorphic hands. IEEE Transactions on Automation Science and Engineering, 19(1):373–386, 2020.
- Accelerating interactive human-like manipulation learning with gpu-based simulation and high-quality demonstrations. In 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids), pages 435–441. IEEE, 2022.
- Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
- Algorithms for inverse reinforcement learning. In ICML, 2000.
- Octo: An open-source generalist robot policy. https://octo-models.github.io, 2023.
- Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
- The surprising effectiveness of representation learning for visual imitation, 2021.
- Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. Touretzky, editor, NeurIPS, volume 1. Morgan-Kaufmann, 1988.
- From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. arXiv preprint arXiv:2204.12490, 2022.
- Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system. arXiv preprint arXiv:2307.04577, 2023.
- Real-world robot learning with masked visual pre-training, 2022. URL https://arxiv.org/abs/2210.03109.
- Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02532, 2023.
- Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
- Nimbro avatar: Interactive immersive telepresence with force-feedback telemanipulation. in 2021 ieee. In RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5312–5319.
- Behavior transformers: Cloning k𝑘kitalic_k modes with one stone. Advances in neural information processing systems, 35:22955–22968, 2022.
- On bringing robots home. arXiv preprint arXiv:2311.16098, 2023.
- ViNT: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning, 2023. URL https://arxiv.org/abs/2306.14846.
- Whole body teleoperation of a humanoid robot development of a simple master device using joysticks. Journal of the Robotics Society of Japan, 22(4):519–527, 2004.
- Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube, 2022.
- A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860, 2022.
- Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. RA-L, 2020.
- Robotic operating system. URL https://www.ros.org.
- Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422, 2023.
- Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint arXiv:2309.13037, 2023.
- A teleoperation interface for loco-manipulation control of mobile collaborative robotic assistant. IEEE Robotics and Automation Letters, 4(4):3593–3600, 2019.
- xArm Developer. xarm python sdk. https://github.com/xArm-Developer/xArm-Python-SDK.
- Mediapipe hands: On-device real-time hand tracking, 2020.
- Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search. In 2016 IEEE international conference on robotics and automation (ICRA), pages 528–535. IEEE, 2016.
- Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
- Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
- Combining marker-based mocap and rgb-d camera for acquiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation, pages 33–42, 2012.
- Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210.11339, 2022. doi: 10.48550/arXiv.2210.11339.