Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation (2403.07870v1)

Published 12 Mar 2024 in cs.RO

Abstract: Open-sourced, user-friendly tools form the bedrock of scientific advancement across disciplines. The widespread adoption of data-driven learning has led to remarkable progress in multi-fingered dexterity, bimanual manipulation, and applications ranging from logistics to home robotics. However, existing data collection platforms are often proprietary, costly, or tailored to specific robotic morphologies. We present OPEN TEACH, a new teleoperation system leveraging VR headsets to immerse users in mixed reality for intuitive robot control. Built on the affordable Meta Quest 3, which costs $500, OPEN TEACH enables real-time control of various robots, including multi-fingered hands and bimanual arms, through an easy-to-use app. Using natural hand gestures and movements, users can manipulate robots at up to 90Hz with smooth visual feedback and interface widgets offering closeup environment views. We demonstrate the versatility of OPEN TEACH across 38 tasks on different robots. A comprehensive user study indicates significant improvement in teleoperation capability over the AnyTeleop framework. Further experiments exhibit that the collected data is compatible with policy learning on 10 dexterous and contact-rich manipulation tasks. Currently supporting Franka, xArm, Jaco, and Allegro platforms, OPEN TEACH is fully open-sourced to promote broader adoption. Videos are available at https://open-teach.github.io/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
  2. Task-oriented hand motion retargeting for dexterous manipulation imitation. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  3. Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation. arXiv preprint arXiv:2203.13251, 2022.
  4. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962–5969. IEEE, 2023.
  5. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.01918, 2023.
  6. Discriminative and adaptive imitation in uni-manual and bi-manual tasks. Robotics and Autonomous Systems, 54(5):370–384, 2006.
  7. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
  8. A systematic review of commercial smart gloves: Current status and applications. Sensors, 2021. ISSN 1424-8220. doi: 10.3390/s21082667.
  9. Extreme parkour with legged robots. arXiv preprint arXiv:2309.14341, 2023.
  10. Iterative residual policy: for goal-conditioned dynamic manipulation of deformable objects. arXiv preprint arXiv:2203.00663, 2022.
  11. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
  12. Imitation learning from pixel observations for continuous control. 2021.
  13. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047, 2022.
  14. Low-cost exoskeletons for learning whole-arm manipulation in the wild. arXiv preprint arXiv:2309.14975, 2023.
  15. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024.
  16. Learning to fly by crashing. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3948–3955. IEEE, 2017.
  17. Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control. IEEE Transactions on Robotics, 2022.
  18. Openvr: Teleoperation for manipulation. arXiv preprint arXiv:2305.09765, 2023.
  19. Telerobotic control in virtual reality. In OCEANS 2019 MTS/IEEE SEATTLE, pages 1–8, 2019. doi: 10.23919/OCEANS40490.2019.8962616.
  20. See to touch: Learning tactile dexterity through visual incentives. arXiv preprint arXiv:2309.12300, 2023.
  21. Watch and match: Supercharging imitation with regularized optimal transport. arXiv preprint arXiv:2206.15469, 2022.
  22. Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497, 2023.
  23. Megatrack: Monochrome egocentric articulated hand-tracking for virtual reality. 2020.
  24. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164–9170, 2020. doi: 10.1109/ICRA40945.2020.9197124.
  25. The dlr bimanual haptic device with optimized workspace. In 2011 IEEE International Conference on Robotics and Automation, pages 3441–3442. IEEE, 2011.
  26. Control of a quadrotor with reinforcement learning. IEEE Robotics and Automation Letters, 2(4):2096–2103, 2017. doi: 10.1109/LRA.2017.2720851.
  27. Benjamin G Katz. A low cost modular actuator for dynamic robots. PhD thesis, Massachusetts Institute of Technology, 2018.
  28. A walking motion imitation framework of a humanoid robot by human walking recognition from imu motion data. In 2009 9th IEEE-RAS International Conference on Humanoid Robots, pages 343–348. IEEE, 2009.
  29. Mujoco haptix: A virtual reality system for hand manipulation. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 657–663, 2015. doi: 10.1109/HUMANOIDS.2015.7363441.
  30. Shared-autonomy control for intuitive bimanual tele-manipulation. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pages 1–9. IEEE, 2018.
  31. Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In 2019 International Conference on Robotics and Automation (ICRA), pages 416–422. IEEE, 2019.
  32. A mobile robot hand-arm teleoperation system by vision and imu. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10900–10906. IEEE, 2020.
  33. A dexterous hand-arm teleoperation system based on hand pose estimation and active vision. IEEE Transactions on Cybernetics, 2022.
  34. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.
  35. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14687–14697, 2021.
  36. Combining learning-based locomotion policy with model-based manipulation for legged mobile manipulators. IEEE Robotics and Automation Letters, 7(2):2377–2384, 2022. doi: 10.1109/LRA.2022.3143567.
  37. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879–893. PMLR, 2018.
  38. A continuous teleoperation subspace with empirical and algorithmic mapping algorithms for nonanthropomorphic hands. IEEE Transactions on Automation Science and Engineering, 19(1):373–386, 2020.
  39. Accelerating interactive human-like manipulation learning with gpu-based simulation and high-quality demonstrations. In 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids), pages 435–441. IEEE, 2022.
  40. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
  41. Algorithms for inverse reinforcement learning. In ICML, 2000.
  42. Octo: An open-source generalist robot policy. https://octo-models.github.io, 2023.
  43. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
  44. The surprising effectiveness of representation learning for visual imitation, 2021.
  45. Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. Touretzky, editor, NeurIPS, volume 1. Morgan-Kaufmann, 1988.
  46. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. arXiv preprint arXiv:2204.12490, 2022.
  47. Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system. arXiv preprint arXiv:2307.04577, 2023.
  48. Real-world robot learning with masked visual pre-training, 2022. URL https://arxiv.org/abs/2210.03109.
  49. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02532, 2023.
  50. Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
  51. Nimbro avatar: Interactive immersive telepresence with force-feedback telemanipulation. in 2021 ieee. In RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5312–5319.
  52. Behavior transformers: Cloning k𝑘kitalic_k modes with one stone. Advances in neural information processing systems, 35:22955–22968, 2022.
  53. On bringing robots home. arXiv preprint arXiv:2311.16098, 2023.
  54. ViNT: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning, 2023. URL https://arxiv.org/abs/2306.14846.
  55. Whole body teleoperation of a humanoid robot development of a simple master device using joysticks. Journal of the Robotics Society of Japan, 22(4):519–527, 2004.
  56. Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube, 2022.
  57. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860, 2022.
  58. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. RA-L, 2020.
  59. Robotic operating system. URL https://www.ros.org.
  60. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422, 2023.
  61. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint arXiv:2309.13037, 2023.
  62. A teleoperation interface for loco-manipulation control of mobile collaborative robotic assistant. IEEE Robotics and Automation Letters, 4(4):3593–3600, 2019.
  63. xArm Developer. xarm python sdk. https://github.com/xArm-Developer/xArm-Python-SDK.
  64. Mediapipe hands: On-device real-time hand tracking, 2020.
  65. Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search. In 2016 IEEE international conference on robotics and automation (ICRA), pages 528–535. IEEE, 2016.
  66. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
  67. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
  68. Combining marker-based mocap and rgb-d camera for acquiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation, pages 33–42, 2012.
  69. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210.11339, 2022. doi: 10.48550/arXiv.2210.11339.
Citations (23)
View on Semantic Scholar

Summary

  • The paper presents a unified teleoperation framework that supports diverse robot types with plug-and-play, calibration-free VR control.
  • It integrates a cost-effective VR system delivering high-frequency (90Hz) visual feedback, achieving an average policy success rate of 86%.
  • User studies show that even novices can efficiently operate varied robotic systems, simplifying teleoperation data collection and collaboration.

Unveiling \method{}: A Unified Teleoperation Framework for Robotic Manipulation Across Varied Morphologies

Introduction to \method{}

The domain of robotic manipulation has witnessed substantial progression with the introduction of data-driven learning techniques, fostering enhancements in multi-fingered dexterity and bimanual manipulation. An emerging challenge within this field is the creation of versatile and accessible platforms for teleoperation data collection that can adapt across diverse robot morphologies without necessitating hefty investments or specialized setups. Addressing this, we introduce \method{}, a novel teleoperation system embedding VR technology to offer a seamless, calibration-free teleoperation experience over a broad spectrum of robotic configurations including single and bimanual arms, mobile manipulators, and multi-fingered hands.

Key Features and Capabilities

\method{} is distinguished by its plug-and-play nature, extensive compatibility across robot types, and the implementation of a cost-effective VR headset for an immersive teleoperation experience. Here are its core attributes:

  • Unified Teleoperation Across Robots: Unlike existing systems that are limited by cost, propriety, or specificity to robot types, \method{} introduces a highly versatile teleoperation framework. It demonstrates compatibility with a range of robots like Franka, xArm, Jaco for arms, and Allegro and Hello Stretch for hands and mobile manipulation.
  • High-Frequency Control and Visual Feedback: Leveraging the Meta Quest 3 VR headset, \method{} enables users to teleoperate robots with up to 90Hz of visual feedback. This supports nuanced control and instant correction capabilities essential for executing intricate and long-horizon tasks effectively.
  • User-Friendly and Calibration-Free Setup: Engineered for ease of use, \method{}'s setup bypasses the need for labor-intensive calibration processes, providing a straightforward pathway for users to engage in robot teleoperation.
  • Open-Source Contribution: In line with fostering community growth and collaboration, the entirety of \method{}—spanning the VR application, interface, and robot controllers— is made available open-source. We invite contributions to expand its applicability further.

Experimental Validation and Impact

A series of experiments involving 38 tasks across variations of robots underline \method{}'s versatility. This system not only enables users to perform a wide array of tasks but also facilitates the collection of high-quality data, demonstrating an average policy success rate of 86\% across tasks. Such achievements mark a significant stride towards simplifying data collection for robot learning applications.

Furthermore, a comprehensive user paper underscores \method{}'s intuitive design. Participants, including those with no prior teleoperation experience, were able to effectively control robots and complete designated tasks, showcasing the system's ease-of-use and potential to reduce training time for teleoperators.

Future Directions and Community Invitation

While \method{} marks a notable advancement in robotic teleoperation, it is not devoid of limitations. The precision of control is contingent upon the accuracy of the VR headset's hand pose estimation, which can sometimes be compromised due to occlusions. Future work aimed at refining hand tracking technologies could further elevate the performance and user experience of teleoperation systems like \method{}.

We envision \method{} as a foundation for more accessible robot teleoperation and encourage the community to engage with our open-source repository. Whether through developing interfaces for additional robots, enhancing the VR experience, or integrating advanced control strategies, there is ample room for extending \method{}'s capabilities. Contributions are warmly welcomed, and we are committed to supporting their integration and dissemination within the broader robotics and AI community.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown