Papers
Topics
Authors
Recent
Search
2000 character limit reached

On Bringing Robots Home

Published 27 Nov 2023 in cs.RO, cs.AI, cs.CV, and cs.LG | (2311.16098v1)

Abstract: Throughout history, we have successfully integrated various machines into our homes. Dishwashers, laundry machines, stand mixers, and robot vacuums are a few recent examples. However, these machines excel at performing only a single task effectively. The concept of a "generalist machine" in homes - a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective - has long been a goal in robotics that has been steadily pursued for decades. In this work, we initiate a large-scale effort towards this goal by introducing Dobb-E, an affordable yet versatile general-purpose system for learning robotic manipulation within household settings. Dobb-E can learn a new task with only five minutes of a user showing it how to do it, thanks to a demonstration collection tool ("The Stick") we built out of cheap parts and iPhones. We use the Stick to collect 13 hours of data in 22 homes of New York City, and train Home Pretrained Representations (HPR). Then, in a novel home environment, with five minutes of demonstrations and fifteen minutes of adapting the HPR model, we show that Dobb-E can reliably solve the task on the Stretch, a mobile robot readily available on the market. Across roughly 30 days of experimentation in homes of New York City and surrounding areas, we test our system in 10 homes, with a total of 109 tasks in different environments, and finally achieve a success rate of 81%. Beyond success percentages, our experiments reveal a plethora of unique challenges absent or ignored in lab robotics. These range from effects of strong shadows, to variable demonstration quality by non-expert users. With the hope of accelerating research on home robots, and eventually seeing robot butlers in every home, we open-source Dobb-E software stack and models, our data, and our hardware designs at https://dobb-e.com

Definition Search Book Streamline Icon: https://streamlinehq.com
References (99)
  1. Steve Carper. Robots in American popular culture. McFarland, 2019.
  2. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in Neural Information Processing Systems, 31:9094–9104, 2018.
  3. Navigating to objects in the real world. Science Robotics, 8(79):eadf6991, 2023.
  4. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. In 2022 International Conference on Robotics and Automation (ICRA), pages 3150–3157. IEEE, 2022.
  5. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  6. Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
  7. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022.
  8. R3m: A universal visual representation for robot manipulation. In CoRL, 2022.
  9. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.18240, 2023.
  10. Robot learning from demonstration. In ICML, volume 97, pages 12–20. Citeseer, 1997.
  11. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879–893. PMLR, 2018.
  12. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962–5969. IEEE, 2023.
  13. Low-cost exoskeletons for learning whole-arm manipulation in the wild. arXiv preprint arXiv:2309.14975, 2023.
  14. De vito: A dual-arm, high degree-of-freedom, lightweight, inexpensive, passive upper-limb exoskeleton for robot teleoperation. In Towards Autonomous Robotic Systems: 20th Annual Conference, TAROS 2019, London, UK, July 3–5, 2019, Proceedings, Part I 20, pages 78–89. Springer, 2019.
  15. A wearable upper limb exoskeleton for intuitive teleoperation of anthropomorphic manipulators. Machines, 11(4):441, 2023.
  16. Bilateral humanoid teleoperation system using whole-body exoskeleton cockpit tablis. IEEE Robotics and Automation Letters, 5(4):6419–6426, 2020.
  17. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3):4978–4985, 2020.
  18. Visual imitation made easy. arXiv e-prints, pages arXiv–2008, 2020.
  19. The surprising effectiveness of representation learning for visual imitation, 2021.
  20. Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation. arXiv preprint arXiv:2203.13251, 2022.
  21. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. arXiv preprint arXiv:2303.12076, 2023.
  22. See to touch: Learning tactile dexterity through visual incentives. arXiv preprint arXiv:2309.12300, 2023.
  23. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
  24. Playful interactions for representation learning. arXiv preprint arXiv:2107.09046, 2021.
  25. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
  26. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference on Robot Learning, 2021.
  27. RH20T: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
  28. Scaling robot supervision to hundreds of hours with RoboTurk: Robotic manipulation dataset through human reasoning and dexterity. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1048–1055. IEEE, 2019.
  29. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In Robotics: Science and Systems (RSS) XVIII, 2022.
  30. Bridgedata v2: A dataset for robot learning at scale, 2023.
  31. Efficient grasping from RGBD images: Learning using a new rectangle representation. In 2011 IEEE International conference on robotics and automation, pages 3304–3311. IEEE, 2011.
  32. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3406–3413, 2015.
  33. Leveraging big data for grasp planning. In ICRA, pages 4304–4311, 2015.
  34. Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Robotics: Science and Systems (RSS), 2017.
  35. Jacquard: A large scale dataset for robotic grasp detection. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3511–3516. IEEE, 2018.
  36. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421–436, 2018.
  37. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293, 2018.
  38. Contactdb: Analyzing and predicting grasp contact via thermal imaging, 04 2019.
  39. Graspnet-1billion: a large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11444–11453, 2020.
  40. ACRONYM: A large-scale grasp dataset based on simulation. In 2021 IEEE Int. Conf. on Robotics and Automation, ICRA, 2020.
  41. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In ICRA, pages 4243–4250, 2018.
  42. Fanuc manipulation: A dataset for learning-based manipulation with fanuc mate 200iD robot. https://sites.google.com/berkeley.edu/fanuc-manipulation, 2023.
  43. More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 30–37. IEEE, 2016.
  44. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2786–2793. IEEE, 2017.
  45. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018.
  46. RoboNet: Large-scale multi-robot learning. In Conference on Robot Learning (CoRL), volume 100, pages 885–897. PMLR, 2019.
  47. MT-Opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212, 2021.
  48. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298, 2021.
  49. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.
  50. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
  51. RoboAgent: Towards sample efficient robot manipulation with semantic augmentations and action chunking. arxiv, 2023.
  52. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023.
  53. Multiple interactions made easy (MIME): Large scale demonstrations data for imitation. In Conference on robot learning, pages 906–915. PMLR, 2018.
  54. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv:1909.12200, 2019.
  55. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint arXiv:2309.01918, 2023.
  56. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018.
  57. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
  58. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023.
  59. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009.
  60. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, pages 4171–4186, 2018.
  61. Language models are few-shot learners, 2020.
  62. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  63. Inverse dynamics pretraining learns good representations for multitask imitation. arXiv preprint arXiv:2305.16985, 2023.
  64. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, 2022.
  65. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022.
  66. Language-driven representation learning for robotics. Robotics: Science and Systems (RSS), 2023.
  67. EC2: Emergent communication for embodied control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6704–6714, 2023.
  68. Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778–13790, 2023.
  69. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials. arXiv preprint arXiv:2210.05178, 2022.
  70. Deep rl at scale: Sorting waste in office buildings with a fleet of mobile manipulators. arXiv preprint arXiv:2305.03270, 2023.
  71. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
  72. Joseph L Jones. Robots at the tipping point: the road to irobot roomba. IEEE Robotics & Automation Magazine, 13(1):76–78, 2006.
  73. P Dempsey. Reviews-consumer technology. the teardown-amazon astro consumer robot. Engineering & Technology, 18(2):70–71, 2023.
  74. Autonomously learning to visually detect where manipulation will succeed. Autonomous Robots, 36:137–152, 2014.
  75. Data-driven thermal recognition of contact with people and objects. In 2016 IEEE Haptics Symposium (HAPTICS), pages 297–304. IEEE, 2016.
  76. Multimodal tactile perception of objects in a real home. IEEE Robotics and Automation Letters, 3(3):2523–2530, 2018.
  77. Human-to-robot imitation in the wild. Robotics: Science and Systems (RSS), 2022.
  78. Viking: Vision-based kilometer-scale navigation with geographic hints. arXiv preprint arXiv:2202.11271, 2022.
  79. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. arXiv preprint arXiv:2109.10892, 2021.
  80. The complex structure of simple devices: A survey of trajectories and forces that open doors and drawers. In 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics, pages 184–190. IEEE, 2010.
  81. Improving robot manipulation with data-driven object-centric models of everyday forces. Autonomous Robots, 35:143–159, 2013.
  82. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
  83. Behavior transformers: Cloning k𝑘kitalic_k modes with one stone. Advances in neural information processing systems, 35:22955–22968, 2022.
  84. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  85. Matthias Adorjan. OpenSfM: A Collaborative Structure-From-Motion System. PhD thesis, Wien, 2016.
  86. Forcesight: Text-guided mobile manipulation with visual-force goals. arXiv preprint arXiv:2309.12312, 2023.
  87. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning, pages 32–43. PMLR, 2023.
  88. Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497, 2023.
  89. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663, 2022.
  90. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729–19739, 2023.
  91. Language embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning, 2023.
  92. D33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT fields: Dynamic 3d descriptor fields for zero-shot generalizable robotic manipulation. arXiv preprint arXiv:2309.16118, 2023.
  93. Distilled feature fields enable few-shot language-guided manipulation. arXiv preprint arXiv:2308.07931, 2023.
  94. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023.
  95. Usa-net: Unified semantic and affordance representations for robot memory. arXiv preprint arXiv:2304.12164, 2023.
  96. Visual language maps for robot navigation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10608–10615. IEEE, 2023.
  97. Reskin: versatile, replaceable, lasting tactile skins. arXiv preprint arXiv:2111.00071, 2021.
  98. Visual pressure estimation and control for soft robotic grippers. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3628–3635. IEEE, 2022.
  99. Visual contact pressure estimation for grippers in the wild. arXiv preprint arXiv:2303.07344, 2023.
Citations (46)
View on Semantic Scholar

Summary

  • The paper introduces the Dobb-E system, which leverages imitation learning and affordable materials to enable versatile home robotics.
  • The research used demonstrations from 22 NYC homes to build the Home Pretrained Representations dataset for training the system.
  • The experiments achieved an 81% success rate over 109 tasks in 10 homes, highlighting practical success and outlining future challenges.

Introduction

In the evolution of domestic appliances, the integration of single-purpose machines like dishwashers and laundry machines into our homes has significantly enhanced convenience and efficiency. The next leap in domestic technology involves developing versatile, general-purpose systems capable of learning a wide array of tasks. This paper introduces an innovative approach called Dobb-E, a system designed to bring the concept of an adaptable, cost-effective domestic assistant robot to fruition.

Technical Framework

The foundation of Dobb-E is built upon several technical components. It all starts with "The Stick," an economically viable tool assembled from inexpensive materials and iPhones. This tool is pivotal in demonstration collection, allowing the robot to learn tasks by mimicking the actions shown by a user in just minutes. Collecting data from 22 homes in New York City using the Stick, the researchers developed a dataset called Home Pretrained Representations (HPR). This dataset is instrumental for training the system to perform household tasks with varying levels of complexity. Additionally, Dobb-E employs a simple imitation learning framework that, despite its simplicity, has proven effective in household environments.

Experimental Results

Through extensive trials in 10 homes over approximately 30 days, Dobb-E was deployed to execute 109 tasks. The overall success rate was an impressive 81%. This high rate of success indicates that Dobb-E can effectively learn and execute simple household tasks, even in environments that it has not previously encountered. The experiments also highlighted unique challenges, such as lighting variations and the limitations of non-expert users in demonstration quality.

Future Directions and Public Resources

Despite the successes, the paper acknowledges that there are obstacles to overcome before the widespread adoption of robots like Dobb-E in homes. These challenges include scaling up to long-horizon tasks, improving the robot's hardware for more demanding tasks, and developing memory capabilities for more complex sequences of actions. Encouragingly, the researchers have made the Dobb-E software stack, models, data, and hardware designs publicly accessible. They invite collaboration to progress the field of home robotics, underscoring the potential for this technology to evolve through shared knowledge and open-source resources.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 9 likes about this paper.

Sign Up for Free