Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HomeRobot: Open-Vocabulary Mobile Manipulation (2306.11565v2)

Published 20 Jun 2023 in cs.RO, cs.AI, and cs.CV

Abstract: HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks. Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. This is a foundational challenge for robots to be useful assistants in human environments, because it involves tackling sub-problems from across robotics: perception, language understanding, navigation, and manipulation are all essential to OVMM. In addition, integration of the solutions to these sub-problems poses its own substantial challenges. To drive research in this area, we introduce the HomeRobot OVMM benchmark, where an agent navigates household environments to grasp novel objects and place them on target receptacles. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch to encourage replication of real-world experiments across labs. We implement both reinforcement learning and heuristic (model-based) baselines and show evidence of sim-to-real transfer. Our baselines achieve a 20% success rate in the real world; our experiments identify ways future research work improve performance. See videos on our website: https://ovmm.github.io/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (104)
  1. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv, 2020.
  2. Navigating to objects in the real world. arXiv, 2022.
  3. Robocup@ home: Scientific competition and benchmarking for domestic service robots. Interaction Studies, 2009.
  4. Towards autonomous robotic butlers: Lessons learned with the pr2. In ICRA, 2011.
  5. Experiences with an interactive museum tour-guide robot. Artificial intelligence, 1999.
  6. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
  7. Virtualhome: Simulating household activities via programs. In CVPR, 2018.
  8. Integrated task and motion planning. Annual Review of Control, Robotics, and Autonomous Systems, 4:265–293, 2021.
  9. Open-vocabulary queryable scene representations for real world planning. arXiv, 2022.
  10. Open-world object manipulation using pre-trained vision-language models. arXiv, 2023.
  11. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv, 2022.
  12. Usa-net: Unified semantic and affordance representations for robot memory. arXiv, 2023.
  13. Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances. In ICRA, 2022.
  14. Learning transferable visual models from natural language supervision. In ICML, 2021.
  15. Conceptfusion: Open-set multimodal 3d mapping. arXiv, 2023.
  16. Beyond the nav-graph: Vision and language navigation in continuous environments. In European Conference on Computer Vision (ECCV), 2020.
  17. Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects. arXiv, 2022.
  18. Palm-e: An embodied multimodal language model. arXiv, 2023.
  19. Habitat Synthetic Scenes Dataset: An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation. arXiv, 2023.
  20. Habitat: A Platform for Embodied AI Research. ICCV, 2019.
  21. Habitat 2.0: Training home assistants to rearrange their habitat. In NeurIPS, 2021.
  22. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. In ICRA, 2022.
  23. Homerobot open vocab mobile manipulation challenge 2023. https://aihabitat.org/challenge/2023_homerobot_ovmm/, 2023.
  24. Spatial-language attention policies for efficient robot learning. arXiv, 2023.
  25. Navigating to objects specified by images. arXiv, 2023.
  26. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022.
  27. Visual room rearrangement. In CVPR, 2021.
  28. Habitat challenge 2023. https://aihabitat.org/challenge/2023/, 2023.
  29. Threedworld: A platform for interactive multi-modal physical simulation. NeurIPS Datasets and Benchmarks Track, 2021.
  30. Procthor: Large-scale embodied ai using procedural generation. In NeurIPS, 2022.
  31. RoboTHOR: An Open Simulation-to-Real Embodied AI Platform. In CVPR, 2020.
  32. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In CoRL, 2023.
  33. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In NeurIPS Datasets and Benchmarks Track, 2021.
  34. The darpa robotics challenge finals: Results and perspectives. The DARPA Robotics Challenge Finals: Humanoid Robots To The Rescue, 2018.
  35. Unmanned vehicles come of age: The darpa grand challenge. Computer, 2006.
  36. The DARPA urban challenge: autonomous vehicles in city traffic. Springer Berlin, Heidelberg, 2009.
  37. Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 2016.
  38. The darpa lagr program: Goals, challenges, methodology, and phase i results. Journal of Field Robotics, 2006.
  39. M. Müller and V. Koltun. Openbot: Turning smartphones into robots. In ICRA, 2021.
  40. Stanford doggo: An open-source, quasi-direct-drive quadruped. In ICRA, 2019.
  41. An open torque-controlled modular robot architecture for legged locomotion research. IEEE Robotics and Automation Letters, 2019.
  42. Replab: A reproducible low-cost arm benchmark platform for robotic learning. arXiv, 2019.
  43. Quasi-direct drive for low-cost compliant robotic manipulation. In ICRA, 2019.
  44. Trifinger: An open-source robot for learning dexterity. In CoRL, 2020.
  45. ROBEL: RObotics BEnchmarks for Learning with low-cost robots. In CoRL, 2019.
  46. Pyrobot: An open-source robotics framework for research and benchmarking. arXiv, 2019.
  47. Duckietown: An open, inexpensive and flexible platform for autonomy education and research. In ICRA, 2017.
  48. The ycb object and model set: Towards common benchmarks for manipulation research. In ICRA, 2015.
  49. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robotics and Automation Letters, 2020.
  50. Benchmarking robot manipulation with the rubik’s cube. IEEE Robotics and Automation Letters, 2020.
  51. Rb2: Robotic manipulation benchmarking with a twist. arXiv, 2022.
  52. Train offline, test online: A real robot learning benchmark. arXiv, 2022.
  53. Benchmarking protocols for evaluating small parts robotic assembly systems. IEEE Robotics and Automation Letters, 2020.
  54. Benchmarking off-the-shelf solutions to robotic assembly tasks. In IROS, 2021.
  55. Are we making real progress in simulated environments? measuring the sim2real gap in embodied visual navigation. arXiv, 2019.
  56. AI2-THOR: an interactive 3d environment for visual AI. arXiv, 2017.
  57. Abo: Dataset and benchmarks for real-world 3d object understanding. In CVPR, 2022.
  58. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022.
  59. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2019.
  60. Evaluating continual learning on a home robot, 2023.
  61. Tidybot: Personalized robot assistance with large language models. arXiv, 2023.
  62. Stereo magnification: Learning view synthesis using multiplane images. SIGGRAPH, 2018.
  63. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In ICRA, 2021.
  64. 6-dof grasping for target-driven object manipulation in clutter. In ICRA, 2020.
  65. Graspnet-1billion: A large-scale benchmark for general object grasping. In CVPR, 2020.
  66. Online replanning in belief space for partially observable task and motion problems. In ICRA, 2020.
  67. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  68. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
  69. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.
  70. SocialIQA: Commonsense reasoning about social interactions. In EMNLP, 2019.
  71. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  72. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, 2019.
  73. Microsoft coco: Common objects in context. In ECCV, 2014.
  74. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
  75. L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA, 2016.
  76. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. IJRR, 2018.
  77. Do as i can and not as i say: Grounding language in robotic affordances. In CoRL, 2022.
  78. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In CoRL, 2018.
  79. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In CoRL, 2018.
  80. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv, 2017.
  81. Robonet: Large-scale multi-robot learning. arXiv, 2019.
  82. Robot learning in homes: Improving generalization and reducing dataset bias. In NeurIPS, 2018.
  83. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2017.
  84. H. Team. Habitat CVPR challenge, 2019. URL https://aihabitat.org/challenge/2019/.
  85. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 2020.
  86. Soundspaces: Audio-visual navigation in 3d environments. In ECCV, 2020.
  87. Manipulathor: A framework for visual object manipulation. In CVPR, 2021.
  88. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2019.
  89. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
  90. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In EMNLP, 2020.
  91. TEACh: Task-driven embodied agents that chat. In AAAI, 2022.
  92. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 2022.
  93. Habitat rearrangement challenge. https://aihabitat.org/challenge/2022_rearrange, 2022.
  94. Simulation of parallel-jaw grasping using incremental potential contact models. In ICRA, 2022.
  95. The robotic vision challenges. https://nikosuenderhauf.github.io/roboticvisionchallenges/cvpr2022, 2022.
  96. 6-dof graspnet: Variational grasp generation for object manipulation. In ICCV, 2019.
  97. Predicting stable configurations for semantic placement of novel objects. In CoRL, 2022.
  98. A flexible and scalable slam system with full 3d motion estimation. In Proc. IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR). IEEE, November 2011.
  99. Rrt-connect: An efficient approach to single-query path planning. In ICRA, 2000.
  100. Object goal navigation using goal-oriented semantic exploration. In NeurIPS, 2020.
  101. B. Yamauchi. A frontier-based approach for autonomous exploration. In IEEE International Symposium on Computational Intelligence in Robotics and Automation, 1997.
  102. J. A. Sethian. Fast marching methods. SIAM review, 1999.
  103. Learning to explore using active neural mapping. ICLR, 2020.
  104. Ros: an open-source robot operating system. In ICRA Workshop on Open Source Software, 2009.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (18)
  1. Sriram Yenamandra (9 papers)
  2. Arun Ramachandran (4 papers)
  3. Karmesh Yadav (16 papers)
  4. Austin Wang (15 papers)
  5. Mukul Khanna (8 papers)
  6. Theophile Gervet (13 papers)
  7. Tsung-Yen Yang (13 papers)
  8. Vidhi Jain (12 papers)
  9. Alexander William Clegg (3 papers)
  10. John Turner (7 papers)
  11. Zsolt Kira (110 papers)
  12. Manolis Savva (64 papers)
  13. Angel Chang (5 papers)
  14. Devendra Singh Chaplot (37 papers)
  15. Dhruv Batra (160 papers)
  16. Roozbeh Mottaghi (66 papers)
  17. Yonatan Bisk (91 papers)
  18. Chris Paxton (59 papers)
Citations (61)
View on Semantic Scholar

Summary

Open-Vocabulary Mobile Manipulation: A Comprehensive Exploration

The paper "HomeRobot: Open-Vocabulary Mobile Manipulation" presents a detailed approach to tackling significant challenges in robotics, particularly in the area of Open-Vocabulary Mobile Manipulation (OVMM). This research addresses the integration of perception, language understanding, navigation, and manipulation, all essential sub-components for creating effective household robotic assistants. This paper introduces the HomeRobot OVMM benchmark, a platform designed to evaluate mobile manipulation in both simulated and real-world environments.

Benchmark Design and Components

The HomeRobot OVMM benchmark has two primary elements: a simulation component and a real-world component. The simulation utilizes an extensive dataset, comprising 200 human-authored 3D scenes within AI Habitat, to present diverse multi-room environments populated with a wide range of objects. This environment is used to create multi-room OVMM challenges, helping bridge sim-to-real transfer barriers.

The real-world component employs the Hello Robot Stretch platform equipped with a software stack to enhance reproducibility across labs. This component is designed with sim-to-real transfer in mind, showing baselines achieving a 20% success rate in real-world tests.

Methodology and Baseline Implementations

The paper provides both heuristic and reinforcement learning (RL) methods as baseline agents. The heuristic approach uses a motion planner integrated with a vision-based object detector, DETIC. This method excels in long-horizon navigation tasks. Conversely, the RL approach demonstrates superior navigation efficiency when visible objects are present. The integration tests reveal a significant performance drop when switching from ground-truth perception to DETIC-based perception, underlining the importance of integrated learning systems for improving home assistant functionality.

Numerical Results and Task Performance

Significant experimental results detail success rates across various sub-tasks within the OVMM framework. The baselines demonstrate potential but also highlight the challenges posed by perception inaccuracies, particularly with DETIC predictions. The RL methods surpassed heuristic methods for specific tasks, yet all systems exhibited marked performance declines when transitioning from simulation to real-world conditions.

Implications and Future Directions

The implications of this research for practical and theoretical advancements in home robotics are profound. By standardizing OVMM as a benchmark, this work catalyzes further research on multi-task integrated systems. The paper suggests that utilizing large pretrained vision-LLMs could be crucial for enhanced OVMM task performance, combined with tailored models for specific robotics tasks.

Looking forward, expanding the complexities of tasks with more intricate language and multi-step commands, alongside deploying end-to-end learning models, is likely to be a pivotal aspect of future research. This aligns the pursuit of robotics towards more human-like interaction and assistance capabilities in real-world environments.

In conclusion, this paper contributes significantly to the discourse on robotics benchmarks and embodies a step towards more autonomous, efficient home robotics systems. The HomeRobot platform serves as a cornerstone for future explorations into open-vocabulary tasks, fostering a deeper understanding of how robots can adapt to and function within complex human environments.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown