Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Opening Cabinets and Drawers in the Real World using a Commodity Mobile Manipulator (2402.17767v1)

Published 27 Feb 2024 in cs.RO, cs.AI, cs.CV, and cs.LG

Abstract: Pulling open cabinets and drawers presents many difficult technical challenges in perception (inferring articulation parameters for objects from onboard sensors), planning (producing motion plans that conform to tight task constraints), and control (making and maintaining contact while applying forces on the environment). In this work, we build an end-to-end system that enables a commodity mobile manipulator (Stretch RE2) to pull open cabinets and drawers in diverse previously unseen real world environments. We conduct 4 days of real world testing of this system spanning 31 different objects from across 13 different real world environments. Our system achieves a success rate of 61% on opening novel cabinets and drawers in unseen environments zero-shot. An analysis of the failure modes suggests that errors in perception are the most significant challenge for our system. We will open source code and models for others to replicate and build upon our system.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691, 2022.
  2. Demonstrating mobile manipulation in the wild: A metrics-driven approach. In Robotics: Science and Systems XIX, RSS2023. Robotics: Science and Systems Foundation, July 2023. doi: 10.15607/rss.2023.xix.055. URL http://dx.doi.org/10.15607/RSS.2023.XIX.055.
  3. Dmitry Berenson. Obeying Constraints During Motion Planning, pages 1–32. Springer Netherlands, 2018.
  4. Task space regions: A framework for pose-constrained manipulation planning. IJRR, 30(12):1435–1460, 2011.
  5. Whole-body motion planning for manipulation of articulated objects. In ICRA, pages 1656–1662, 2013. ISBN 9781467356411. doi: 10.1109/ICRA.2013.6630792.
  6. Planning for autonomous door opening with a mobile manipulator. In 2010 IEEE International Conference on Robotics and Automation, pages 1799–1806. IEEE, 2010.
  7. Manipulathor: A framework for visual object manipulation. In CVPR, pages 4497–4506, 2021.
  8. Real-time motion planning of legged robots: A model predictive control approach. In ICHR, pages 577–584, 2017.
  9. Deep whole-body control: Learning a unified policy for manipulation and locomotion. In Conference on Robot Learning (CoRL), 2022.
  10. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024.
  11. Threedworld: A platform for interactive multi-modal physical simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
  12. Predicting motion plans for articulating everyday objects. In International Conference on Robotics and Automation (ICRA). IEEE, 2023.
  13. Mask r-cnn. In ICCV, pages 2961–2969, 2017.
  14. Pulling open doors and drawers: Coordinating an omni-directional base and a compliant arm with equilibrium point control. In 2010 IEEE International Conference on Robotics and Automation, pages 1807–1814. IEEE, 2010.
  15. Opd: Single-view 3d openable part detection. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 410–426, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19842-7.
  16. An adaptive control approach for opening doors and drawers under uncertainties. IEEE Transactions on Robotics, 32(1):161–175, 2016.
  17. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE transactions on Robotics and Automation, 12(4):566–580, 1996.
  18. Sampling-based methods for motion planning with constraints. Annual review of control, robotics, and autonomous systems, 1:159–185, 2018.
  19. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
  20. RRT-connect: An efficient approach to single-query path planning. In ICRA, 2000.
  21. Paris: Part-level reconstruction and motion analysis for articulated objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 352–363, October 2023.
  22. Autonomous door opening and plugging in with a personal robot. In 2010 IEEE International Conference on Robotics and Automation, pages 729–736. IEEE, 2010.
  23. Articulated object interaction in unknown scenes with whole-body mobile manipulation. In IROS, 2022.
  24. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6813–6823, October 2021.
  25. Where2explore: Few-shot affordance learning for unseen novel categories of articulated objects. In Advances in Neural Information Processing Systems, 2023.
  26. Perceptive model predictive control for continuous mobile manipulation. IEEE RA-L, pages 6177–6184, 2020.
  27. High-level control of a mobile manipulator for door opening. In Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000)(Cat. No. 00CH37113), volume 3, pages 2333–2338. IEEE, 2000.
  28. Understanding 3d object interaction from a single image. arXiv preprint arXiv:2305.09664, 2023.
  29. Habitat-matterport 3d dataset (HM3D): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=-v4OuqNs5P.
  30. A generalized framework for opening doors and drawers in kitchen environments. In ICRA, pages 3852–3858, 2012. doi: 10.1109/ICRA.2012.6224929.
  31. Motion planning with sequential convex optimization and convex collision checking. The International Journal of Robotics Research, 33(9):1251–1270, 2014.
  32. On bringing robots home, 2023.
  33. A unified mpc framework for whole-body dynamic locomotion and manipulation. IEEE RA-L, pages 4688–4695, 2021.
  34. Versatile multicontact planning and control for legged loco-manipulation. Science Robotics, 8(81), August 2023. ISSN 2470-9476. doi: 10.1126/scirobotics.adg5014. URL http://dx.doi.org/10.1126/scirobotics.adg5014.
  35. Opdmulti: Openable part detection for multiple objects, 2023.
  36. Robot placement based on reachability inversion. In ICRA, pages 1970–1975, 2013. doi: 10.1109/ICRA.2013.6630839.
  37. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=iEx3PiooLy.
  38. Sapien: A simulated part-based interactive environment. In CVPR, pages 11097–11107, 2020.
  39. Harmonic mobile manipulation. arXiv preprint arXiv:2312.06639, 2023.
  40. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv:2306.11565, 2023.
  41. Chomp: Covariant hamiltonian optimization for motion planning. The International Journal of Robotics Research, 32(9-10):1164–1193, 2013.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets