Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects (2403.01694v3)

Published 4 Mar 2024 in cs.RO

Abstract: Integrating robots into human-centric environments such as homes, necessitates advanced manipulation skills as robotic devices will need to engage with articulated objects like doors and drawers. Key challenges in robotic manipulation of articulated objects are the unpredictability and diversity of these objects' internal structures, which render models based on object kinematics priors, both explicit and implicit, inadequate. Their reliability is significantly diminished by pre-interaction ambiguities, imperfect structural parameters, encounters with unknown objects, and unforeseen disturbances. Here, we present a prior-free strategy, Tac-Man, focusing on maintaining stable robot-object contact during manipulation. Without relying on object priors, Tac-Man leverages tactile feedback to enable robots to proficiently handle a variety of articulated objects, including those with complex joints, even when influenced by unexpected disturbances. Demonstrated in both real-world experiments and extensive simulations, it consistently achieves near-perfect success in dynamic and varied settings, outperforming existing methods. Our results indicate that tactile sensing alone suffices for managing diverse articulated objects, offering greater robustness and generalization than prior-based approaches. This underscores the importance of detailed contact modeling in complex manipulation tasks, especially with articulated objects. Advancements in tactile-informed approaches significantly expand the scope of robotic applications in human-centric environments, particularly where accurate models are difficult to obtain. See additional material at https://tacman-aom.github.io.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. E. Krotkov, D. Hackett, L. Jackel, M. Perschbacher, J. Pippine, J. Strauss, G. Pratt, and C. Orlowski, “The darpa robotics challenge finals: Results and perspectives,” Journal of Field Robotics, vol. 34, no. 2, pp. 229–240, 2017.
  2. A. Billard and D. Kragic, “Trends and challenges in robot manipulation,” Science, vol. 364, no. 6446, p. eaat8414, 2019.
  3. S. Chitta, B. Cohen, and M. Likhachev, “Planning for autonomous door opening with a mobile manipulator,” in International Conference on Robotics and Automation (ICRA), 2010.
  4. F. Burget, A. Hornung, and M. Bennewitz, “Whole-body motion planning for manipulation of articulated objects,” in International Conference on Robotics and Automation (ICRA), 2013.
  5. Z. Jiao, Z. Zhang, X. Jiang, D. Han, S.-C. Zhu, Y. Zhu, and H. Liu, “Consolidating kinematic models to promote coordinated mobile manipulations,” in International Conference on Intelligent Robots and Systems (IROS), 2021.
  6. Y. Karayiannidis, C. Smith, F. E. Vina, P. Ogren, and D. Kragic, ““open sesame!” adaptive force/velocity control for opening unknown doors,” in International Conference on Intelligent Robots and Systems (IROS), 2012.
  7. K. Hausman, S. Niekum, S. Osentoski, and G. S. Sukhatme, “Active articulation model estimation through interactive perception,” in International Conference on Robotics and Automation (ICRA), 2015.
  8. Y. Karayiannidis, C. Smith, F. E. V. Barrientos, P. Ögren, and D. Kragic, “An adaptive control approach for opening doors and drawers under uncertainties,” Transactions on Robotics (T-RO), vol. 32, no. 1, pp. 161–175, 2016.
  9. R. Hu, W. Li, O. Van Kaick, A. Shamir, H. Zhang, and H. Huang, “Learning to predict part mobility from a single static snapshot,” ACM Transactions on Graphics (TOG), vol. 36, no. 6, pp. 1–13, 2017.
  10. B. Abbatematteo, S. Tellex, and G. Konidaris, “Learning to generalize kinematic models to novel objects,” in Conference on Robot Learning (CoRL), 2019.
  11. X. Li, H. Wang, L. Yi, L. J. Guibas, A. L. Abbott, and S. Song, “Category-level articulated object pose estimation,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  12. C. Moses, M. Noseworthy, L. P. Kaelbling, T. Lozano-Pérez, and N. Roy, “Visual prediction of priors for articulated object interaction,” in International Conference on Robotics and Automation (ICRA), 2020.
  13. V. Zeng, T. E. Lee, J. Liang, and O. Kroemer, “Visual identification of articulated object parts,” in International Conference on Intelligent Robots and Systems (IROS), 2021.
  14. M. Mittal, D. Hoeller, F. Farshidian, M. Hutter, and A. Garg, “Articulated object interaction in unknown scenes with whole-body mobile manipulation,” in International Conference on Intelligent Robots and Systems (IROS), 2022.
  15. M. Han, Z. Zhang, Z. Jiao, X. Xie, Y. Zhu, S.-C. Zhu, and H. Liu, “Scene reconstruction with functional objects for robot autonomy,” International Journal of Computer Vision (IJCV), vol. 130, no. 12, pp. 2940–2961, 2022.
  16. J. Lv, Q. Yu, L. Shao, W. Liu, W. Xu, and C. Lu, “Sagci-system: Towards sample-efficient, generalizable, compositional, and incremental robot learning,” in International Conference on Robotics and Automation (ICRA), 2022.
  17. Z. Jiang, C.-C. Hsu, and Y. Zhu, “Ditto: Building digital twins of articulated objects from interaction,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  18. B. Eisner, H. Zhang, and D. Held, “Flowbot3d: Learning 3D articulation flow to manipulate articulated objects,” in Robotics: Science and Systems (RSS), 2022.
  19. R. Martín-Martín and O. Brock, “Coupled recursive estimation for online interactive perception of articulated objects,” International Journal of Robotics Research (IJRR), vol. 41, no. 8, pp. 741–777, 2022.
  20. Z. Zhang, L. Zhang, Z. Wang, Z. Jiao, M. Han, Y. Zhu, S.-C. Zhu, and H. Liu, “Part-level scene reconstruction affords robot interaction,” in International Conference on Intelligent Robots and Systems (IROS), 2023.
  21. P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, “Learning and generalization of motor skills by learning from demonstration,” in International Conference on Robotics and Automation (ICRA), 2009.
  22. T. Welschehold, C. Dornhege, and W. Burgard, “Learning mobile manipulation actions from human demonstrations,” in International Conference on Intelligent Robots and Systems (IROS), 2017.
  23. D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles, “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  24. T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in International Conference on Robotics and Automation (ICRA), 2018.
  25. C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, “Learning latent plans from play,” in Conference on Robot Learning (CoRL), 2020.
  26. H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg, “Learning by watching: Physical imitation of manipulation skills from human videos,” in International Conference on Intelligent Robots and Systems (IROS), 2021.
  27. Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang, “Dexmv: Imitation learning for dexterous manipulation from human videos,” in European Conference on Computer Vision (ECCV), 2022.
  28. J. Wong, A. Tung, A. Kurenkov, A. Mandlekar, L. Fei-Fei, S. Savarese, and R. Martín-Martín, “Error-aware imitation learning from teleoperation data for mobile manipulation,” in Conference on Robot Learning (CoRL), 2022.
  29. R. Gong, J. Huang, Y. Zhao, H. Geng, X. Gao, Q. Wu, W. Ai, Z. Zhou, D. Terzopoulos, and S.-C. Zhu, “Arnold: A benchmark for language-grounded task learning with continuous states in realistic 3d scenes,” in International Conference on Computer Vision (ICCV), 2023.
  30. J. Ye, J. Wang, B. Huang, Y. Qin, and X. Wang, “Learning continuous grasping function with a dexterous hand from human demonstrations,” IEEE Robotics and Automation Letters (RA-L), vol. 8, no. 5, pp. 2882–2889, 2023.
  31. Y. Urakami, A. Hodgkinson, C. Carlin, R. Leu, L. Rigazio, and P. Abbeel, “Doorgym: A scalable door opening environment and baseline agent,” in Advances in Neural Information Processing Systems (NeurIPS), 2019.
  32. Z. Xu, Z. He, and S. Song, “Universal manipulation policy network for articulated objects,” IEEE Robotics and Automation Letters (RA-L), vol. 7, no. 2, pp. 2447–2454, 2022.
  33. Y. Chen, T. Wu, S. Wang, X. Feng, J. Jiang, Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang, “Towards human-level bimanual dexterous manipulation with reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.
  34. H. Geng, Z. Li, Y. Geng, J. Chen, H. Dong, and H. Wang, “Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  35. K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su, “Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  36. F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al., “Sapien: A simulated part-based interactive environment,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  37. L. Liu, W. Xu, H. Fu, S. Qian, Q. Yu, Y. Han, and C. Lu, “AKB-48: a real-world articulated object knowledge base,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  38. H. Geng, H. Xu, C. Zhao, C. Xu, L. Yi, S. Huang, and H. Wang, “Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  39. Y. Zhu, T. Gao, L. Fan, S. Huang, M. Edmonds, H. Liu, F. Gao, C. Zhang, S. Qi, Y. N. Wu, et al., “Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense,” Engineering, vol. 6, no. 3, pp. 310–345, 2020.
  40. M. K. Johnson and E. H. Adelson, “Retrographic sensing for the measurement of surface texture and shape,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
  41. W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017.
  42. Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.-W. Chao, and D. Fox, “Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system,” in Robotics: Science and Systems (RSS), 2023.
  43. B. Zheng, S. Verma, J. Zhou, I. W. Tsang, and F. Chen, “Imitation learning: Progress, taxonomies and challenges,” IEEE Transactions on Neural Networks and Learning Systems, no. 99, pp. 1–16, 2022.
  44. Y. Qin, H. Su, and X. Wang, “From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation,” IEEE Robotics and Automation Letters (RA-L), vol. 7, no. 4, pp. 10873–10881, 2022.
  45. Z. Huang, J. Xu, S. Dai, K. Xu, H. Zhang, H. Huang, and R. Hu, “Nift: Neural interaction field and template for object manipulation,” in International Conference on Robotics and Automation (ICRA), 2023.
  46. A. Tekden, M. P. Deisenroth, and Y. Bekiroglu, “Grasp transfer based on self-aligning implicit representations of local surfaces,” IEEE Robotics and Automation Letters (RA-L), 2023.
  47. P. Li, T. Liu, Y. Li, Y. Geng, Y. Zhu, Y. Yang, and S. Huang, “Gendexgrasp: Generalizable dexterous grasping,” in International Conference on Robotics and Automation (ICRA), 2023.
  48. N. Likar, B. Nemec, and L. Žlajpah, “Virtual mechanism approach for dual-arm manipulation,” Robotica, vol. 32, no. 6, p. E3, 2014.
  49. A. J. Schmid, N. Gorges, D. Goger, and H. Worn, “Opening a door with a humanoid robot using multi-sensory tactile feedback,” in International Conference on Robotics and Automation (ICRA), 2008.
  50. H. Yousef, M. Boukallel, and K. Althoefer, “Tactile sensing for dexterous in-hand manipulation in robotics—a review,” Sensors and Actuators A: physical, vol. 167, no. 2, pp. 171–187, 2011.
  51. J. Lloyd and N. F. Lepora, “Pose and shear-based tactile servoing,” International Journal of Robotics Research (IJRR), vol. 0, no. 0, 2024.
  52. J. A. Fishel and G. E. Loeb, “Sensing tactile microvibrations with the biotac—comparison with human sensitivity,” in International Conference on Biomedical Robotics and Biomechatronics (BioRob), 2012.
  53. H. Liu, X. Xie, M. Millar, M. Edmonds, F. Gao, Y. Zhu, V. J. Santos, B. Rothrock, and S.-C. Zhu, “A glove-based system for studying hand-object manipulation via joint pose and force sensing,” in International Conference on Intelligent Robots and Systems (IROS), 2017.
  54. Z. Lu and H. Yu, “Gtac-hand: A robotic hand with integrated tactile sensing and extrinsic contact sensing capabilities,” Transactions on Mechatronics (TMECH), 2023.
  55. J. Kim, M. Lee, H. J. Shim, R. Ghaffari, H. R. Cho, D. Son, Y. H. Jung, M. Soh, C. Choi, S. Jung, et al., “Stretchable silicon nanoribbon electronics for skin prosthesis,” Nature Communications, vol. 5, no. 1, pp. 1–11, 2014.
  56. Y. Wu, Y. Liu, Y. Zhou, Q. Man, C. Hu, W. Asghar, F. Li, Z. Yu, J. Shang, G. Liu, et al., “A skin-inspired tactile sensor for smart prosthetics,” Science Robotics, vol. 3, no. 22, p. eaat0429, 2018.
  57. C. M. Boutry, M. Negre, M. Jorda, O. Vardoulis, A. Chortos, O. Khatib, and Z. Bao, “A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics,” Science Robotics, vol. 3, no. 24, p. eaau6914, 2018.
  58. W. W. Lee, Y. J. Tan, H. Yao, S. Li, H. H. See, M. Hon, K. A. Ng, B. Xiong, J. S. Ho, and B. C. Tee, “A neuro-inspired artificial peripheral nervous system for scalable electronic skins,” Science Robotics, vol. 4, no. 32, p. eaax2198, 2019.
  59. G. Li, S. Liu, L. Wang, and R. Zhu, “Skin-inspired quadruple tactile sensors integrated on a robot hand enable object recognition,” Science Robotics, vol. 5, no. 49, p. eabc8134, 2020.
  60. Y. Yu, J. Li, S. A. Solomon, J. Min, J. Tu, W. Guo, C. Xu, Y. Song, and W. Gao, “All-printed soft human-machine interface for robotic physicochemical sensing,” Science Robotics, vol. 7, no. 67, p. eabn0495, 2022.
  61. R. Li, R. Platt, W. Yuan, A. Ten Pas, N. Roscup, M. A. Srinivasan, and E. Adelson, “Localization and manipulation of small parts using gelsight tactile sensing,” in International Conference on Intelligent Robots and Systems (IROS), 2014.
  62. A. C. Abad and A. Ranasinghe, “Visuotactile sensors with emphasis on gelsight sensor: A review,” IEEE Sensors Journal, vol. 20, no. 14, pp. 7628–7638, 2020.
  63. A. C. Abad and A. Ranasinghe, “Low-cost gelsight with uv markings: Feature extraction of objects using alexnet and optical flow without 3d image reconstruction,” in International Conference on Robotics and Automation (ICRA), 2020.
  64. M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al., “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” IEEE Robotics and Automation Letters (RA-L), vol. 5, no. 3, pp. 3838–3845, 2020.
  65. I. H. Taylor, S. Dong, and A. Rodriguez, “Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger,” in International Conference on Robotics and Automation (ICRA), pp. 10781–10787, 2022.
  66. W. Li, M. Wang, J. Li, Y. Su, D. K. Jha, X. Qian, K. Althoefer, and H. Liu, “L33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT f-touch: A wireless gelsight with decoupled tactile and three-axis force sensing,” IEEE Robotics and Automation Letters (RA-L), 2023.
  67. J. Zhao and E. H. Adelson, “Gelsight svelte: A human finger-shaped single-camera tactile robot finger with large sensing coverage and proprioceptive sensing,” in International Conference on Intelligent Robots and Systems (IROS), 2023.
  68. W. Li, A. Alomainy, I. Vitanov, Y. Noh, P. Qi, and K. Althoefer, “F-touch sensor: Concurrent geometry perception and multi-axis force measurement,” IEEE Sensors Journal, vol. 21, no. 4, pp. 4300–4309, 2020.
  69. S. Wang, Y. She, B. Romero, and E. Adelson, “Gelsight wedge: Measuring high-resolution 3D contact geometry with a compact robot finger,” in International Conference on Robotics and Automation (ICRA), 2021.
  70. P. J. Besl and N. D. McKay, “Method for registration of 3-d shapes,” in Sensor fusion IV: control paradigms and data structures, vol. 1611, pp. 586–606, Spie, 1992.
  71. K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani, “Where2act: From pixels to actions for articulated 3D objects,” in International Conference on Computer Vision (ICCV), 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Zihang Zhao (8 papers)
  2. Yuyang Li (22 papers)
  3. Wanlin Li (26 papers)
  4. Zhenghao Qi (5 papers)
  5. Lecheng Ruan (17 papers)
  6. Yixin Zhu (102 papers)
  7. Kaspar Althoefer (28 papers)
Citations (3)