Papers
Topics
Authors
Recent
AI Research Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 77 tok/s
Gemini 2.5 Pro 52 tok/s Pro
GPT-5 Medium 30 tok/s Pro
GPT-5 High 31 tok/s Pro
GPT-4o 91 tok/s Pro
Kimi K2 178 tok/s Pro
GPT OSS 120B 385 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

RealDex: Towards Human-like Grasping for Robotic Dexterous Hand (2402.13853v2)

Published 21 Feb 2024 in cs.RO, cs.AI, and cs.CV

Abstract: In this paper, we introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns, enriched by multi-view and multimodal visual data. Utilizing a teleoperation system, we seamlessly synchronize human-robot hand poses in real time. This collection of human-like motions is crucial for training dexterous hands to mimic human movements more naturally and precisely. RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios. Moreover, we introduce a cutting-edge dexterous grasping motion generation framework, which aligns with human experience and enhances real-world applicability through effectively utilizing Multimodal LLMs. Extensive experiments have demonstrated the superior performance of our method on RealDex and other open datasets. The complete dataset and code will be made available upon the publication of this work.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Goal directed multi-finger manipulation: Control policies and analysis. Computers & Graphics, 37(7):830–839, 2013.
  2. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020.
  3. Dexterous manipulation using both palm and fingers. In ICRA, pages 1560–1565. IEEE, 2014.
  4. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239–256, 1992.
  5. etc. Chao, Yu-Wei. Dexycb: A benchmark for capturing hand grasping of objects. In CVPR, pages 9044–9053, 2021.
  6. Learning robust real-world dexterous grasping policies via implicit shape augmentation. arXiv preprint arXiv:2210.13638, 2022.
  7. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In CVPR, pages 20577–20586, 2022.
  8. Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerf. In ICRA, pages 1757–1763. IEEE, 2023.
  9. Push-grasping with dexterous hands: Mechanics and a method. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2123–2130. IEEE, 2010.
  10. Einscan-pro+. https://www.einscan.com/handheld-3d-scanner/einscan-pro-plus/, 2018.
  11. Graspnet-1billion: A large-scale benchmark for general object grasping. In CVPR, pages 11444–11453, 2020.
  12. Transcg: A large-scale real-world dataset for transparent object depth completion and a grasping baseline. IEEE Robotics and Automation Letters, 7(3):7383–7390, 2022.
  13. Gemini. https://deepmind.google/technologies/gemini/#introduction, 2023.
  14. Synthesis of compositional animations from textual descriptions. In ICCV, pages 1396–1406, 2021.
  15. The columbia grasp database. In 2009 IEEE international conference on robotics and automation, pages 1710–1716. IEEE, 2009.
  16. Gpt-4. https://openai.com/research/gpt-4, 2023.
  17. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807–11816, 2019.
  18. Learning joint reconstruction of hands and manipulated objects, April 2019. arXiv:1904.05767 [cs].
  19. Generalization in dexterous manipulation via geometry-aware multi-task learning. arXiv preprint arXiv:2111.03062, 2021.
  20. IFL-CAMP. easy_handeye: Automated, hardware-independent hand-eye calibration. https://github.com/IFL-CAMP/easy_handeye, 2021. Accessed: 2024-01-09.
  21. Hand-Object Contact Consistency Reasoning for Human Grasps Generation, April 2021. arXiv:2104.03304 [cs].
  22. Flexible layouts for fiducial tags. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 2019.
  23. GenDexGrasp: Generalizable Dexterous Grasping, March 2023. arXiv:2210.00722 [cs].
  24. Deep differentiable grasp planner for high-dof grippers. arXiv preprint arXiv:2002.01530, 2020.
  25. Ddgc: Generative deep dexterous grasping in clutter. IEEE Robotics and Automation Letters, 6(4):6899–6906, 2021.
  26. Graspit! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4):110–122, 2004.
  27. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652–660, 2017.
  28. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413, 2017.
  29. Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
  30. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022.
  31. Shadowrobot. https://www.shadowrobot.com/dexterous-hand-series/, 2005.
  32. Learning high-dof reaching-and-grasping via dynamic representation of gripper-object interaction. arXiv preprint arXiv:2204.13998, 2022.
  33. Neural state machine for character-scene interactions. ACM Trans. Graph., 38(6):209–1, 2019.
  34. stytim. k4a-calibration: Azure Kinect multi-camera extrinsic calibration. https://github.com/stytim/k4a-calibration, 2023. Accessed: 2024-01-06.
  35. Grab: A dataset of whole-body human grasping of objects. In European Conference on Computer Vision, pages 581–600. Springer, 2020.
  36. Goal: Generating 4d whole-body motion for hand-object grasping. In CVPR, pages 13263–13273, 2022.
  37. Teleoperation system of shadowrobot. https://www.shadowrobot.com/teleoperation/, 2022.
  38. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012.
  39. Capturing hands in action using discriminative salient points and physics simulation. CoRR, abs/1506.02178, 2015.
  40. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  41. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. arXiv preprint arXiv:2304.00464, 2023.
  42. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In ICCV, pages 913–922, 2021.
  43. DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation, March 2023. arXiv:2210.02697 [cs].
  44. Dvgg: Deep variational grasp generation for dextrous manipulation. IEEE Robotics and Automation Letters, 7(2):1659–1666, 2022.
  45. Saga: Stochastic whole-body grasping with contact. arXiv preprint arXiv:2112.10103, 2021.
  46. Saga: Stochastic whole-body grasping with contact. In ECCV, pages 257–274. Springer, 2022.
  47. Learning generalizable dexterous manipulation from human grasp affordance. In Conference on Robot Learning, pages 618–629. PMLR, 2023.
  48. UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy, March 2023. arXiv:2303.00938 [cs].
  49. Human-centric scene understanding for 3d large-scale scenarios. In ICCV, pages 20349–20359, 2023.
  50. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In CVPR, pages 20953–20962, 2022.
  51. Manipnet: Neural manipulation synthesis with a hand-object spatial representation. ACM Transactions on Graphics (ToG), 40(4):1–14, 2021.
  52. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018.
  53. Cylindrical and asymmetrical 3d convolution networks for lidar-based perception. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6807–6822, 2021.
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up