Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation (2308.14277v2)

Published 28 Aug 2023 in cs.RO

Abstract: The advancements in vision-based tactile sensors have boosted the aptitude of robots to perform contact-rich manipulation, particularly when precise positioning and contact state of the manipulated objects are crucial for successful execution. In this work, we present 9DTact, a straightforward yet versatile tactile sensor that offers 3D shape reconstruction and 6D force estimation capabilities. Conceptually, 9DTact is designed to be highly compact, robust, and adaptable to various robotic platforms. Moreover, it is low-cost and easy-to-fabricate, requiring minimal assembly skills. Functionally, 9DTact builds upon the optical principles of DTact and is optimized to achieve 3D shape reconstruction with enhanced accuracy and efficiency. Remarkably, we leverage the optical and deformable properties of the translucent gel so that 9DTact can perform 6D force estimation without the participation of auxiliary markers or patterns on the gel surface. More specifically, we collect a dataset consisting of approximately 100,000 image-force pairs from 175 complex objects and train a neural network to regress the 6D force, which can generalize to unseen objects. To promote the development and applications of vision-based tactile sensors, we open-source both the hardware and software of 9DTact, along with a comprehensive video tutorial, all of which are available at https://linchangyi1.github.io/9DTact.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. C. Lin, Z. Lin, S. Wang, and H. Xu, “Dtact: A vision-based tactile sensor that measures high-resolution 3d geometry directly from darkness,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 10 359–10 366.
  2. W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017.
  3. I. H. Taylor, S. Dong, and A. Rodriguez, “Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 10 781–10 787.
  4. M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer et al., “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 3838–3845, 2020.
  5. GelSight. (2022) Gelsight-mini. [Online]. Available: https://www.gelsight.com/gelsightmini/
  6. R. Li, R. Platt, W. Yuan, A. Ten Pas, N. Roscup, M. A. Srinivasan, and E. Adelson, “Localization and manipulation of small parts using gelsight tactile sensing,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2014, pp. 3988–3993.
  7. S. Dong, D. K. Jha, D. Romeres, S. Kim, D. Nikovski, and A. Rodriguez, “Tactile-rl for insertion: Generalization to objects of unknown geometry,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 6437–6443.
  8. Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. Adelson, “Cable manipulation with a tactile-reactive gripper,” The International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1385–1401, 2021.
  9. A. Wilson, H. Jiang, W. Lian, and W. Yuan, “Cable routing and assembly using tactile-driven motion primitives,” arXiv preprint arXiv:2303.11765, 2023.
  10. S. Dong, W. Yuan, and E. H. Adelson, “Improved gelsight tactile sensor for measuring geometry and slip,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2017, pp. 137–144.
  11. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  12. D. Ma, E. Donlon, S. Dong, and A. Rodriguez, “Dense tactile force estimation using gelslim and inverse fem,” in 2019 International Conference on Robotics and Automation (ICRA).   IEEE, 2019, pp. 5418–5424.
  13. G. Zhang, Y. Du, H. Yu, and M. Y. Wang, “Deltact: A vision-based tactile sensor using a dense color pattern,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 778–10 785, 2022.
  14. Y. Zhang, Z. Kan, Y. Yang, Y. A. Tse, and M. Y. Wang, “Effective estimation of contact force and torque for vision-based tactile sensors with helmholtz–hodge decomposition,” IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 4094–4101, 2019.
  15. H. Bhatia, V. Pascucci, and P.-T. Bremer, “The natural helmholtz-hodge decomposition for open-boundary flow analysis,” IEEE transactions on visualization and computer graphics, vol. 20, no. 11, pp. 1566–1578, 2014.
  16. W. K. Do, B. Jurewicz, and M. Kennedy, “Densetact 2.0: Optical tactile sensor for shape and force reconstruction,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 12 549–12 555.
  17. M. K. Johnson and E. H. Adelson, “Retrographic sensing for the measurement of surface texture and shape,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition.   IEEE, 2009, pp. 1070–1077.
  18. S. Wang, Y. She, B. Romero, and E. Adelson, “Gelsight wedge: Measuring high-resolution 3d contact geometry with a compact robot finger,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 6468–6475.
  19. G. Bradski, “The opencv library.” Dr. Dobb’s Journal: Software Tools for the Professional Programmer, vol. 25, no. 11, pp. 120–123, 2000.
  20. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.
  21. D. F. Gomes, P. Paoletti, and S. Luo, “Generation of gelsight tactile images for sim2real learning,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 4177–4184, 2021.
  22. A. Church, J. Lloyd, N. F. Lepora et al., “Tactile sim-to-real policy transfer via real-to-sim image translation,” in Conference on Robot Learning.   PMLR, 2022, pp. 1645–1654.
  23. W. Chen, Y. Xu, Z. Chen, P. Zeng, R. Dang, R. Chen, and J. Xu, “Bidirectional sim-to-real transfer for gelsight tactile sensors with cyclegan,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6187–6194, 2022.
  24. Q. Zhou and A. Jacobson, “Thingi10k: A dataset of 10,000 3d-printing models,” arXiv preprint arXiv:1605.04797, 2016.
  25. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
  26. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  27. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  28. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
Citations (27)
View on Semantic Scholar

Summary

  • The paper introduces 9DTact, a sensor offering both high-precision 3D shape reconstruction (MAE: 0.0462 mm) and reliable 6D force estimation.
  • It employs a streamlined design with an OV5647 camera and custom LED board to ensure uniform illumination and ease of fabrication.
  • The sensor’s open-source design and robust performance promise broadened applications in advanced robotic manipulation tasks.

Overview of 9DTact: A Compact Vision-Based Tactile Sensor for 3D Shape Reconstruction and 6D Force Estimation

The paper presents 9DTact, a novel vision-based tactile sensor designed to enhance robotic tactile sensing capabilities by providing both 3D shape reconstruction and 6D force estimation functionalities. This research focuses on addressing the limitations of previous tactile sensors, such as their bulkiness, complexity of assembly, and fragility, by introducing a sensor that is compact, robust, low-cost, and easy to fabricate.

Sensor Design and Features

9DTact is distinguished by its meticulous hardware design, which aims to simplify fabrication and maximizes adaptability across robotic platforms. It uses a wide-angle OV5647 camera coupled with a custom LED board for even illumination, effectively eliminating issues with non-uniform light distribution that plagued earlier designs. The camera and LED arrangements, secured with 3D printed frames, contribute to the system's overall robustness. The materials used are readily accessible, with the assembly process requiring minimal technical skills, enhancing the sensor's reproducibility.

A critical highlight of 9DTact is its transparent and translucent gel layers, which play a vital role in achieving high-resolution tactile feedback by capturing the interaction with objects through deformation of the gel. This design supports both the 3D shape reconstruction and force estimation functionalities, without the need for additional markers or pre-patterned surfaces, thereby preserving the gel's integrity and the sensor's manufacturability.

3D Shape Reconstruction Capabilities

The 3D shape reconstruction mechanism leverages a mapping model to convert sensed optical data into depth maps. The paper details a "single image" calibration method which efficiently aligns tactile image data with actual depth information, thus enabling accurate 3D modeling of contacted surfaces. The sensor achieves a mean absolute error (MAE) of 0.0462 mm, verifying its precision. These capabilities allow 9DTact to reconstruct shapes with a degree of detail that is highly beneficial for nuanced manipulation tasks in robotics.

6D Force Estimation

For force estimation, 9DTact utilizes an innovative method that exploits the gel's deformable properties. Upon object interaction, the gel flows differently depending on the external force applied, altering pixel brightness and providing a rich set of data for force inference. This dense deformation representation allows the extraction of normal, shear, and torque forces without auxiliary markers. A deep learning model trained on a dataset of approximately 100,000 image-force pairs from a diverse set of 175 objects validates the sensor's efficacy in force decoding. The sensor achieved a mean absolute error in force estimation of 0.370 N and in torque estimation of 0.0077 Nm for previously unseen objects, demonstrating robust generalization.

Implications and Future Directions

The development of 9DTact represents a significant step forward in tactile sensing technology for robotics, combining ease of fabrication with high performance in shape and force detection. The open-source release of the sensor's design files and training datasets promises to catalyze further advancements and applications in robotic manipulation. The future scope of this research lies in exploring its integration into more complex and dynamic robotic systems, potentially expanding to collaborative and adaptive robotic tasks in unstructured environments.

In conclusion, 9DTact's design leverages recent advancements in materials and optical engineering, while its dual functional capabilities in shape reconstruction and force estimation align with the growing demand for advanced tactile sensing technologies in robotics. This paper offers valuable insights into not only the technical intricacies of sensor design but also the practical aspects of deployment and open-source collaboration in the field of robotics research.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. 9DTact
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube