Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Proximity and Visuotactile Point Cloud Fusion for Contact Patches in Extreme Deformation (2307.03839v1)

Published 7 Jul 2023 in cs.RO

Abstract: Equipping robots with the sense of touch is critical to emulating the capabilities of humans in real world manipulation tasks. Visuotactile sensors are a popular tactile sensing strategy due to data output compatible with computer vision algorithms and accurate, high resolution estimates of local object geometry. However, these sensors struggle to accommodate high deformations of the sensing surface during object interactions, hindering more informative contact with cm-scale objects frequently encountered in the real world. The soft interfaces of visuotactile sensors are often made of hyperelastic elastomers, which are difficult to simulate quickly and accurately when extremely deformed for tactile information. Additionally, many visuotactile sensors that rely on strict internal light conditions or pattern tracking will fail if the surface is highly deformed. In this work, we propose an algorithm that fuses proximity and visuotactile point clouds for contact patch segmentation that is entirely independent from membrane mechanics. This algorithm exploits the synchronous, high-res proximity and visuotactile modalities enabled by an extremely deformable, selectively transmissive soft membrane, which uses visible light for visuotactile sensing and infrared light for proximity depth. We present the hardware design, membrane fabrication, and evaluation of our contact patch algorithm in low (10%), medium (60%), and high (100%+) membrane strain states. We compare our algorithm against three baselines: proximity-only, tactile-only, and a membrane mechanics model. Our proposed algorithm outperforms all baselines with an average RMSE under 2.8mm of the contact patch geometry across all strain ranges. We demonstrate our contact patch algorithm in four applications: varied stiffness membranes, torque and shear-induced wrinkling, closed loop control for whole body manipulation, and pose estimation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. “Tactile sensors for advanced intelligent systems” In Advanced Intelligent Systems 1.8 Wiley Online Library, 2019, pp. 1900090
  2. Kazuhiro Shimonomura “Tactile image sensors employing camera: A review” In Sensors 19.18 MDPI, 2019, pp. 3933
  3. “Electronic skins and machine learning for intelligent soft robots” In Science Robotics 5.41 American Association for the Advancement of Science, 2020, pp. eaaz9239
  4. Wenzhen Yuan, Siyuan Dong and Edward H Adelson “Gelsight: High-resolution robot tactile sensors for estimating geometry and force” In Sensors 17.12 Multidisciplinary Digital Publishing Institute, 2017, pp. 2762
  5. “Omnitact: A multi-directional high-resolution touch sensor” In 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 618–624 IEEE
  6. Won Kyung Do and Monroe Kennedy “DenseTact: Optical tactile sensor for dense shape reconstruction” In 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 6188–6194 IEEE
  7. Nathan F Lepora “Soft biomimetic optical tactile sensing with the TacTip: A review” In IEEE Sensors Journal 21.19 IEEE, 2021, pp. 21131–21143
  8. “Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation” In 2019 2nd IEEE International Conference on Soft Robotics (RoboSoft), 2019, pp. 597–604 IEEE
  9. “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation” In IEEE Robotics and Automation Letters 5.3 IEEE, 2020, pp. 3838–3845
  10. Yipai Du, Guanlan Zhang and Michael Yu Wang “3D Contact Point Cloud Reconstruction From Vision-Based Tactile Flow” In IEEE Robotics and Automation Letters 7.4 IEEE, 2022, pp. 12177–12184
  11. “Fingervision tactile sensor design and slip detection using convolutional lstm network” In arXiv preprint arXiv:1810.02653, 2018
  12. “Continuous Marker Patterns for Representing Contact Information in Vision-Based Tactile Sensor: Principle, Algorithm, and Verification” In IEEE Transactions on Instrumentation and Measurement 71 IEEE, 2022, pp. 1–12
  13. Mingxuan Li, Tiemin Li and Yao Jiang “Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2D to 3D-A Review” In IEEE Sensors Journal IEEE, 2023
  14. Ian H Taylor, Siyuan Dong and Alberto Rodriguez “GelSlim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger” In 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 10781–10787 IEEE
  15. Megha H Tippur and Edward H Adelson “GelSight360: An Omnidirectional Camera-Based Tactile Sensor for Dexterous Robotic Manipulation” In 2023 IEEE International Conference on Soft Robotics (RoboSoft), 2023, pp. 1–8 IEEE
  16. “High-Precision 3D Reconstruction Study with Emphasis on Refractive Calibration of GelStereo-Type Sensors” In Sensors 23.5 MDPI, 2023, pp. 2675
  17. “ShapeMap 3-D: Efficient shape mapping through dense touch and vision” In 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 7073–7080 IEEE
  18. Maria Bauza, Antonia Bronars and Alberto Rodriguez “Tac2pose: Tactile object pose estimation from the first touch” In arXiv preprint arXiv:2204.11701, 2022
  19. “MidasTouch: Monte-Carlo inference over distributions across sliding touch” In Conference on Robot Learning, 2023, pp. 319–331 PMLR
  20. “In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors” In IEEE Access 11 IEEE, 2023, pp. 17218–17232
  21. “Tracking objects with point clouds from vision and touch” In 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4000–4007 IEEE
  22. “Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control” In Conference on Robot Learning, 2023, pp. 1850–1859 PMLR
  23. “Proximity perception in human-centered robotics: A survey on sensing systems and applications” In IEEE Transactions on Robotics 38.3 IEEE, 2021, pp. 1599–1620
  24. Jessica Yin, Tess Hellebrekers and Carmel Majidi “Closing the Loop with Liquid-Metal Sensing Skin for Autonomous Soft Robot Gripping” In 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), 2020, pp. 661–667 DOI: 10.1109/RoboSoft48309.2020.9116000
  25. “Liquid Metal-Microelectronics Integration for a Sensorized Soft Robot Skin” In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018, pp. 5924–5929 IEEE
  26. Radhen Patel, Rebecca Cox and Nikolaus Correll “Integrated proximity, contact and force sensing using elastomer-embedded commodity proximity sensors” In Autonomous Robots 42 Springer, 2018, pp. 1443–1458
  27. “Design of a Multimodal Fingertip Sensor for Dynamic Manipulation” In arXiv preprint arXiv:2209.11368, 2022
  28. “Towards Robust Autonomous Grasping with Reflexes Using High-Bandwidth Sensing and Actuation” In arXiv preprint arXiv:2209.11367, 2022
  29. Akihiko Yamaguchi and Christopher G Atkeson “Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables” In 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), 2016, pp. 1045–1051 IEEE
  30. “Seeing through your skin: Recognizing objects with a novel visuotactile sensor” In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 1218–1227
  31. “Finger-STS: Combined Proximity and Tactile Sensing for Robotic Manipulation” In IEEE Robotics and Automation Letters 7.4 IEEE, 2022, pp. 10865–10872
  32. “Soft Robotic Link with Controllable Transparency for Vision-based Tactile and Proximity Sensing” In arXiv preprint arXiv:2211.03253, 2022
  33. “Multimodal proximity and visuotactile sensing with a selectively transmissive soft membrane” In 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft), 2022, pp. 802–808 IEEE
  34. “Interpreting and predicting tactile signals via a physics-based and data-driven framework” In arXiv preprint arXiv:2006.03777, 2020
  35. “Ground truth force distribution for learning-based tactile sensing: A finite element approach” In IEEE Access 7 IEEE, 2019, pp. 173438–173449
  36. Raymond W Ogden, Giuseppe Saccomandi and Ivonne Sgura “Fitting hyperelastic models to experimental data” In Computational Mechanics 34.6 Springer, 2004, pp. 484–502
  37. “Experimental and theoretical analyses of the age-dependent large-strain behavior of Sylgard 184 (10:1) silicone elastomer” In Journal of the Mechanical Behavior of Biomedical Materials 60, 2016, pp. 425–437 DOI: https://doi.org/10.1016/j.jmbbm.2016.02.022
  38. “Dense tactile force estimation using GelSlim and inverse FEM” In 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 5418–5424 IEEE
  39. “Fast model-based contact patch and pose estimation for highly deformable dense-geometry tactile sensors” In IEEE Robotics and Automation Letters 5.2 IEEE, 2019, pp. 1811–1818
  40. “Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides” In Science robotics 1.1 American Association for the Advancement of Science, 2016, pp. eaai7529
  41. “High-resolution 3-dimensional contact deformation tracking for FingerVision sensor with dense random color pattern” In IEEE Robotics and Automation Letters 6.2 IEEE, 2021, pp. 2147–2154
  42. “Taxim: An example-based simulation model for gelsight tactile sensors” In IEEE Robotics and Automation Letters 7.2 IEEE, 2022, pp. 2361–2368
  43. “Contactdb: Analyzing and predicting grasp contact via thermal imaging” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8709–8719
  44. Arjun Lakshmipathy, Dominik Bauer and Nancy S Pollard “Contact tracing: A low cost reconstruction framework for surface contact interpolation” In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 5165–5172 IEEE
  45. “Stretchable surfaces with programmable 3D texture morphing for synthetic camouflaging skins” In Science 358.6360 American Association for the Advancement of Science, 2017, pp. 210–214
  46. “Programming 3D curves with discretely constrained cylindrical inflatables” In Advanced Materials Wiley Online Library, pp. 2300535
  47. “Electroadhesive Clutches for Programmable Shape Morphing of Soft Actuators” In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022, pp. 11594–11599 IEEE
  48. “Multi-environment robotic transitions through adaptive morphogenesis” In Nature 610.7931 Nature Publishing Group UK London, 2022, pp. 283–289
  49. “Punyo-1: Soft tactile-sensing upper-body robot for large object manipulation and physical human interaction” In 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft), 2022, pp. 844–851 IEEE
  50. Qian-Yi Zhou, Jaesik Park and Vladlen Koltun “Open3D: A Modern Library for 3D Data Processing” In arXiv:1801.09847, 2018
  51. “Tactile SLAM: Real-time inference of shape and pose from planar pushing” In 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 11322–11328 IEEE
  52. Jialiang Zhao, Maria Bauza and Edward H Adelson “FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback” In arXiv preprint arXiv:2303.07997, 2023
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now