Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation (2403.15107v1)

Published 22 Mar 2024 in cs.RO and cs.CV

Abstract: Humans seemingly incorporate potential touch signals in their perception. Our goal is to equip robots with a similar capability, which we term \ourmodel. \ourmodel aims to predict the expected touch signal based on a visual patch representing the touched area. We frame this problem as the task of learning a low-dimensional visual-tactile embedding, wherein we encode a depth patch from which we decode the tactile signal. To accomplish this task, we employ ReSkin, an inexpensive and replaceable magnetic-based tactile sensor. Using ReSkin, we collect and train PseudoTouch on a dataset comprising aligned tactile and visual data pairs obtained through random touching of eight basic geometric shapes. We demonstrate the efficacy of PseudoTouch through its application to two downstream tasks: object recognition and grasp stability prediction. In the object recognition task, we evaluate the learned embedding's performance on a set of five basic geometric shapes and five household objects. Using PseudoTouch, we achieve an object recognition accuracy 84% after just ten touches, surpassing a proprioception baseline. For the grasp stability task, we use ACRONYM labels to train and evaluate a grasp success predictor using PseudoTouch's predictions derived from virtual depth information. Our approach yields an impressive 32% absolute improvement in accuracy compared to the baseline relying on partial point cloud data. We make the data, code, and trained models publicly available at http://pseudotouch.cs.uni-freiburg.de.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. C. Lang, A. Braun, and A. Valada, “Robust object detection using knowledge graph embeddings,” in DAGM German Conference on Pattern Recognition, 2022, pp. 445–461.
  2. E. Chisari, N. Heppert, T. Welschehold, W. Burgard, and A. Valada, “Centergrasp: Object-aware implicit representation learning for simultaneous shape reconstruction and 6-dof grasp estimation,” arXiv preprint arXiv:2312.08240, 2023.
  3. J. O. von Hartz, E. Chisari, T. Welschehold, W. Burgard, J. Boedecker, and A. Valada, “The treachery of images: Bayesian scene keypoints for deep policy learning in robotic manipulation,” IEEE Robotics and Automation Letters, 2023.
  4. M. Bauza, A. Bronars, and A. Rodriguez, “Tac2pose: Tactile object pose estimation from the first touch,” arXiv preprint arXiv:2204.11701, 2022.
  5. R. Buchanan, A. Röfer, J. Moura, A. Valada, and S. Vijayakumar, “Online estimation of articulated objects with factor graphs using vision and proprioceptive sensing,” arXiv preprint arXiv:2309.16343, 2023.
  6. J. Xu, H. Lin, S. Song, and M. Ciocarlie, “Tandem3d: Active tactile exploration for 3d object recognition,” in Proc. IEEE Int. Conf. on Rob. and Auto., 2023, pp. 10 401–10 407.
  7. L. Chumbley, M. Gu, R. Newbury, J. Leitner, and A. Cosgun, “Integrating high-resolution tactile sensing into grasp stability prediction,” in Conference on Robots and Vision, 2022, pp. 98–105.
  8. R. Bhirangi, T. Hellebrekers, C. Majidi, and A. Gupta, “Reskin: versatile, replaceable, lasting tactile skins,” in Proc. Conf. on Rob. Learn., 2021.
  9. W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” IEEE Sensors, vol. 17, no. 12, p. 2762, 2017.
  10. M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon et al., “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 3838–3845, 2020.
  11. N. F. Lepora, Y. Lin, B. Money-Coomes, and J. Lloyd, “Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 9382–9388, 2022.
  12. S. Wang, M. Lambeta, P.-W. Chou, and R. Calandra, “Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3930–3937, 2022.
  13. N. Wettels, V. J. Santos, R. S. Johansson, and G. E. Loeb, “Biomimetic tactile sensor array,” Advanced Robotics, vol. 22, no. 8, pp. 829–849, 2008.
  14. A. SaLoutos, E. Stanger-Jones, M. Guo, H. Kim, and S. Kim, “Design of a multimodal fingertip sensor for dynamic manipulation,” in Proc. IEEE Int. Conf. on Rob. and Auto., 2023, pp. 8017–8024.
  15. T. Le Signor, N. Dupré, and G. F. Close, “A gradiometric magnetic force sensor immune to stray magnetic fields for robotic hands and grippers,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3070–3076, 2022.
  16. R. Bhirangi, A. DeFranco, J. Adkins, C. Majidi, A. Gupta, T. Hellebrekers, and V. Kumar, “All the feels: A dexterous hand with large area sensing,” arXiv preprint arXiv:2210.15658, 2022.
  17. Y. Yan, Z. Hu, Z. Yang, W. Yuan, C. Song, J. Pan, and Y. Shen, “Soft magnetic skin for super-resolution tactile sensing with force self-decoupling,” Sci. Robot., vol. 6, no. 51, p. eabc8801, 2021.
  18. A. Younes, D. Honerkamp, T. Welschehold, and A. Valada, “Catch me if you hear me: Audio-visual navigation in complex unmapped environments with moving sounds,” IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 928–935, 2023.
  19. J. V. Hurtado and A. Valada, “Semantic scene segmentation for robotics,” in Deep learning for robot perception and cognition, 2022, pp. 279–311.
  20. L. Rustler, J. Lundell, J. K. Behrens, V. Kyrki, and M. Hoffmann, “Active Visuo-Haptic Object Shape Completion,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 5254–5261, 2022.
  21. E. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal, “3D Shape Reconstruction from Vision and Touch,” in Proc. Adv. Neural Inform. Process. Syst., 2020.
  22. P. K. Murali, B. Porr, and M. Kaboli, “Touch if it’s transparent! ACTOR: Active Tactile-based Category-Level Transparent Object Reconstruction,” in Proc. IEEE Int. Conf. on Intel. Rob. and Syst., 2023.
  23. P. Falco, S. Lu, A. Cirillo, C. Natale, S. Pirozzi, and D. Lee, “Cross-modal visuo-tactile object recognition using robotic active exploration,” in Proc. IEEE Int. Conf. on Rob. and Auto., 2017.
  24. P. Falco, S. Lu, C. Natale, S. Pirozzi, and D. Lee, “A transfer learning approach to cross-modal object recognition: from visual observation to robotic haptic exploration,” IEEE Trans. Robot., 2019.
  25. M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks,” IEEE Trans. on Robotics, vol. 36, no. 3, 2020.
  26. S. Zhong, A. Albini, O. P. Jones, P. Maiolino, and I. Posner, “Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data Generation,” in Proc. Conf. on Rob. Learn., 2023, pp. 1618–1628.
  27. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  28. J.-T. Lee, D. Bollegala, and S. Luo, “”Touching to See” and ”Seeing to Feel”: Robotic Cross-modal Sensory Data Generation for Visual-Tactile Perception,” in Proc. IEEE Int. Conf. on Rob. and Auto., 2019.
  29. F. Yang, J. Zhang, and A. Owens, “Generating visual scenes from touch,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 22 070–22 080.
  30. F. Yang, C. Feng, Z. Chen, H. Park, D. Wang, Y. Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens, and A. Wong, “Binding Touch to Everything: Learning Unified Multimodal Tactile Representations,” arXiv preprint arXiv:2401.18084, 2024.
  31. R. Corcodel, S. Jain, and J. van Baar, “Interactive Tactile Perception for Classification of Novel Object Instances,” in Proc. IEEE Int. Conf. on Intel. Rob. and Syst., 2020, pp. 9861–9868.
  32. R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, J. Lin, E. H. Adelson, and S. Levine, “The feeling of success: Does touch sensing help predict grasp outcomes?” in Proc. Conf. on Rob. Learn., 2017, pp. 314–323.
  33. Q.-Y. Zhou and V. Koltun, “Dense scene reconstruction with points of interest,” ACM Trans. Graph., vol. 32, no. 4, 2013.
  34. J. A. E. Andersson, J. Gillis, G. Horn, J. B. Rawlings, and M. Diehl, “CasADi – A software framework for nonlinear optimization and optimal control,” Mathematical Programming Computation, vol. 11, no. 1, pp. 1–36, 2019.
  35. M. Savva, A. X. Chang, and P. Hanrahan, “Semantically-Enriched 3D Models for Common-sense Knowledge,” CVPR Workshop on Functionality, Physics, Intentionality and Causality, 2015.
  36. C. Eppner, A. Mousavian, and D. Fox, “ACRONYM: A Large-Scale Grasp Dataset Based on Simulation,” in Proc. IEEE Int. Conf. on Rob. and Auto., 2021, pp. 6222–6227.
  37. M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox, “Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes,” in Proc. IEEE Int. Conf. on Rob. and Auto., 2021, pp. 13 438–13 444.
  38. Y. Xiang, C. Xie, A. Mousavian, and D. Fox, “Learning rgb-d feature embeddings for unseen object instance segmentation,” in Proc. Conf. on Rob. Learn., 2021, pp. 461–470.
  39. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” Adv, in neural information processing systems, vol. 30, 2017.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets