Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (2401.18084v1)

Published 31 Jan 2024 in cs.CV and cs.RO

Abstract: The ability to associate touch with other modalities has huge implications for humans and computational systems. However, multimodal learning with touch remains challenging due to the expensive data collection process and non-standardized sensor outputs. We introduce UniTouch, a unified tactile model for vision-based touch sensors connected to multiple modalities, including vision, language, and sound. We achieve this by aligning our UniTouch embeddings to pretrained image embeddings already associated with a variety of other modalities. We further propose learnable sensor-specific tokens, allowing the model to learn from a set of heterogeneous tactile sensors, all at the same time. UniTouch is capable of conducting various touch sensing tasks in the zero-shot setting, from robot grasping prediction to touch image question answering. To the best of our knowledge, UniTouch is the first to demonstrate such capabilities. Project page: https://cfeng16.github.io/UniTouch/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (128)
  1. Simulation of vision-based tactile sensors using physics based rendering. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1–7, 2020.
  2. Labelling unlabelled videos from scratch with multi-modal self-supervision. Advances in Neural Information Processing Systems, 33:4660–4671, 2020.
  3. See, hear, and read: Deep aligned representations. ArXiv, abs/1706.00932, 2017.
  4. Learning to taste: A multimodal wine dataset. arXiv preprint arXiv:2308.16900, 2023.
  5. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06595, 2023.
  6. The feeling of success: Does touch sensing help predict grasp outcomes? Conference on Robot Learning (CoRL), 2017.
  7. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3:3300–3307, 2018.
  8. Multimodal perception for dexterous manipulation. ArXiv, abs/2112.14298, 2021.
  9. Spatio-temporal attention model for tactile texture recognition. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9896–9902, 2020.
  10. Touchroller: A rolling optical tactile sensor for rapid assessment of large surfaces. ArXiv, abs/2103.00595, 2021.
  11. Vis2hap: Vision-based haptic rendering by cross-modal generation. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12443–12449, 2023.
  12. Using collocated vision and tactile sensors for visual servoing and localization. IEEE Robotics and Automation Letters, 7:3427–3434, 2022.
  13. iquery: Instruments as queries for audio-visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14675–14686, 2023a.
  14. Movies2scenes: Using movie metadata to learn scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6535–6544, 2023b.
  15. Sound localization from motion: Jointly learning sound direction and camera rotation. arXiv preprint arXiv:2303.11329, 2023c.
  16. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
  17. Tactile sim-to-real policy transfer via real-to-sim image translation. In Conference on Robot Learning, 2021.
  18. Contrastive vision-language pre-training with limited resources. In European Conference on Computer Vision, 2022.
  19. Force and tactile sensors. In Springer Handbook of Robotics, 2008.
  20. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500, 2023.
  21. Canonical partial least squares and continuum power regression. Journal of Chemometrics, 15, 2001.
  22. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  23. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11162–11173, 2021.
  24. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  25. Conditional generation of audio from video via foley analogies. In Conference on Computer Vision and Pattern Recognition 2023, 2023.
  26. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  27. Self-supervised video forensics by audio-visual anomaly detection. Computer Vision and Pattern Recognition (CVPR), 2023.
  28. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023a.
  29. Supervised autoencoder joint learning on heterogeneous tactile sensory data: Improving material classification performance. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10907–10913, 2020.
  30. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In CoRL, 2021a.
  31. On explainability and sensor-adaptability of a robot tactile texture representation using a two-stage recurrent networks. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1296–1303, 2021b.
  32. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In CVPR, 2022.
  33. The objectfolder benchmark: Multisensory learning with neural and real objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17276–17286, 2023b.
  34. Controllable visual-tactile synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7040–7052, 2023c.
  35. Imagebind: One embedding space to bind them all. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180–15190, 2023.
  36. Beyond flat gelsight sensors: Simulation of optical tactile sensors of complex morphologies for sim2real learning. ArXiv, abs/2305.12605, 2023.
  37. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615, 2023.
  38. Tactile image-to-image disentanglement of contact geometry from motion-induced shear. In 5th Annual Conference on Robot Learning, 2021.
  39. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play, 2023.
  40. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976–980. IEEE, 2022.
  41. Learning an action-conditional model for haptic texture generation. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 11088–11095, 2019.
  42. Learning to read braille: Bridging the tactile reality gap with diffusion models. 2023.
  43. Harold Hotelling. Relations between two sets of variates. Biometrika, 28:321–377, 1936.
  44. Mix and localize: Localizing sound sources in mixtures. Computer Vision and Pattern Recognition (CVPR), 2022.
  45. Understanding dynamic tactile sensing for liquid property estimation. ArXiv, abs/2205.08771, 2022.
  46. Fabian Hutmacher. Why is there so much more research on vision than on any other sensory modality? Frontiers in psychology, 10:2246, 2019.
  47. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
  48. Mrtnet: Multi-resolution temporal network for video sentence grounding. ICASSP, 2023a.
  49. Online distillation-enhanced multi-modal transformer for sequential recommendation. In Proceedings of the 31th ACM international conference on Multimedia, 2023b.
  50. Robotic perception of object properties using tactile sensing. ArXiv, abs/2112.14119, 2021.
  51. Vision-guided active tactile perception for crack detection and reconstruction. 2021 29th Mediterranean Conference on Control and Automation (MED), pages 930–936, 2021.
  52. Learn from incomplete tactile data: Tactile representation learning with masked autoencoders. In Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023.
  53. Reducing tactile sim2real domain gaps via deep texture generation networks. 2022 International Conference on Robotics and Automation (ICRA), pages 8305–8311, 2021.
  54. Retrographic sensing for the measurement of surface texture and shape. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1070–1077. IEEE, 2009.
  55. Microgeometry capture using an elastomeric sensor. ACM SIGGRAPH 2011 papers, 2011.
  56. Tactile sensing in dexterous robot hands - review. Robotics Auton. Syst., 74:195–220, 2015.
  57. Self-supervised visuo-tactile pretraining to locate and follow garment features. In Robotics: Science and Systems, 2023.
  58. Adam: A method for stochastic optimization. In International Conference on Learning Representation, 2015.
  59. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5:3838–3845, 2020.
  60. Pytouch: A machine learning library for touch processing. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13208–13214, 2021.
  61. Hand movements: A window into haptic object recognition. Cognitive Psychology, 19:342–368, 1987.
  62. Tutorial review haptic perception: A tutorial. 2009.
  63. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36:582–596, 2019.
  64. Sound-guided semantic image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3377–3386, 2022.
  65. In-hand manipulation of unknown objects with tactile sensing for insertion. In Embracing Contacts - Workshop at ICRA 2023, 2023.
  66. Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch. IEEE Robotics and Automation Letters, 7:9382–9388, 2022.
  67. See, hear, and feel: Smart sensory fusion for robotic manipulation. In Conference on Robot Learning, 2022.
  68. Vihope: Visuotactile in-hand object 6d pose estimation with shape completion. IEEE Robotics and Automation Letters, 8(11):6963–6970, 2023a.
  69. Dynamic network for language-based fashion retrieval. In Proceedings of the 1st International Workshop on Deep Multimodal Learning for Information Retrieval, pages 49–57, 2023b.
  70. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023c.
  71. Connecting touch and vision via cross-modal prediction. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10601–10610, 2019.
  72. Learning to identify object instances by touch: Tactile recognition via multimodal matching. 2019 International Conference on Robotics and Automation (ICRA), pages 3644–3650, 2019.
  73. David J Linden. Touch: The science of the hand, heart, and mind. Penguin Books, 2016.
  74. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
  75. Goal-driven robotic pushing using tactile and proprioceptive feedback. IEEE Transactions on Robotics, 38:1201–1212, 2020.
  76. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  77. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293–304, 2022.
  78. Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2722–2727, 2018.
  79. Paul R Manske. The sense of touch. Journal of Hand Surgery, 24(2):213–214, 1999.
  80. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475–12486, 2021.
  81. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  82. Learning sight from sound: Ambient sound provides supervision for visual learning. 2018.
  83. In-hand manipulation of unknown objects with tactile sensing for insertion. 2022.
  84. Visual-tactile multimodality for following deformable linear objects using reinforcement learning. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3987–3994, 2022.
  85. General in-hand object rotation with vision and touch. ArXiv, abs/2309.09979, 2023.
  86. Vt-clip: Enhancing vision-language models with visual-guided texts. arXiv preprint arXiv:2112.02399, 2021.
  87. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021a.
  88. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021b.
  89. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  90. Taxim: An example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters, PP:1–1, 2021.
  91. The development of embodied cognition: Six lessons from babies. Artificial life, 2005.
  92. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  93. Sound to visual scene generation by audio-to-visual latent alignment. Computer Vision and Pattern Recognition (CVPR), 2023.
  94. MidasTouch: Monte-Carlo inference over distributions across sliding touch. In Proc. Conf. on Robot Learning, CoRL, Auckland, NZ, 2022a.
  95. ShapeMap 3-D: Efficient shape mapping through dense touch and vision. In Proc. IEEE Intl. Conf. on Robotics and Automation, ICRA, Philadelphia, PA, USA, 2022b.
  96. Fast texture classification using tactile neural coding and spiking neural network. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9890–9895, 2020.
  97. Gelslim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger. 2022 International Conference on Robotics and Automation (ICRA), pages 10781–10787, 2021.
  98. Contrastive multiview coding. In European conference on computer vision, pages 776–794. Springer, 2020.
  99. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  100. Real-time soft body 3d proprioception via deep vision-based sensing. IEEE Robotics and Automation Letters, 5:3382–3389, 2019.
  101. Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7:3930–3937, 2020.
  102. Mgh: Metadata guided hypergraph modeling for unsupervised person re-identification. In Proceedings of the 29th ACM International Conference on Multimedia, pages 1571–1580, 2021.
  103. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  104. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733–3742, 2018.
  105. Ava-avd: Audio-visual speaker diarization in the wild. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3838–3847, 2022.
  106. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021a.
  107. Towards learning to play piano with dexterous hands and touch. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10410–10416, 2021b.
  108. Visual-tactile sensing for in-hand object reconstruction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8803–8812, 2023.
  109. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179–1189, 2022.
  110. Sparse and complete latent organization for geospatial semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1809–1818, 2022.
  111. Touch and go: Learning from human-collected vision and touch. Neural Information Processing Systems (NeurIPS) - Datasets and Benchmarks Track, 2022a.
  112. Generating visual scenes from touch. International Conference on Computer Vision (ICCV), 2023.
  113. Unified contrastive learning in image-text-label space. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19141–19151, 2022b.
  114. Rotating without seeing: Towards in-hand dexterity through touch. Robotics: Science and Systems, 2023.
  115. Mimictouch: Learning human’s control strategy with multi-modal tactile feedback. ArXiv, abs/2310.16917, 2023.
  116. Fully proprioceptive slip-velocity-aware state estimation for mobile robots via invariant kalman filtering and disturbance observer. arXiv preprint arXiv:2209.15140, 2022.
  117. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors (Basel, Switzerland), 17, 2017a.
  118. Connecting look and feel: Associating the visual and tactile properties of physical materials. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4494–4502, 2017b.
  119. Shape-independent hardness estimation using deep learning and a gelsight tactile sensor. In International Conference on Robotics and Automation (ICRA), 2017c.
  120. Learning rich touch representations through cross-modal self-supervision. In Conference on Robot Learning, 2021.
  121. Investigating vision foundational models for tactile representation learning. ArXiv, abs/2305.00596, 2023.
  122. Pointclip: Point cloud understanding by clip. arXiv preprint arXiv:2112.02413, 2021.
  123. Can language understand depth? In Proceedings of the 30th ACM International Conference on Multimedia, pages 6868–6874, 2022.
  124. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023.
  125. Deep supervised cross-modal retrieval. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10386–10395, 2019.
  126. Exif as language: Learning cross-modal associations between images and camera metadata. Computer Vision and Pattern Recognition (CVPR), 2023.
  127. Touching a nerf: Leveraging neural radiance fields for tactile sensory data generation. In Conference on Robot Learning, pages 1618–1628. PMLR, 2023.
  128. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. ICCV 2023, 2022.
Citations (29)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now