Toward Spatial Temporal Consistency of Joint Visual Tactile Perception in VR Applications (2312.16391v2)
Abstract: With the development of VR technology, especially the emergence of the metaverse concept, the integration of visual and tactile perception has become an expected experience in human-machine interaction. Therefore, achieving spatial-temporal consistency of visual and tactile information in VR applications has become a necessary factor for realizing this experience. The state-of-the-art vibrotactile datasets generally contain temporal-level vibrotactile information collected by randomly sliding on the surface of an object, along with the corresponding image of the material/texture. However, they lack the position/spatial information that corresponds to the signal acquisition, making it difficult to achieve spatiotemporal alignment of visual-tactile data. Therefore, we develop a new data acquisition system in this paper which can collect visual and vibrotactile signals of different textures/materials with spatial and temporal consistency. In addition, we develop a VR-based application call "V-Touching" by leveraging the dataset generated by the new acquisition system, which can provide pixel-to-taxel joint visual-tactile perception when sliding over the surface of objects in the virtual environment with distinct vibrotactile feedback of different textures/materials.
- E. Steinbach, M. Strese, M. Eid, X. Liu, A. Bhardwaj, Q. Liu, M. Al-Ja’afreh, T. Mahmoodi, R. Hassen, A. El Saddik et al., “Haptic codecs for the tactile internet,” Proceedings of the IEEE, vol. 107, no. 2, pp. 447–470, 2018.
- J. Song, J. H. Lim, and M. H. Yun, “Finding the latent semantics of haptic interaction research: A systematic literature review of haptic interaction using content analysis and network analysis,” Human Factors and Ergonomics in Manufacturing & Service Industries, vol. 26, no. 5, pp. 577–594, 2016.
- Q. Tong, W. Wei, Y. Zhang, J. Xiao, and D. Wang, “Survey on hand-based haptic interaction for virtual reality,” IEEE Transactions on Haptics, 2023.
- J. M. Romano and K. J. Kuchenbecker, “Creating realistic virtual textures from contact acceleration data,” IEEE Transactions on Haptics, vol. 5, no. 2, pp. 109–119, 2012.
- M. Strese, J.-Y. Lee, C. Schuwerk, Q. Han, H.-G. Kim, and E. Steinbach, “A haptic texture database for tool-mediated texture recognition and classification,” in 2014 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE) Proceedings, 2014, pp. 118–123.
- J. Jiao, Y. Zhang, D. Wang, X. Guo, and X. Sun, “Haptex: A database of fabric textures for surface tactile display,” in 2019 IEEE World Haptics Conference (WHC), 2019, pp. 331–336.
- H. Culbertson, J. J. López Delgado, and K. J. Kuchenbecker, “One hundred data-driven haptic texture models and open-source methods for rendering on 3d objects,” in 2014 IEEE Haptics Symposium (HAPTICS), 2014, pp. 319–325.
- S. Luo, W. Yuan, E. Adelson, A. G. Cohn, and R. Fuentes, “Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 2722–2727.
- J. Kirsch, A. Noll, M. Strese, Q. Liu, and E. Steinbach, “A low-cost acquisition, display, and evaluation setup for tactile codec development,” in 2018 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), 2018, pp. 1–6.
- M. Strese, C. Schuwerk, and E. Steinbach, “Surface classification using acceleration signals recorded during human freehand movement,” in WHC, 2015.
- H. Culbertson, J. Unwin, and K. J. Kuchenbecker, “Modeling and rendering realistic textures from unconstrained tool-surface interactions,” ToH, 2014.
- J. Sinapov, V. Sukhoy, R. Sahai, and A. Stoytchev, “Vibrotactile recognition and categorization of surfaces by a humanoid robot,” ToR, 2011.
- B. A. Richardson and K. J. Kuchenbecker, “Improving haptic adjective recognition with unsupervised feature learning,” in ICRA, 2019.
- P. K and S. Chaudhuri, “Enhancing haptic distinguishability of surface materials with boosting technique,” in 2022 IEEE Haptics Symposium (HAPTICS), 2022, pp. 1–6.
- Y. Gao, L. A. Hendricks, K. J. Kuchenbecker, and T. Darrell, “Deep learning for tactile understanding from visual and haptic data,” in ICRA, 2016.
- M. Strese, C. Schuwerk, A. Iepure, and E. Steinbach, “Multimodal feature-based surface material classification,” ToH, 2017.
- M. Strese, L. Brudermueller, J. Kirsch, and E. Steinbach, “Haptic material analysis and classification inspired by human exploratory procedures,” ToH, 2019.
- Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on pattern analysis and machine intelligence, vol. 22, no. 11, pp. 1330–1334, 2000.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.