Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images (1803.03435v4)

Abstract: Estimation of tactile properties from vision, such as slipperiness or roughness, is important to effectively interact with the environment. These tactile properties help us decide which actions we should choose and how to perform them. E.g., we can drive slower if we see that we have bad traction or grasp tighter if an item looks slippery. We believe that this ability also helps robots to enhance their understanding of the environment, and thus enables them to tailor their actions to the situation at hand. We therefore propose a model to estimate the degree of tactile properties from visual perception alone (e.g., the level of slipperiness or roughness). Our method extends a encoder-decoder network, in which the latent variables are visual and tactile features. In contrast to previous works, our method does not require manual labeling, but only RGB images and the corresponding tactile sensor data. All our data is collected with a webcam and uSkin tactile sensor mounted on the end-effector of a Sawyer robot, which strokes the surfaces of 25 different materials. We show that our model generalizes to materials not included in the training data by evaluating the feature space, indicating that it has learned to associate important tactile properties with images.

• The paper introduces an encoder-decoder framework that predicts tactile properties from visual inputs without using discrete labels.
• It evaluates the model on 25 textured materials, effectively estimating parameters such as roughness, hardness, and friction.
• The research enhances robotic perception by reducing reliance on tactile sensors, paving the way for more autonomous material handling.

Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images

The paper "Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images" by Kuniyuki Takahashi and Jethro Tan addresses the challenge of estimating tactile properties using only visual data. This research promotes the concept of enabling robots to perceive tactile features like slipperiness and roughness directly from images, thereby enhancing their capability to interact effectively with various environments.

Overview of the Proposed Method

The authors propose a novel framework that leverages an encoder-decoder network structure to model the relationship between visual and tactile data. The network encodes visual information from RGB images and decodes it to predict tactile properties using latent variables. This structure eliminates the need for discrete class labels, offering a flexible approach to learn the nuanced differences in tactile attributes across various materials—the dataset includes textures of 25 different materials. The tactile data is collected using a webcam and a uSkin tactile sensor mounted on a Sawyer robot end-effector, which enables stroking motion across material surfaces to acquire data comprehensively.

Evaluation and Results

This innovative approach demonstrates its strength through the model’s capacity to generalize beyond the materials it was trained on. The suggested network equips robots with a more refined understanding of material properties in a continuous space rather than the traditional discrete classification methods. The numerical results indicate that the model effectively captures roughness, hardness, and friction, vital parameters for planning robotic actions in tactile-centric environments. Notably, materials exhibiting high friction or unique textural traits were successfully mapped to their corresponding latent variable regions, offering insights into the mapping of images to tactile features.

Implications and Future Work

The implications of this paper are notably significant for advancing robotic perception. By facilitating tactile estimation from visual input, robots can operate in real-world scenarios with reduced reliance on physical tactile sensors during task execution. This methodological shift reduces the hardware burden and enhances the versatility of robotic systems, particularly in dynamic and complex environments where tactile sensors may be impractical.

The research paves the way for future investigations into more sophisticated networks that could incorporate 3D image data, potentially enhancing the model’s capability to process visual nuances linked to tactile features. Such advancements could further improve tactile estimation accuracy and extend applications to more intricate robotic tasks in manufacturing, service industry, and beyond.

In conclusion, by circumventing the need for manual labeling and tactile sensors during runtime, this paper contributes to the development of autonomous systems capable of sophisticated material property estimations, thus improving interaction efficacy in both industrial and everyday applications.