Self-supervised perception for tactile skin covered dexterous hands (2505.11420v1)

Abstract: We present Sparsh-skin, a pre-trained encoder for magnetic skin sensors distributed across the fingertips, phalanges, and palm of a dexterous robot hand. Magnetic tactile skins offer a flexible form factor for hand-wide coverage with fast response times, in contrast to vision-based tactile sensors that are restricted to the fingertips and limited by bandwidth. Full hand tactile perception is crucial for robot dexterity. However, a lack of general-purpose models, challenges with interpreting magnetic flux and calibration have limited the adoption of these sensors. Sparsh-skin, given a history of kinematic and tactile sensing across a hand, outputs a latent tactile embedding that can be used in any downstream task. The encoder is self-supervised via self-distillation on a variety of unlabeled hand-object interactions using an Allegro hand sensorized with Xela uSkin. In experiments across several benchmark tasks, from state estimation to policy learning, we find that pretrained Sparsh-skin representations are both sample efficient in learning downstream tasks and improve task performance by over 41% compared to prior work and over 56% compared to end-to-end learning.

Self-supervised Perception for Tactile Skin Covered Dexterous Hands

In this paper, the authors introduce Sparsh-skin, a pre-trained encoder designed to leverage magnetic tactile sensors distributed across the fingertips, phalanges, and palm of a dexterous robot hand. This research tackles the inherent limitations of magnetic skin sensors and aims to advance tactile sensing capabilities in robotic hands through self-supervised learning.

Overview of Sparsh-skin

Sparsh-skin represents a paradigm shift from traditional vision-based tactile sensors. While vision-based sensors provide high-resolution outputs that are human-interpretable, they are constrained by their bulky form factor and limited bandwidth. Conversely, magnetic tactile sensors promise rapid response through lower dimensionality and flexible form factors that can be adapted to complex embodied structures, such as multifinger robot hands. However, challenges with interpreting magnetic flux, hysteresis, and calibration have restricted their widespread adoption.

The Sparsh-skin framework is composed of three key facets:

1. Self-supervised Learning: The encoder extracts meaningful tactile embeddings from unlabeled data, specifically targeting hand-object interactions to refine manipulation tasks. The model employs self-distillation using a classification objective, bypassing the need for task-specific training data, which is often difficult to obtain in tactile applications.
2. Comprehensive Tactile Dataset: A four-hour dataset of various interactions was created using an Allegro hand equipped with Xela uSkin sensors to train Sparsh-skin. This dataset captures atomic manipulation actions, including squeezing, sliding, rotation, and pressing.
3. Performance Improvement and Sample Efficiency: Sparsh-skin showcases a significant performance boost, improving task efficiency by 41% compared to prior work and 56% compared to end-to-end learning. Such improvements validate the efficacy of self-supervised learning for tactile-based robotic systems.

Empirical Evaluation

The authors executed several experiments to validate Sparsh-skin’s capability across different tasks: force estimation, joystick state estimation, pose estimation, and policy learning through plug insertion.

• Force Estimation: The encoder significantly outperformed baseline models in predicting both normal and shear forces applied to the robot hand’s palm. Its ability to retain performance even with limited labeled data underscores its sample efficiency.
• Joystick State Estimation and Pose Estimation: Sparsh-skin showcased robustness in predicting joystick states and object poses, effectively handling slip accumulation under sensors. Notably, Sparsh-skin excelled in rotation tracking, a task inherently complex due to torsion effects.
• Policy Learning for Plug Insertion: The visuo-tactile policies trained using Sparsh-skin representations surpassed end-to-end models, demonstrating higher success rates in real-world deployment scenarios.

Discussion on Limitations

While Sparsh-skin offers promising advancements, there are discernible limitations in model design and evaluation:

• Temporal Correlation: Currently, Sparsh-skin implicitly handles temporal dynamics. Future iterations may benefit from explicitly modeling temporal changes to align tactile data sequences more seamlessly with downstream tasks.
• Generalization of Manipulation Policies: Addressing the potential overfitting to specific tactile signatures remains a critical challenge in achieving policy generalization over diverse tactile scenarios.

Theoretical and Practical Implications

The introduction of Sparsh-skin heralds new avenues for enhancing full-hand tactile perception in robotics. Theoretically, the model sets the stage for extensive exploration into self-supervised tactile representation learning, potentially catalyzing the development of foundation models in this domain. Practically, Sparsh-skin’s performance improvements suggest a pathway towards more proficient and versatile robotic systems equipped with tactile sensors, advancing applications in both industrial settings and personal assistive devices.

In conclusion, Sparsh-skin articulates a compelling case for the maturation of tactile sensing technologies in robotics, exemplifying how self-supervision and magnetic skin sensors can enhance dexterous manipulation and expand the utility of tactile data in complex, real-world tasks.