TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

Abstract: Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.

• The paper introduces TacVerse, a multi-sensor dataset with 106,800 tactile images from seven VBTS platforms, addressing sensor-shift challenges.
• It details three benchmark tasks—shape classification, grating classification, and force regression—and employs protocols for within-sensor, zero-shot, and few-shot evaluations.
• The study shows that MAE self-supervised pretraining enhances cross-sensor generalization, though sensor-shift remains a significant bottleneck.

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

Introduction

TacVerse presents a comprehensive, multi-sensor dataset and benchmark, specifically targeting the challenges of cross-sensor transfer in vision-based tactile perception (2606.25877). Vision-based tactile sensors (VBTSs) acquire local contact information through imaging deformation, supporting fine-grained robotic manipulation. However, substantial heterogeneity exists among VBTSs due to differences in sensing principles and hardware configurations, creating significant sensor-shift effects. Existing datasets are largely sensor-specific and fail to provide controlled, reproducible comparisons across platforms, limiting robust benchmarking and generalization studies.

TacVerse directly addresses this shortfall by delivering 106,800 tactile images from seven VBTS platforms, spanning diverse sensing principles—Intensity Mapping Method (IMM), Marker Displacement Method (MDM), and Modality Fusion Method (MFM), including hybrid configurations. The dataset is organized into three representative tasks: shape classification, grating classification, and force regression, each constructed to ensure shared task definitions and label spaces across sensors, thereby isolating sensor-based variation.

Figure 1: Overview of the seven vision-based tactile sensors in TacVerse, demonstrating substantial cross-sensor variation.

Dataset Composition and Benchmark Tasks

TacVerse covers seven VBTS platforms: GelSightNoMarker, GelSightMarker, MagicGripper, MagicTac, TacTip, ViTac, and ViTacTip. Each sensor represents either a single sensing principle or a hybrid design, enabling systematic study of sensor-shift phenomena.

The dataset supports three benchmark tasks:

• Shape Classification: 9-way recognition using 3D-printed indenters, targeting geometric contact cues.
• Grating Classification: 30-way recognition with line and dot gratings, exploring fine spatial textures and contact orientation.
• Force Regression: Ground-truth force measurement with a six-axis F/T transducer, considering normal and shear components.

These tasks encompass both discrete and continuous tactile problems, offering complementary testbeds for evaluating cross-sensor generalization.

Figure 2: Representative samples from shape, grating, and force tasks, illustrating diversity in tactile observations and testbed scope.

Experimental Protocols

TacVerse employs three evaluation protocols:

• Within-Sensor Training: Models are trained and tested on single-sensor data, establishing upper-bound performance.
• Zero-Shot Cross-Sensor Transfer: Models trained on one sensor are directly deployed to a different sensor without exposure to target data, measuring sensor-shift degradation.
• Few-Shot Target Adaptation: Source-trained models are fine-tuned on sparse, labelled samples from target sensors, probing data-efficient adaptation.

Self-supervised pretraining using Masked Autoencoders (MAE) is applied to unify tactile representations across all tasks. MAE operates by reconstructing masked image patches, capturing spatial dependencies between deformation, geometry, and sensor-specific cues.

Figure 3: TacVerse benchmark protocols and MAE-based self-supervised pretraining pipeline.

Shape Classification: Sensor-Shifts and Embedding Analysis

Exhaustive source-target heatmaps reveal strong within-sensor accuracy (up to 0.981) and F1 scores, affirming task learnability given sensor-specific cues. Zero-shot transfer between closely related sensors (e.g., GelSightMarker to GelSightNoMarker) is comparatively robust and, in a specific case, yields even higher transfer accuracy than the source domain. In contrast, sensor-shift between heterogeneous platforms causes marked degradation, demonstrating sensitivity to sensing mechanism and image formation.

t-SNE visualizations corroborate quantitative trends, showing compact class clusters for within-sensor embeddings, while cross-sensor settings result in dispersed and overlapping clusters—direct evidence of representation shift due to sensor mismatch.

Figure 4: Shape-classification heatmap, showing pronounced sensor-shift degradation except between highly similar sensors.

Figure 5: t-SNE visualisation, highlighting tighter clusters in within-sensor settings and more dispersed embeddings under sensor shift.

Grating Classification: Sensitivity to Sensor Domain

Grating classification is notably more sensitive to sensor shift. Within-sensor evaluation on GelSightMarker reaches accuracy and F1 of 0.903, but cross-sensor transfer drops sharply, with accuracy on GelSightNoMarker at only 0.248, and near-chance performance on MagicGripper and ViTacTip.

Grad-CAM analysis exposes qualitative differences in model attention. Within-sensor models focus crisply on contact regions, effectively capturing grating structure. Cross-sensor transfer leads to diffuse activations, often missing discriminative cues, underscoring the detrimental impact of sensor mismatch.

Figure 6: Grad-CAM visualisations for grating classification demonstrate weakened localisation under sensor mismatch.

Force Regression: Adaptation and Transfer Limits

Force regression exhibits severe sensor-shift effects. Within-sensor models on GelSightNoMarker achieve RMSE of 0.186 and $R^2$ of 0.590. Zero-shot transfer yields strongly negative $R^2$ values, with large prediction errors on MagicGripper and ViTacTip.

Few-shot adaptation improves performance as the proportion of labelled target data increases; for example, RMSE drops with more supervised samples and $R^2$ correspondingly increases. However, adapted models do not reach within-sensor upper bounds, indicating persistent information loss through direct transfer, and that force regression is highly sensitive to sensor structure and contact-response variations.

Figure 7: Few-shot adaptation narrows but does not close the cross-sensor gap for force regression.

Representation Study: Efficacy of MAE Pretraining

A controlled comparison of backbone and initialisation strategies across tasks demonstrates a clear advantage for vision transformers with MAE pretraining. MAE yields the most consistent improvements in shape, grating, and force tasks, outperforming both randomly initialised networks and ImageNet-pretrained ViTs. The robustness of MAE-initialised representations across sensor-task configurations points to its value as a foundation for future tactile models.

Implications and Future Directions

TacVerse validates several strong claims:

• Sensor-shift remains the dominant bottleneck in cross-sensor tactile perception, especially for texture and force estimation.
• Few-shot adaptation partially mitigates transfer degradation, but cannot fully recover within-sensor performance.
• Self-supervised MAE pretraining achieves consistent gains, making it an effective initialisation for multi-sensor tactile learning.

Practically, TacVerse establishes a controlled, reproducible benchmark for investigating sensor-agnostic representation learning and data-efficient adaptation strategies. Theoretically, it exposes the distributional fragility of tactile perception methods, motivating new approaches in sensor-invariant learning, foundation models, and multimodal integration.

Future work should expand TacVerse toward broader sensor and task coverage, more dynamic interactions, and real-world manipulation scenarios. Unified evaluation of tactile foundation models and continued investigation of sensor-invariant representation objectives are pressing research directions.

Conclusion

TacVerse sets a new standard for benchmarking cross-sensor vision-based tactile perception, providing rich, task-aligned data and unified protocols for reproducible evaluation. Empirical results confirm the severity of sensor-shift effects, the limited efficacy of direct transfer, and consistent advantages from self-supervised MAE pretraining. TacVerse enables systematic study of tactile representation learning, adaptation, and sensor invariance, and is poised to drive future advances in tactile perception and robotic manipulation.