---
title: 'TacVerse: Cross-Sensor Tactile Perception Benchmark'
url: https://www.emergentmind.com/topics/tacverse
type: topic
---

# TacVerse: Cross-Sensor Tactile Perception Benchmark

Searching arXiv for the TacVerse paper and closely related tactile-perception context.
TacVerse is a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. It targets vision-based tactile sensors (VBTSs), which infer contact geometry and force-related cues by imaging deformation through an internal camera, and it is explicitly designed to study how generalisation changes across sensor designs. The resource contains 106,800 tactile images from seven VBTSs, supports three downstream tasks—shape classification, grating classification, and force regression—and evaluates three regimes: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation [2606.25877].

## 1. Benchmark scope and sensor coverage

TacVerse spans seven vision-based tactile sensors distributed across three sensing principles: IMM (Intensity Mapping Method), MDM (Marker Displacement Method), and MFM (Modality Fusion Method), including hybrid configurations. This multi-sensor construction is the central design choice of the benchmark: it makes cross-sensor transfer a first-class evaluation target rather than an incidental by-product of single-sensor learning [2606.25877].

| Sensor | Design | Images |
|---|---|---:|
| GelSightNoMarker | IMM | 16 917 |
| GelSightMarker | IMM + MDM | 15 487 |
| MagicGripper | IMM + MDM + MFM | 16 892 |
| MagicTac | IMM + MDM + MFM | 12 496 |
| TacTip | MDM | 11 000 |
| ViTac | IMM + MFM | 11 000 |
| ViTacTip | MDM + MFM | 23 008 |

The sensor inventory is not merely a scale increase. It creates controlled variation across sensing principles and hybrid combinations, which in turn exposes sensor shift directly. A plausible implication is that TacVerse is useful not only for supervised tactile perception, but also for isolating which downstream objectives are comparatively invariant to sensing principle and which remain tightly coupled to sensor-specific observation statistics.

## 2. Tasks, labels, and data curation

TacVerse supports three downstream tasks. Shape classification contains 9 classes and 30 094 images. Grating classification contains 30 classes—line and dot gratings with widths 0.3–1.75 mm—and 40 509 images. Force regression contains 36 197 images with continuous $(F_x,F_y,F_z)$ labels obtained from a 6-axis F/T transducer [2606.25877].

The curation and split protocol is deliberately constrained. Temporally adjacent frames with SSIM $\ge 0.99$ are removed. For shape classification, frames are grouped by contact trial, stratified by sensor and shape, and split chronologically into 60/20/20 train/val/test with no trial overlap. For grating classification and force regression, the unit is trial or trajectory, again with a chronological 60/20/20 split. All images are resized to $224 \times 224$ px [2606.25877].

These decisions matter for interpretation. Chronological splitting and no-trial-overlap reduce trivial leakage, while the SSIM filtering step prevents excessive redundancy from temporally adjacent frames. This suggests that the benchmark is intended as a reproducible testbed for transfer and adaptation rather than a pure data-collection exercise.

## 3. Evaluation settings, metrics, and optimization

TacVerse defines three evaluation settings. The first is within-sensor training, where train, validation, and test are all drawn from the same sensor. The second is zero-shot cross-sensor transfer, where a model is trained on a source sensor and evaluated directly on a target sensor with no target labels. The third is few-shot adaptation, used for force regression only, where the source-trained model is fine-tuned on $0.5\%$, $1\%$, $2.5\%$, $5\%$, or $10\%$ of target-sensor training data, with validation on the target validation set and final evaluation on the target test set [2606.25877].

For classification, TacVerse reports accuracy and macro-averaged precision, recall, and F1, with equal weighting over classes:
$$
\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\hat y_i=y_i\}
$$
$$
\mathrm{Prec}_c = \frac{\mathrm{TP}_c}{\mathrm{TP}_c+\mathrm{FP}_c},\quad
\mathrm{Rec}_c = \frac{\mathrm{TP}_c}{\mathrm{TP}_c+\mathrm{FN}_c},\quad
\mathrm{F1}_c = \frac{2\,\mathrm{Prec}_c\,\mathrm{Rec}_c}{\mathrm{Prec}_c+\mathrm{Rec}_c}
$$
$$
\mathrm{Prec} = \frac1C\sum_c \mathrm{Prec}_c,\;
\mathrm{Rec} = \frac1C\sum_c \mathrm{Rec}_c,\;
\mathrm{F1} = \frac1C\sum_c \mathrm{F1}_c
$$

For regression, it reports MAE, MSE, RMSE, and $R^2$:
$$
\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^N |y_i - \hat y_i|
$$
$$
\mathrm{MSE} = \frac{1}{N}\sum_{i=1}^N (y_i - \hat y_i)^2
$$
$$
\mathrm{RMSE} = \sqrt{\mathrm{MSE}}
$$
$$
R^2 = 1 - \frac{\sum_i (y_i-\hat y_i)^2}{\sum_i (y_i-\bar y)^2}
$$

Optimization is fixed across the main supervised experiments: AdamW, batch size 128, learning rate $1\times10^{-4}$, weight decay $1\times10^{-4}$, cosine decay with warmup, gradient clipping at $1.0$, early stopping patience $=10$, 30 epochs, and seed 42. The use of a uniform optimization stack helps attribute performance differences to sensor and task factors rather than ad hoc tuning [2606.25877].

## 4. Cross-sensor generalization behavior

The core empirical finding is asymmetry between within-sensor competence and cross-sensor transfer. Within-sensor performance is strong across tasks, indicating that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, degrades substantially, and the severity of that degradation depends strongly on the task [2606.25877].

For shape classification, representative within-sensor results are TacTip with $\mathrm{Acc}=0.547$ and $\mathrm{F1}=0.506$, GelSightMarker with $\mathrm{Acc}=0.955$ and $\mathrm{F1}=0.950$, and MagicTac with $\mathrm{Acc}=0.862$ and $\mathrm{F1}=0.855$. Zero-shot transfer shows that some closely related sensor pairs remain robust: GelSightMarker $\rightarrow$ GelSightNoMarker reaches $\mathrm{Acc}=0.981$ and $\mathrm{F1}=0.981$. By contrast, more dissimilar pairs fall below $0.2$ in both accuracy and F1, with TacTip $\rightarrow$ others around $0.11$ [2606.25877].

For grating classification, the transfer problem becomes much sharper. Using GelSightMarker as source, within-sensor performance is $\mathrm{Acc}=0.903$ and $\mathrm{F1}=0.903$, but transfer to GelSightNoMarker drops to $\mathrm{Acc}=0.248$ and $\mathrm{F1}=0.238$, to MagicGripper to $\mathrm{Acc}=0.054$ and $\mathrm{F1}=0.013$, and to ViTacTip to $\mathrm{Acc}=0.041$ and $\mathrm{F1}=0.006$. The reported best cross-sensor transfer for grating is GelSightMarker $\rightarrow$ GelSightNoMarker, while the worst transfer is to MagicGripper and ViTacTip, where performance approaches chance [2606.25877].

Force regression is even more sensitive to sensor shift. With GelSightNoMarker as source, within-sensor performance is $\mathrm{MAE}=0.126$, $\mathrm{MSE}=0.035$, $\mathrm{RMSE}=0.186$, and $R^2=0.590$. Transfer to GelSightMarker gives $\mathrm{MAE}=0.287$, $\mathrm{MSE}=0.126$, $\mathrm{RMSE}=0.348$, and $R^2=-0.014$; transfer to MagicGripper gives $\mathrm{MAE}=0.766$, $\mathrm{MSE}=0.954$, $\mathrm{RMSE}=0.976$, and $R^2=-6.046$; transfer to ViTacTip gives $\mathrm{MAE}=0.863$, $\mathrm{MSE}=1.631$, $\mathrm{RMSE}=1.277$, and $R^2=-9.058$. The best cross-sensor force transfer is GelSightNoMarker $\rightarrow$ GelSightMarker, whereas transfer to MagicGripper and ViTacTip is the worst, with highly negative $R^2$ values [2606.25877].

Taken together, these results establish a task hierarchy under sensor shift. Shape classification exhibits moderate robustness; grating classification and force regression suffer drastic drops. The reported summary characterizes these patterns as accuracy reductions from approximately $0.90 \rightarrow 0.25 \rightarrow 0.05$ for grating, and RMSE roughly doubling or tripling with $R^2$ becoming strongly negative for force. No statistical significance tests were reported, but the shifts are consistently large across multiple source-target pairs. A common misconception in tactile benchmarking is that strong within-sensor learning implies deployable cross-sensor generalization; TacVerse directly separates these regimes and shows that the implication does not hold in general.

## 5. Few-shot adaptation and self-supervised representation learning

Few-shot adaptation is evaluated only for force regression. The protocol fine-tunes a source-trained ViT plus head on $\{0.5\%,1\%,2.5\%,5\%,10\%\}$ of target training data using the same optimizer settings and early stopping on target validation data. The reported outcome is monotonic but incomplete recovery: as the number of shots increases, RMSE steadily decreases toward the within-sensor bound and $R^2$ rises from strongly negative values toward $0$, yet even at $10\%$ of target data the adapted model remains below within-sensor performance, leaving a residual transfer gap [2606.25877].

TacVerse also includes a self-supervised pretraining study using MAE (Masked Autoencoder). Pretraining is performed on all shape, grating, and force training images for 200 epochs with batch size 256, learning rate $1.5\times10^{-4}$, weight decay $0.05$, and mask ratio $0.5$, followed by transfer through a $\mathrm{ViT}_{\mathrm{MAE}}$ backbone plus task head [2606.25877].

The representative within-sensor comparisons are heterogeneous but systematic. For shape on GelSightMarker, ResNet-18 obtains $0.966$, $\mathrm{ViT}_{\mathrm{Rand}}$ $0.422$, $\mathrm{ViT}_{\mathrm{ImgNet}}$ $0.958$, and $\mathrm{ViT}_{\mathrm{MAE}}$ $0.848$. For shape on MagicGripper, the corresponding accuracies are $0.931$, $0.716$, $0.908$, and $0.949$. For grating on GelSightMarker they are $0.808$, $0.552$, $0.887$, and $0.944$. For force on GelSightNoMarker, the RMSE values are $0.258$, $0.335$, $0.186$, and $0.170$. The benchmark summary therefore characterizes MAE initialization as providing the best or second-best performance across most task-sensor pairs, especially for fine-grained grating and force [2606.25877].

These results support two distinct conclusions. First, limited supervised target data helps but does not remove sensor shift. Second, self-supervised pretraining on TacVerse produces a versatile tactile representation with consistent downstream gains, making representation quality a central variable in cross-sensor tactile perception rather than a secondary implementation choice.

## 6. Limitations, interpretation, and projected extensions

The benchmark states several limitations explicitly. Interactions are controlled lab interactions only. The protocols for grating and force use fixed-source settings. There is no comparison to specialized tactile foundation models. These constraints delimit the scope of the reported results: TacVerse is a controlled benchmark for transfer analysis, not a claim of exhaustive coverage over tactile sensing conditions [2606.25877].

The stated future directions follow directly from those constraints. They include broader sensor and task coverage, dynamic interactions, real-world manipulation scenarios, sensor-invariant representation learning, more data-efficient adaptation methods, and systematic evaluation of emerging tactile foundation models under the same benchmark. This suggests that TacVerse is positioned less as a terminal leaderboard and more as infrastructure for a broader research program on cross-sensor tactile generalization [2606.25877].

In that role, the benchmark’s main contribution is methodological clarity. By combining 106,800 images, 7 sensors, 3 tasks, and 3 protocols in a single controlled setup, it offers a reproducible testbed for studying sensor shift, adaptation, and self-supervised learning in vision-based tactile perception. TacVerse thus offers a controlled, reproducible testbed for advancing cross-sensor vision-based tactile perception and transfer learning [2606.25877].

Source: https://www.emergentmind.com/topics/tacverse