Heterogeneous Tactile Transformer

Abstract: Tactile sensors are inherently heterogeneous: a model trained on one sensor cannot be directly used on another, which limits learning contact-rich manipulation policies from diverse tactile data at scale. To bridge this gap, we propose the Heterogeneous Tactile Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors. HTT consists of sensor-specific encoders and a shared transformer trunk, and is pretrained with per-modality masked reconstruction together with cross-modal alignment between paired sensors. Pretraining uses our novel Heterogeneous Paired Tactile (HPT) dataset, containing 1.6M synchronized paired frames across four vision- and array-based tactile sensors. Across distinct tactile perception and real-world manipulation tasks, HTT is shown to learn transferable representations that adapt to new tasks and previously unseen sensors. Dataset, code, and model checkpoints will be released upon publication at https://jxbi1010.github.io/htt-gh-page/.

• The paper introduces a unified tactile transformer that employs self-supervised pretraining on a novel 1.6M-sample heterogeneous tactile dataset.
• It aligns optical and taxel sensor modalities using a tailored MAE-based architecture, achieving notable improvements in classification and force estimation metrics.
• The approach enhances cross-sensor transfer learning and enables robust manipulation policies in real-world tasks such as slip detection and precise grasping.

Heterogeneous Tactile Transformer: A Unified Framework for Cross-Sensor Tactile Representation Learning

Introduction

Tactile sensing, a fundamental modality for robotic manipulation in contact-rich tasks, has long faced challenges in scalable representation learning due to the inherent heterogeneity among tactile sensors. Optical sensors (e.g., GelSight, 9DTact) capture spatially-vivid but temporally-limited images of surface deformations, whereas array-based (taxel) sensors (e.g., Xela uSkin, TAC-02) deliver high-frequency, force-sensitive distributed data but at lower spatial resolution and with different data formats. This heterogeneity has historically impeded the development of sensor-agnostic, reusable tactile representations—resulting in models with limited cross-device generalization and hampering the formulation of broadly applicable manipulation policies.

The "Heterogeneous Tactile Transformer" (HTT) proposes a cross-sensor representation learning framework designed to bridge these divides through self-supervised pretraining on multimodal tactile data. The introduction of the Heterogeneous Paired Tactile (HPT) dataset—with 1.6M temporally synchronized, paired interactions from both optical and taxel modalities—and a custom MAE-based transformer architecture forms the core of this effort. The method aims for robust alignment of disparate sensor modalities within a shared latent space, preserving beneficial sensor-specific inductive biases while enabling transferable perception and policy learning.

Figure 1: HTT pretraining pipeline: sensor-specific encoders process modality-specific data, mapped through a shared transformer trunk; self-supervised objectives align representations via masked reconstruction and cross-sensor prediction, enabling downstream transfer to novel sensors and tasks.

Dataset Construction and Sensor Heterogeneity

Current tactile datasets rarely cover the multimodal regime, focusing predominantly on (simulated or real) optical sensors, with few providing synchronized cross-family data. HTT addresses this by building HPT using a Universal Manipulation Interface (UMI), enabling paired acquisition from four distinct sensors—two optical (GelSight Mini, 9DTact) and two array-based (TAC-02, Xela uSkin)—mounted in modular shells for simultaneous measurement during robot-object interaction.

The collection spans unscripted object contact episodes (for pretraining and object classification), controlled force probing with synchronized 6-DoF F/T ground truth (force estimation), and slip regimes with post-hoc class labeling based on change-point detection of friction coefficient dynamics (slip detection). Notably, the slip dataset is heavily imbalanced, with rare static/incipient events, posing a particular challenge for learning robust representations.

Figure 2: Data acquisition setup, example tactile frames, and dataset statistics—showcasing force/sheer ranges and label imbalance for slip events.

Figure 3: The four heterogeneous sensors used in HPT pretraining: GelSight Mini, 9DTact (optical), TAC-02, Xela uSkin (array-based).

Heterogeneous Tactile Transformer (HTT) Framework

HTT builds on the Masked Autoencoder (MAE) paradigm, incorporating specialized architectural and training considerations for the multi-modality context.

• Sensor-Specific Encoders: Each sensor's raw signal—image frames for optical, time-series for array-based—is patchified and embedded via tailored encoders (e.g., ViT backbone for images; transformer for time series).
• Shared Transformer Trunk: Encoded tokens from all modalities are projected to a common dimension and propagated through a unified transformer, producing a shared latent space.
• Self-Supervised Pretraining: Training optimizes a composite loss with two objectives:
  • MAE-style Masked Reconstruction: Each modality reconstructs its masked input patches from the trunk output, enforcing high-fidelity, sensor-specific encoding.
  • Bidirectional Cross-Sensor Alignment: For each sensor pair, a cross-attention predictor is trained to reconstruct masked tokens of one sensor from the other's full embedding plus the visible tokens. This aligns modalities in latent space without collapsing sensor-specific features, enforced via stop-gradient tricks and scheduled loss weighting.

This carefully decoupled, phased optimization preserves modality-specific embeddings initially before strong cross-modal alignment, promoting both intra-modality informativeness and inter-modality transfer.

Experimental Results

Tactile Perception: Object Classification, Force Estimation, and Slip Detection

Across all four sensors and three tasks, HTT-pretrained models consistently outperform both from-scratch baselines and prior pretraining (T3, SITR) methods—especially for force-sensitive and cross-modal transfer tasks. Notably, for object classification, HTT improves optical sensor accuracy by 4–5 points over MAE, and by 13–17% over the strongest published baselines. For force estimation and slip detection, HTT delivers the lowest MAE (normal/shear/3D) and the highest macro-F1, achieving substantial gains in slip classification for rare classes, where previous models collapsed to predicting only the dominant sliding regime.

Crucial findings:

• Cross-modal alignment is especially beneficial for force- and slip-sensitive tasks, which require integrating both spatial and force cues, underlining the necessity of paired, heterogeneous pretraining for generalizable tactile backbones.
• Optical-only pretraining transfers poorly to novel optical or array sensors in force-grounded settings, further motivating the cross-modal approach.

  Figure 4: Representative samples for optical encoder input—raw and background-subtracted processed frames for diverse objects and force tasks, illustrating the deformation cues leveraged by the encoders.

Real-World Manipulation: Policy Learning with Unseen Sensors

Deployments of HTT representations on a Sharpa hand (not present in pretraining) show that policies leveraging HTT-derived embeddings dramatically outperform those based only on raw joint positions (qpos) or per-sensor force vectors (wrench). On the toy screw task, HTT achieves 95% success—maintaining grip and completing full rotations—while tactile-naive and wrench-only policies stall or lose grip. For grasping tofu, HTT reduces slip failures by a third compared to the wrench baseline and avoids crushing events entirely, validating the richer and more actionable contact embeddings.

Figure 5: Policy results from real-world tasks—grasp tofu and toy screw—demonstrate superior task completion rates when using HTT representations vs. conventional tactile or proprioceptive features.

Simulated Manipulation: Sensor-Agnostic Embedding Transfer

Evaluation on ManiFeel simulated benchmark tasks with both tactile RGB and force field modalities confirms that HTT-trained backbones readily transfer to new simulated sensors and boost control learning in unseen settings, matching or exceeding other state-of-the-art tactile pretraining models.

Implications and Future Directions

HTT provides strong empirical evidence that foundation models for tactile representation are feasible if supported by sufficiently broad and well-aligned multi-sensor data. The approach demonstrates the utility of paired, heterogeneous pretraining for cross-sensor generalization in both perception and policy learning, offering a modular pathway for new tactile modalities to inherit robust, transferrable embeddings with minimal adaptation.

Practical impact lies in enabling robust tactile-driven policies across the rapidly-evolving sensor ecosystem, paving the way for universal tactile backbones similar in spirit to vision or language foundation models. Theoretically, HTT advances multi-modal representation learning by offering a scalable architecture for aligning modalities with highly divergent input statistics and semantics.

Key open questions remain:

• Extending to additional sensor modalities beyond optical and array-based (e.g., magnetic, piezoresistive, fluidic) will challenge and further test the limits of shared-latent alignment.
• More nuanced spatial alignment and correspondence modeling (across sensors without explicit spatial calibration) represents a rich area for future refinement.
• Exploring additional supervisory signals—e.g., multi-task, vision-language grounding—could further enrich and regularize tactile representations for embodied AI.

Conclusion

The Heterogeneous Tactile Transformer introduces a principled route to scalable, transferable tactile backbone models for robotics. By leveraging a large paired dataset, self-supervised cross-sensor alignment, and transformer architectures, HTT advances the field toward universal, sensor-agnostic tactile representations—achieving strong empirical results in diverse perception and manipulation tasks, including zero-shot transfer to previously unseen sensors.