Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation

Abstract: As an essential modality for dexterous and contact-rich tasks, tactile sensing provides precise force feedback that cannot be reliably inferred from vision. However, limited by hardware and data collection systems, existing datasets with tactility remain small in scale and narrow in contact coverage. Meanwhile, Vision-Language-Action (VLA) models with tactile modality are constrained on dynamics-agnostic post-training, which limits the performance ceiling on downstream tasks. In this paper, we present H-Tac, a large-scale tactile-action dataset with 160-hour egocentric human videos containing more than 300 tasks and 135k episodes. Building upon this, we propose Transferable Tactile Pre-Training (TTP), a system of tactile-based pre-training on human data for fine-grained robotic tasks. To bridge the gap between humans and robots, we use unified tactile and action spaces throughout the pre-training and post-training phases, preserving prior knowledge during human-to-robot transfer. By leveraging a tactile expert for future tactile prediction, our framework explicitly models the contact dynamics and precise physical interactions. Extensive experiments in simulation and on real robots demonstrate that our model achieves superior performance, exhibiting robust generalization and fine-grained manipulation capabilities. TTP paves the way for scalable tactile pre-training via human-to-robot transfer.

• The paper introduces TTP, a tactile pre-training system that leverages 160 hours of human-centric tactile data from over 300 manipulation tasks.
• It employs a multimodal transformer with unified tactile and action spaces, integrating vision, language, and tactile sensing for robust skill transfer.
• Experimental results show superior performance in precise, fine-grained manipulation and zero-shot generalization compared to leading VLA baselines.

Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation

Introduction and Motivation

The integration of tactile sensing with vision and language has remained a missing ingredient in generalizing robotic manipulation to contact-rich, dexterous tasks. Visual perception, while potent, is inherently limited in occluded or ambiguous interaction scenarios, whereas tactile sensing delivers precise force feedback vital for contact-intensive operations. However, the expense, heterogeneity, and laborious nature of collecting large-scale, high-quality tactile data directly from robotic platforms have historically hindered progress. The presented work addresses this challenge by assembling extensive human-centric tactile datasets and introducing a pre-training system—Transferable Tactile Pre-Training (TTP)—designed specifically for scalable human-to-robot skill transfer.

Figure 1: Overview of the Transferable Tactile Pre-Training (TTP) system.

Large-Scale Human-Centric Tactile Dataset: H-Tac

The H-Tac suite constitutes a foundational asset, encompassing three primary subcomponents: HOI-Tac, DeskTask-Tac, and InternData-Tac. Collectively, H-Tac comprises 160 hours of egocentric human vision-tactile-action data, covering over 300 diverse manipulation tasks and 135,000 episodes. The datasets employ unified representations to facilitate seamless transfer between human and robotic embodiments:

• HOI-Tac: Synthesized by projecting contact labels from multiple public datasets onto a 351-taxel UniTacHand UV space, capturing a diverse range of hand-object, bimanual, hand-face, and hand-scene interactions.
• DeskTask-Tac: Real-world bimanual manipulation captured via specialized hardware, including synchronized RGB video, pose estimation, and dense tactile annotation.
• InternData-Tac: Synthetic, robot-agnostic tactile traces generated via InternDataEngine with force and geometric proximity information projected onto the same unified tactile representation.

  Figure 2: The H-Tac datasets: (a) HOI-Tac, (b) DeskTask-Tac, (c) InternData-Tac, with 160 hours of data spanning 300+ tasks and 135k+ episodes.

  

  Figure 3: Data collection system for DeskTask-Tac, integrating multi-view vision and tactile glove streams.

Statistical analysis reveals that the right hand and finger taxels dominate contact events, aligning with natural human manipulation patterns. Instruction language statistics show "grasp" is the overwhelmingly most common action primitive, reflecting task distributions.

Figure 4: Dataset statistics: mean tactile intensities on hand surfaces and distribution of language instruction prefixes.

TTP Model Architecture

TTP is implemented atop BeingH-0.5, an advanced VLA foundation model. The architecture consists of multimodal transformers with three expert modules: an understanding expert (vision and language), an action expert, and a novel tactile prediction expert. Unified action (200-D) and tactile (351-D per hand) spaces are used throughout pre-training (human data) and post-training (robot data), ensuring architectural and representational consistency for effective cross-embodiment transfer.

Figure 5: Training architecture of TTP with an understanding expert, action expert, and tactile expert sharing a unified action and tactile space.

The training employs a VQA-style flow matching framework, jointly predicting future action trajectories and tactile sequences. To stabilize the policy under distributional shifts during rollout, the Tactile-Action Manifold-Preserving Gating (MPG) module adaptively conditions decoding based on reliability signals computed from the alignment of current context to learned action/tactile manifolds.

Tactile-Driven Pre-Training and Generalization

Large-scale pre-training on H-Tac enables TTP to learn rich, aligned representations for vision, language, action, and tactile modalities. The model captures fine-grained physical interaction dynamics essential for robust, dexterous behavior. Qualitative visualization confirms that TTP can accurately predict hand motion and tactile feedback not just for in-distribution examples, but also for out-of-distribution and inpainted scenes, indicative of strong generalization.

Figure 6: TTP predictions of tactile feedback and hand motion generalize to out-of-distribution and inpainted input scenes.

Experimental Evaluation

Simulation Benchmarks

Comprehensive evaluation was performed on the LIBERO, LIBERO-plus, and RoboCasa simulation suites, using tactile proxy signals where no ground truth tactile data existed. Despite the additional token-level burden from tactile prediction, TTP demonstrated competitive or superior performance compared to top-performing VLA baselines, especially in long-horizon and zero-shot generalization splits.

Real-World Robotic Manipulation

A diverse set of real-robot experiments validated TTP's practical impact. Tasks covered a span from fine-grained (e.g., peeling, vase wiping) and contact-rich (e.g., pick-and-place with fragile objects, paper folding) to vision-ambiguous (e.g., plug insertion under occlusion) scenarios, realized with multiple robot arms and end effectors with different tactile sensor arrays.

Figure 7: Hardware configurations for real-robot experiments, including both single and dual-arm, and various hand and gripper embodiments.

Figure 8: TTP consistently exceeds alternative baselines in precise, fine-grained manipulation across real-robot tasks.

TTP produced large, consistent gains in metrics such as length of peeled skin (>20 cm, a margin far above competitive methods), successful undamaged pick-and-place with fragile objects, and superior performance under occlusion or changing object materials. Out-of-distribution (OOD) tests further demonstrated TTP's robustness to novel objects (e.g., carrots, cucumbers, unseen paper), scene configurations, and sensor interruptions.

Figure 9: In-distribution real-robot task demonstrations across peeling, wiping, folding, picking, and plugging tasks.

Figure 10: OOD real-robot demonstrations show TTP's generalization to unseen objects, materials, and significant perceptual variation.

Ablation studies confirmed the necessity of both the tactile expert and MPG module, as well as the benefits of scaling dataset size. Notably, TTP without tactile pre-training suffered noticeable degradation, confirming the value of human-centric tactile knowledge as a transfer scaffold.

Implications and Future Directions

TTP evidences that high-fidelity human-to-robot skill transfer is achievable when tactile information is captured and structured at scale. The unified tactile/action backbone and explicit tactile dynamic modeling facilitate a new class of VTLA models that achieve robust, generalizable contact-rich manipulation across radically different embodiments. The demonstrated architectural generality paves the way towards universal robot learning agents able to seamlessly incorporate new sensor modalities.

Key directions for further research include (i) extending unified tactile/action representations to new sensor and effector designs, (ii) scaling up the variety and realism of human-centric data for broader task domains, and (iii) fusing tactile with other modalities such as proprioceptive force, temperature, or auditory feedback, towards holistic embodied intelligence.

Conclusion

This work establishes a new paradigm for tactile-informed, scalable robotics pre-training. By leveraging an extensive human-centric tactile dataset and a dual-expert transformer with unified representations, TTP achieves significant gains in precision, robustness, and generalization for fine-grained manipulation. The approach sets a strong precedent for further integration of tactile sensing in foundation models for robot learning, illuminating scalable pathways for endowing robots with complex, human-like interaction capabilities.

(2607.01067)