T-Rex: Tactile-Reactive Dexterous Manipulation

Abstract: The ability to react dynamically to tactile signals has long been considered crucial to agile human-level dexterity. Yet contemporary learning-based Vision-Language-Action (VLA) models for robotic manipulation generally either overlook the tactile modality or are limited to encoders with static cues, due in part to the scarcity of diverse training data and standardized evaluation, architectural constraints in current VLA models, and limitations of static tactile encoders. In this paper, we push the frontier of tactile-reactive manipulation by addressing all of these limitations. We propose a large-scale, 100-hour tactile-rich dataset collected via a novel, data-efficient recipe that prioritizes elementary motor primitives. To effectively exploit naturally high-frequency touch signals without sacrificing the existing capabilities of existing VLAs, we introduce a variable-rate Mixture-of-Transformers (MoT) architecture equipped with a novel temporal tactile VQ-VAE encoder. We demonstrate the effectiveness of tactile-reactive policies on 12 manipulation tasks requiring delicate force control and deformable object manipulation, achieving over 30% higher average success rate than the strongest baseline.

• The paper presents T-Rex, a novel tactile-reactive framework that combines a large, diverse dataset with a Mixture-of-Transformer-Experts model for enhanced dexterous manipulation.
• The methodology splits control into slow visuomotor planning and fast tactile refinement using asynchronous cascaded flow matching and specialized expert modules.
• Empirical evaluations demonstrate a 30% improvement over baselines, emphasizing the benefits of tactile-grounded mid-training and architectural specialization.

Tactile-Reactive Dexterous Manipulation with T-Rex

Overview and Motivation

The T-Rex framework addresses a significant gap in dexterous robotic manipulation: the lack of models capable of high-frequency, tactile-reactive control comparable to human-level agility. Most Vision-Language-Action (VLA) paradigms either disregard tactile input or utilize static, low-frequency tactile encoding, challenged by both data scarcity and unsuitable architectures. T-Rex overcomes these bottlenecks by proposing (1) a large-scale, tactile-synchronized bimanual manipulation dataset and (2) a Mixture-of-Transformer-Experts (MoT) model architecture that integrates variable-rate visuomotor planning with high-frequency tactile refinement.

The proposed T-Rex dataset comprises 100 hours of teleoperated, tactile-rich robot interaction spanning 200+ everyday objects and 22 motor primitives, designed for broad coverage and contact diversity. Data collection emphasizes compositional verb-noun pairs for motor primitives and objects, enabling scalable mid-training without the prohibitive costs associated with fully tactile-grounded pretraining.

Figure 1: Distribution statistics of the T-Rex dataset show broad object and motor primitive coverage, with a pronounced long-tail in object diversity.

Model Architecture and Tactile Integration

T-Rex leverages a Mixture-of-Transformer-Experts backbone, comprising three specialized modules:

1. Latent Expert: Encodes visual and language inputs, facilitating temporally grounded future prediction and context alignment.
2. Action Expert: Produces low-frequency action plans, denoising from random noise to a mid-trajectory intermediate action representation.
3. Tactile Expert: Consumes cached visual-language context and real-time spatio-temporal tactile tokens to asynchronously refine actions at high-frequency, yielding closed-loop tactile-reactive control.

Calculation is split into slow visuomotor planning and fast tactile refinement through asynchronous cascaded flow matching. Tactile observations are encoded using both temporal force dynamics (quantized via VQ-VAE) and spatial deformation maps (convolutional encoder), forming a compact, unified tactile token sequence for downstream action refinement.

Figure 2: The MoT backbone divides responsibilities: slow action planning from visual-language context, and high-frequency tactile refinement with spatial-temporal encoders.

Training Paradigm and Data Efficiency

T-Rex utilizes a three-stage training protocol:

1. Large-scale Human Egocentric Pre-training: Action and latent experts are pretrained on 22,889 hours of egocentric video for broad visuomotor priors.
2. Tactile-Grounded Robot Mid-training: Robot domain adaptation and tactile expert training are performed on the 100-hour T-Rex dataset, aligning pretrained priors with fine-grained contact dynamics.
3. Skill-Specific Post-training: For complex tasks, additional fine-tuning on ~100 task demonstrations is used, allowing task specialization without forfeiting tactile-reactivity.

Ablative experiments demonstrate the criticality of tactile-modality integration and the advantage brought by asynchronous refinement. The VQ-VAE-based encoding improves drift robustness and temporal contact precision. Asynchronous cascaded flow matching achieves superior performance compared to synchronous baselines, validating the decoupling of slow visuomotor and fast tactile streams.

Figure 3: T-Rex outperforms on data efficiency in the low demonstration regime when tactile-grounded mid-training is used.

Empirical Evaluation and Results

T-Rex is evaluated against six baseline architectures (ViTacFormer, RDP, Tactile-VLA, EgoScale, $\pi_{0.5}$ visual-only, and $\pi_{0.5}$ + tactile) on 12 real-world tactile-reactive manipulation tasks, including object insertion, deformation handling, force-sensitive interaction, and bimanual coordination. Metrics quantify success rates and robustness under randomized initializations.

T-Rex achieves a 30% higher average success rate over the strongest baseline, consistently outperforming both task-specific and pretrained foundation policies across all categories. Naïve conditioning of pretrained models on tactile signals (e.g., $\pi_{0.5}$ + tactile) yields degraded results, highlighting the necessity for architectural specialization in tactile integration. Failure analysis shows T-Rex’s limitations in multi-finger coordination, force regulation, and sliding misalignment, emphasizing areas for refinement in future iterations.

Figure 4: Failure case analysis reveals specific deficiencies in visual alignment, grip maintenance, force regulation, and finger-level coordination.

Robotic System and Real-World Deployment

Experiments are conducted on the Dexmate Vega-1 robot equipped with dual Sharpa Wave dexterous hands, utilizing multimodal RGB streams from head and wrist cameras as well as fingertip tactile sensors. Teleoperation is performed via Manus gloves and VIVE trackers, driving data collection and labeled demonstration acquisition.

Figure 5: Hardware setup, showing Dexterous bimanual manipulation, synchronized visual-tactile proprioception, and teleoperation stack.

Task Demonstrations

Key manipulation tasks highlight the expressive range and contact-rich complexity achieved by T-Rex, spanning page flipping, egg transfer, plate wiping, toothpaste application, cup splitting, mahjong sorting, lock opening, tablet refilling, acid-base neutralization, card extraction, poker dealing, and lightbulb screwing.

Figure 6: Sequential stages for the page flip task, requiring precise tactile-driven finger motion and closed-loop control.

Figure 7: Execution sequence for acid-base neutralization, necessitating force-sensitive aspiration, precision placement, and swirling for mixing.

Implications and Future Directions

T-Rex demonstrates the efficacy of mid-training tactile-reactive policies for scalable dexterous manipulation, unlocking data-efficient adaptation under broad object/motor primitive coverage. The asynchronous MoT architecture sets a new baseline for tactile-aware VLA models, with robust closed-loop refinement in contact-rich settings. The data-efficient training recipe—combining human video pretraining, tactile-grounded mid-training, and skill-specific finetuning—enables efficient transfer and adaptation without costly tactile pretraining.

Theoretical implications extend to multimodal policy design, emphasizing explicit frequency separation and specialized modality fusion. Practical developments anticipated include reinforcement learning for long-horizon multi-contact tasks, unified representations for heterogeneous tactile sensors, and advances in hardware for dense palm tactile sensing.

Conclusion

The T-Rex framework integrates scalable tactile-reactive dexterous manipulation via a Mixture-of-Transformer-Experts model and open tactile-synchronized dataset. Experimental results reveal substantial gains in success rate and data efficiency relative to current VLA and tactile-aware baselines. The architecture advances both practical robot manipulation capability and theoretical insights into multimodal policy optimization. Future work will focus on sensor integration, reinforcement learning for complex coordination, and broadening tactile coverage to further approach human-level manipulation proficiency.