TacVLA: Tactile-Aware Vision-Language-Action

• TacVLA is a Vision-Language-Action model that selectively integrates tactile feedback via a contact-aware gating mechanism to improve robotic manipulation.
• It employs a transformer-based multimodal fusion strategy that activates tactile input only upon detecting physical contact, ensuring robust cross-modal grounding.
• Empirical evaluations on tasks like disassembly and in-box picking demonstrate TacVLA's superior performance over vision-only baselines.

TacVLA denotes a family of Vision-Language-Action (VLA) models for robotic control enhanced by the selective integration of tactile feedback. This approach addresses the limitations of vision and language-only VLA systems in contact-rich manipulation scenarios which demand precise, real-time responses to physical interactions. TacVLA introduces a gating mechanism that activates tactile input processing only upon contact detection, enabling robust cross-modal grounding and improved manipulation performance, particularly under occlusion and fine-grained mechanical constraints (Zhang et al., 13 Mar 2026).

1. Architectural Foundation: Transformer-Based Multimodal Fusion

TacVLA builds on a standard VLA policy backbone with three principal observation streams:

• Visual stream: Two RGB cameras (front and wrist), encoded via SigLIP, produce visual token sequences $\mathbf{z}_t^{\rm vis}$.
• Language + proprioception stream: Textual instructions and proprioceptive states are concatenated and tokenized into $\mathbf{z}_t^{\rm lan+pro}$ using a PaliGemma tokenizer.
• Tactile stream: A $15 \times 8$ taxel pressure array (120 dimensions) from the end-effector is projected by a lightweight MLP to yield $N_{\rm tac} = 36$ tactile tokens, each augmented with 2D positional encodings, i.e., $$\mathbf{z}_t^{\rm tac} = \mathcal{E}_{\rm tac}({\rm tac}_t) \in \mathbb{R}^{36 \times d}$$.

After gating, the full token sequence

$$\tilde{\mathbf{z}}_t = [\mathbf{z}_t^{\rm vis} \,\Vert\, \mathbf{z}_t^{\rm lan+pro} \,\Vert\, \tilde{\mathbf{z}}_t^{\rm tac}]$$

is passed to a non-causal transformer (OpenPI base), allowing unrestricted cross-attention among modalities. The policy head predicts a continuous action sequence $\mathbf{a}_{t:t+H}$ conditioned on the fused token representation.

2. Contact-Aware Tactile Gating: Selective Modality Activation

TacVLA mitigates detrimental effects of unconditional tactile fusion by employing a contact-aware gating mechanism. At each timestep, a binary decision variable $c_t$ is computed via thresholded taxel activation:

• $c_t=1$ if the number of taxels exceeding a fixed pressure threshold $\tau$ meets or surpasses a count threshold $\mathbf{z}_t^{\rm lan+pro}$0;
• $\mathbf{z}_t^{\rm lan+pro}$1 otherwise.

Gating enforces:

• Masking: When $\mathbf{z}_t^{\rm lan+pro}$2, all tactile tokens are zeroed and masked from attention.
• Activation: When $\mathbf{z}_t^{\rm lan+pro}$3, tactile tokens are passed to the transformer unaltered.

The gating is realized both as an attention mask and as embedding-level gating:

$\mathbf{z}_t^{\rm lan+pro}$4

This mechanism ensures that tactile cues are introduced into policy inference exclusively during physical contact, preserving the structure of the transformer’s input space and preventing spurious attentional shifts outside contact phases.

3. Adaptive Multimodal Fusion in the Transformer Backbone

The concatenated and gated tokens are processed through all transformer layers with standard multi-head attention:

$\mathbf{z}_t^{\rm lan+pro}$5

The tactile token block is masked by applying a large negative bias to the corresponding rows in $\mathbf{z}_t^{\rm lan+pro}$6 when $\mathbf{z}_t^{\rm lan+pro}$7. Unmodified attention facilitates free cross-modal grounding when contact is detected. There are no model changes beyond the introduction of contact-aware gating and token concatenation.

4. Training Protocol and Loss Functions

TacVLA employs Low-Rank Adaptation (LoRA) for fine-tuning on top of a pretrained Pi0.5 VLA backbone. The only training loss is the flow-matching imitation loss:

$\mathbf{z}_t^{\rm lan+pro}$8

where $\mathbf{z}_t^{\rm lan+pro}$9 is the conditional flow predictor and $15 \times 8$0 is the instantaneous flow. No auxiliary tactile reconstruction losses are used, and tactile encoder weights are frozen. Only the LoRA adapters and policy head are updated (10k steps, AdamW).

5. Empirical Evaluation and Ablation Results

TacVLA was evaluated on the Franka Panda 7-DoF arm across contact-rich and visually challenging tasks.

Task domains:

• Constraint-locked disassembly (pressing, twisting, sliding)
• In-box picking (occluded retrieval)

Performance table:

Method	Avg Disassembly	In-Box Picking
3D Diffusion + Tactile	31.25 %	5 %
2D Diffusion + Tactile	48.75 %	0 %
Finetuned Pi0.5 (vision)	63.75 %	10 %
TacVLA	83.75 %	70 %

Under severe front-camera occlusion, TacVLA maintains 62.5% average success (vs. 30% for vision-only Pi0.5 and 45% for naive tactile fusion). Ablation shows that unconditional tactile token concatenation degrades performance (71.25%/40% compared to 83.75%/70% for TacVLA). Qualitative analysis identifies failures in baselines as spurious attention and repeatedly unstable grasps during non-contact phases.

6. Key Insights, Limitations, and Extensions

TacVLA demonstrates that effective cross-modal grounding in VLA models for manipulation is possible with a compact tactile array, low-dimensional tokenization, and hard, contact-aware gating. This design not only provides significant gains in contact-rich tasks and when visual input is unreliable, but also mitigates the challenge of distribution shift inherent in adding new sensory modalities to pretrained transformer policies.

Observed limitations include:

• Continued susceptibility to failure if visual–tactile gating does not cleanly align with actual physical contact, or if the tactile encoder itself introduces ambiguity under certain contact geometries.
• The fixed gating approach may not suffice for scenarios requiring anticipation rather than reaction; extensions may incorporate predictive or soft gating based on learned contact probabilities.

Potential future directions entail:

• Integration with richer tactile modalities (e.g., high-resolution skins, multi-finger hands)
• Adaptive, end-to-end learned gating thresholds
• Model-based force prediction for anticipatory control
• Multi-task or reinforcement-learning fine-tuning for grasp stability and long-horizon tasks

TacVLA's compact, contact-aware tactile fusion architecture forms a strong empirical and conceptual baseline for future Vision-Language-Action models seeking robustness in contact-rich and occluded environments (Zhang et al., 13 Mar 2026).