Feeling the Unexpected: ResTacVLA for Contact-Rich Manipulation via Residual Tactile Representation

Abstract: Tactile perception is indispensable for contact-rich manipulation, yet integrating it into Vision-Language-Action (VLA) models often induces modality collapse, where high-bandwidth visual features overshadow sparse tactile cues. Inspired by Predictive Coding, a neural mechanism where the brain attenuates predictable inputs to prioritize surprising stimuli, we propose ResTacVLA. Rather than treating tactile data as raw input, we reformulate it as a Residual Tactile Representation capturing the discrepancy between visual priors and physical sensations. By filtering out visually predictable dynamics, this formulation transforms sparse tactile signals into dense, high-value information gain, thereby inherently resolving the bandwidth mismatch. These residuals are discretized through a Vector Quantized (VQ) bottleneck into Latent Contact Primitives that capture critical events missed by vision. Analogous to the neural surprise signal, we leverage the uncertainty of the visual prior to adaptively gate tactile integration, prioritizing residuals specifically during visually unreliable phases to explicitly prevent visual dominance. Experimental results show that ResTacVLA consistently outperforms all baselines on a diverse set of contact-rich manipulation tasks, while remaining robust to unexpected dynamic disturbances. Project page: https://awilekong.github.io/ResTacVLA/

• The paper introduces a novel predictive coding-inspired methodology that reformulates tactile signals as residuals to enhance sensor fusion in contact-rich manipulation tasks.
• It employs a Cross-Modal Predictor and Surprise-Aware Gate to distill discrete tactile events and inject them adaptively, boosting performance up to 86.7% in precise tasks.
• Experimental results validate significant improvements over vision-only models with a +34.6% success rate increase, demonstrating robustness under dynamic conditions.

Residual Tactile Representation for Contact-Rich Manipulation: ResTacVLA

Motivation and Problem Statement

Contact-rich manipulation tasks in robotics, such as precision insertion, surface wiping, and assembly, inherently require tactile perception to resolve fine-grained physical ambiguities and occlusions that vision-centric approaches cannot overcome. The dominant Vision-Language-Action (VLA) models prioritize high-bandwidth visual signals, relegating sparse tactile cues to ineffective auxiliary features, a phenomenon termed 'Modality Collapse'. Inspired by predictive coding mechanisms in neuroscience—which prioritize sensory surprise and attenuate predictable information—ResTacVLA introduces a new paradigm for multimodal sensor fusion that reformulates tactile signals as high-value residuals with respect to visual priors.

Figure 1: Predictive Coding-Inspired Residual Tactile Fusion in ResTacVLA. The Residual Tactile Encoder extracts high-information-gain representations by modeling the discrepancy between visual priors and physical sensations.

Methodology

Architecture Overview

The ResTacVLA pipeline explicitly models tactile surprise via two core components:

1. Cross-Modal Predictor (CMP): Pre-trained to estimate latent tactile features from wrist-camera observations, extracting residuals that quantify the deviation of physical sensations from visual expectations. These residuals are transformed into discrete Latent Contact Primitives using a Vector Quantization (VQ) bottleneck, reducing high-dimensional noise into semantically meaningful contact events.
2. Surprise-Aware Gate (SAG): Prediction uncertainty (σ_t) from the CMP activates the SAG, adaptively injecting contact primitives into the action expert during visually ambiguous phases and suppressing tactile cues in free-space, thereby preventing visual feature dominance.

   Figure 2: Overall architecture of ResTacVLA. CMP generates residual tactile primitives via VQ, and SAG gates tactile injection into flow-matching policy during critical contact events.

Residual Tactile Representation and Latent Primitives

Raw tactile images are encoded (UniT or ResNet backbone), and the CMP predicts expected tactile states based solely on visual input. The difference (residual) forms the tactile surprise signal, capturing dynamics not predictable from vision (e.g., collisions, force compliance). Vector Quantization enforces a compact, codebook-based representation, distilling the residuals into discrete tokens corresponding to semantically significant contact events.

Surprise-Aware Policy Learning

The SAG leverages CMP uncertainty as gating signal, ensuring tactile information is injected only under high-surprise phases (e.g., contact onset, occlusion). This explicit modulation is critical; naive fusion leads to performance degradation due to noise sensitivity and redundant tactile injection during free-space motion.

Experimental Protocol

Task Suite

Evaluation spans five real-world manipulation tasks designed for tactile dependence:

• Lightbulb Screwing: Detects thread engagement and rotational friction not reliably observed via vision.
• Plug Insertion and Peg-in-Hole: Require sub-millimeter accuracy under severe occlusion, necessitating tactile-guided alignment.
• Peg Transfer: Sequential, dexterous manipulation requiring high tactile sensitivity.
• Plate Wiping: Robust force control to maintain sustained contact for effective cleaning.

  Figure 3: Overview of five contact-rich manipulation tasks, each posing unique tactile perception and control challenges.

Quantitative Performance

ResTacVLA achieves average success rates of 62.8%, a +34.6% improvement over vision-only baseline ($\pi_{0.5}$), and up to 86.7% success in Peg alignment and interaction, outperforming direct tactile fusion and pretrained tactile feature baselines. Phase-wise breakdown demonstrates that vision-only policies excel in initial alignment but fail in physical interaction (e.g., Plug Insertion drops from 36.0% to 20.0%), confirming the necessity of tactile feedback for resolving ambiguities.

Interpretability and Ablations

Semantic Structure of Latent Primitives

t-SNE visualization reveals that contact primitives learned via VQ cluster distinctly according to task and interaction phase. Free-space motion representations collapse into a single compact cluster reflecting low tactile information gain, whereas physical contact phases produce task-specific clusters organized by true contact events (e.g., collision, alignment success).

Figure 4: t-SNE Visualization of Latent Contact Primitives, showing task-phase-dependent semantic clustering.

Surprise-Aware Gate Dynamics

Temporal analysis of gate value during Lightbulb Screwing illustrates adaptive gating: tactile pathway is suppressed during approach, rising sharply upon contact and thread engagement, aligning activation with phases where vision is unreliable and touch is critical.

Figure 5: Evolution of the Surprise-Aware Gate $g_t$ through task phases; contact onset triggers sharp gating activation.

Architectural Ablation

Removing VQ bottleneck (continuous residual embeddings) degrades performance (-26.7%), primarily due to increased noise sensitivity and instability in policy. Disabling SAG (fixed tactile fusion) induces trajectory drift and further degradation (-13.3%). These findings validate the role of both discrete residual contact primitives and adaptive gating in robust tactile integration.

Robustness to Dynamic Disturbances

ResTacVLA was stress-tested under:

• Initial grasp perturbations (translational/rotational noise)
• Dynamic target displacements during Peg-in-Hole
• Plate height variations for active compliance

It maintains high task success (e.g., 66.7% under dynamic displacement, 53.3%–40.0% under height variation), outperforming vision-only and naive tactile baselines, demonstrating generalization and adaptability to nonstationary physical environments.

Figure 6: Model robustness evaluation under environmental and actuation perturbations; ResTacVLA sustains high performance.

Theoretical and Practical Implications

ResTacVLA exemplifies biologically-plausible sensor fusion for generalist robot policy learning, demonstrating that predictive coding-inspired residual representations elevate tactile perception from auxiliary input to a dominant modality when physical dynamics dictate. The explicit, phase-dependent gating mechanism improves policy stability, interpretability, and robustness, enabling high-level semantic reasoning to be grounded in fine-grained tactile feedback. The architecture is agnostic to policy backbone (Diffusion Policy, $\pi_{0.5}$), suggesting broad applicability to other multimodal embodied AI systems.

Future Directions

This approach opens avenues for:

• Extension to additional sensor modalities (e.g., force/torque, audio)
• Hierarchical semantic abstraction for compositional physical reasoning
• Continual learning scenarios where residual surprise guides exploration and adaptation
• Autonomous discovery of contact primitives in larger manipulation vocabularies

Conclusion

ResTacVLA systematically resolves modality imbalance in VLA models, enabling robust, adaptive contact-rich manipulation via predictive coding-inspired residual tactile representations and phase-aware gating. The architecture achieves state-of-the-art performance and resilience to environmental disturbances, providing a principled framework for integrating tactile sensation into generalist robot policies and advancing the theoretical understanding of multimodal embodied intelligence.