VT-Refine: Visuo-Tactile Robotic Assembly

• VT-Refine is a visuo-tactile policy learning framework that integrates synchronized visual and tactile data for precise, contact-rich robotic assembly.
• It employs high-fidelity tactile simulation with a Kelvin–Voigt model and diffusion policy architecture to bridge the sim-to-real gap.
• Reinforcement learning fine-tuning achieves up to 40% improved assembly success in both simulated and real-world conditions.

VT-Refine is a visuo-tactile policy learning framework developed for bimanual robotic assembly tasks characterized by precise, contact-rich manipulation and challenging sim-to-real transfer. The approach couples multimodal demonstrations, high-fidelity tactile simulation, and reinforcement learning (RL) fine-tuning to address the gap between diverse, adaptive human assembly strategies and standard behavioral cloning in robotics. VT-Refine exploits synchronized visual and tactile feedback, GPU-accelerated simulation of tactile sensors, and policy-gradient RL to enhance robustness and generalization in complex assembly environments, with demonstrated gains in both simulated and real-world conditions (Huang et al., 16 Oct 2025).

1. Architecture and Procedural Stages

VT-Refine consists of three tightly interlinked components:

1. Real-World Demonstration and Behavioral Cloning Pre-training: Teleoperated human demonstrations are collected with synchronized visual (ego-centric camera point clouds) and tactile (piezoresistive sensor array) modalities. The dataset size is typically limited (e.g., 30 episodes), reflecting practical constraints in human demonstration diversity and optimality. The initial policy is trained using behavioral cloning within a diffusion policy architecture based on denoising diffusion probabilistic models (DDPMs).
2. Simulated Digital Twin and Tactile Modeling: The pre-trained policy is imported into a simulated digital twin environment with realistic geometry and tactile feedback, leveraging GPU-accelerated simulation (TacSL integrated with Isaac Gym). Tactile signals are generated using a penalty-based spring-damper model employing the Kelvin–Voigt viscoelastic formulation for normal force prediction at each tactel.
3. Fine-Tuning via Reinforcement Learning and Sim-to-Real Transfer: Policy refinement is performed using a sparse-reward RL approach—specifically, Diffusion Policy Policy Optimization (DPPO), which builds on PPO principles and propagates reward signals across the denoising steps inherent to the diffusion policy. This process enriches the policy with micro-adjustment behaviors crucial for fault-tolerant assembly and improves robustness against environmental uncertainty before deployment on physical robots.

2. Visuo-Tactile Representation and Diffusion Policy Learning

The multimodal input to VT-Refine integrates:

• Visual Point Cloud: $$P_t^{\text{visual}} \in \mathbb{R}^{N_\text{vis}\times4}$$, representing spatial coordinates and visual cues from the ego-centric camera.
• Tactile Point Cloud: $$P_t^{\text{tactile}} \in \mathbb{R}^{N_\text{tac}\times4}$$, with each tactel contributing position and normal force data.
• Proprioceptive State: Joint encoders provide robot configuration states.

The observations are concatenated and encoded by a PointNet, with a one-hot channel identifying point modality. The diffusion policy maps these observations to action chunks, conditioned over multiple denoising steps (typically $T=100$), producing control signals aligned to the robot’s bimanual kinematics.

Behavioral cloning is performed despite demonstration suboptimality; the diffusion process captures both coarse task structures and primitives for initial policy deployment.

3. Tactile Sensor Design and Simulation Fidelity

FlexiTac sensors, custom piezoresistive arrays with typical $12\times32$ spatial resolution and $2~\text{mm}$ separation, are chosen for high fidelity and simulation tractability. The hardware consists of triple-layer stacks incorporating a piezoresistive film and flexible printed circuit (FPC) layers to maintain spatial accuracy.

The salient properties are:

• Normal Force Sensing: Sensors are calibrated to output only normal force signals, which aids realistic simulation and avoids the complexity of shear or fine texture modeling.
• Kelvin-Voigt Model for Simulation: Each tactel’s interaction is computed by

$f_n = -(k_n d + k_d \dot{d}) n$

where $d$ is penetration calculated via signed distance fields (SDF), $\dot{d}$ is the relative velocity, and $k_n, k_d$ are the stiffness and damping coefficients.

The sim-to-real gap is narrowed by matching simulated force–response curves to DMA-measured hardware responses, enabling effective transfer of policies across domains.

4. Simulation-Driven RL Fine-Tuning

Post-pretraining, the policy is fine-tuned within simulation using DPPO under a sparse reward structure ($+1$ for successful assembly, $0$ otherwise). Key technical aspects include:

• Policy Update: Only the critic is initialized randomly; the actor is initialized from the diffusion policy.
• Exploratory Micro-adjustments: RL rollouts induce behaviors like wiggle-and-dock, essentially iterative re-positionings in response to misaligned contacts—critical for sub-millimeter tolerance insertion tasks.
• Parallelization: GPU acceleration supports both simulation (TacSL) and network training (denoising steps, encoding), allowing large-scale fine-tuning and rapid experience diversification.

Empirical results show a gain of up to 40% in assembly success for visuo-tactile policies compared to vision-only, both in simulation and real deployment, with minor degradation ($5\%–10\%$) attributed to sim-to-real sensor and kinematic mismatch.

5. Experimental Evaluation and Data Efficiency

Experimental ablation demonstrates the framework’s data efficiency. With as few as 10 demonstrations, RL fine-tuning substantially improves performance; however, saturation occurs at 30 demonstrations (with near-perfect post-RL success), and further increases yield diminishing returns.

Evaluation is carried out in both tabletop and semi-humanoid settings, focused on plug-and-socket insertions with sub-2 mm clearances. Simulation and real-world policy outcomes are compared, verifying the robust transfer and adaptability conferred by VT-Refine’s architecture.

Stage	Input Modalities	Primary Learning Step	Gains (Simulation)	Gains (Real World)
Demonstration	Visual + Tactile	Behavioral Cloning	–	–
Sim RL Fine-Tune	Visual + Tactile (Sim)	RL (DPPO)	+30% assembly	+30% (after transfer)
Ablation (10→30)	Demos count	RL improvement	Steady up to 30 demos	Saturation >30

6. Technical Specifics and Implementation Resources

Critical technical parameters from the implementation:

• Diffusion Steps: $T=100$ denoising steps per policy rollout.
• Observation Encoding: PointNet encoder with five-channel point flags (differentiates visual/tactile inputs).
• Simulation Engine: TacSL + Isaac Gym for tactile simulation (Kelvin–Voigt force calculation per tactel).
• Policy Optimization: DPPO updates actor using PPO with critic initialization, KL regularization for stability.
• Hyperparameters: Full details, hardware fabrication guidelines, and tutorial resources are available at [https://binghao-huang.github.io/vt_refine/].

7. Context, Applicability, and Limitations

VT-Refine is designed for contact-rich assembly settings where multimodal tactile feedback and sim-to-real transfer are central challenges. The framework is distinct from purely vision-based or end-to-end imitation learning approaches due to its multi-stage blending of human priors, realistic simulation, and reinforcement learning. It addresses the limitations of suboptimal and sparse demonstrations by leveraging simulation for experience augmentation.

A plausible implication is that VT-Refine’s modularity—particularly in tactile simulation and RL integration—can be extended to other assembly domains requiring robust contact handling and fault tolerance. However, real-world sensor alignment and kinematic mismatches remain sources of transfer error, and further refinement in tactile point calibration or domain adaptation could improve final robustness.

8. Summary

VT-Refine advances bimanual assembly policy learning by integrating multimodal demonstrations, high-fidelity sim RL fine-tuning, and accurate tactile sensor modeling. The approach achieves marked improvements in robustness and generalization for precise contact-rich assembly and narrows the sim-to-real transfer gap. The technical resources and code base are made available for practitioners seeking to adapt or extend this methodology (Huang et al., 16 Oct 2025).