TacSL: GPU-Accelerated Visuotactile Simulation

• TacSL is a GPU-accelerated library that simulates visuotactile sensors by integrating contact physics and image rendering, enabling realistic tactile data generation.
• TacSL employs a Kelvin–Voigt contact model with tensorized depth-to-RGB mapping, achieving over 200× speed-up compared to traditional CPU-bound methods.
• TacSL supports sim-to-real policy learning on tasks like peg insertion by combining imitation and reinforcement learning to enhance tactile-based robotic manipulation.

TacSL is a GPU-accelerated library for visuotactile sensor simulation and learning, designed to advance robotic tactile perception, sensor signal generation, and sim-to-real policy learning. Integrating seamlessly with NVIDIA’s Isaac Gym, TacSL delivers a fully parallelized pipeline for high-throughput, physically realistic visuotactile simulation and provides a comprehensive learning toolkit targeting contact-rich manipulation tasks with visuotactile feedback (Akinola et al., 12 Aug 2024).

1. Motivation and Scope

TacSL addresses three historically challenging dimensions in robotic tactile sensing: (1) interpreting complex sensor signals, (2) generating realistic visuotactile data in novel contact scenarios, and (3) enabling efficient, scalable learning of tactile-based manipulation policies. Traditional approaches to simulating visuotactile sensors—such as CPU-bound finite-element methods or per-pixel ray tracing—are computationally prohibitive, particularly when modeling realistic combinations of deformation, contact, lighting, and imaging. This bottleneck impedes large-scale policy learning and robust sim-to-real transfer, a central aim for dexterous robotic manipulation. TacSL is designed to overcome these limitations by exploiting GPU parallelism to deliver more than 200× speed-up relative to prior work, while supporting both geometric (RGB, depth) and force-field outputs on arbitrary 3D meshes.

2. Simulation Architecture and Computational Performance

TacSL's simulation pipeline decomposes tactile sensing into two tightly coupled, GPU-parallelized stages: contact/deformation computation and tactile image rendering.

Contact and Deformation

TacSL employs a Kelvin–Voigt contact model in which rigid bodies are permitted "soft" interpenetration: the contact force is computed as $f = \max(-\kappa \epsilon - c \dot\epsilon, 0)$, where $\epsilon$ is penetration depth, $\dot\epsilon$ is penetration rate, $\kappa$ is stiffness, and $c$ is damping. Implicit time-stepping with a semi-implicit Euler integrator ensures dynamic stability under stiff contacts. Each contact impulse is applied sequentially via a Gauss–Seidel substep solver, enabling robust, physically plausible deformation response.

Tactile Image Rendering

Rather than relying on computationally expensive ray tracing through elastic media, TacSL renders a depth image $I_{depth}$ from a virtual camera aligned with the real sensor lens and applies a calibrated, tensorized look-up function $F$ to map depth to RGB, yielding $I_{rgb} = F(I_{depth})$.

Force-Field Computation

On a regular sampling grid, normal and tangential (shear/friction) forces are computed in parallel, utilizing precomputed signed distance functions and analytically derived contact normals.

Performance Benchmarks

On NVIDIA RTX 3090 hardware, TacSL achieves:

• Tactile image generation: 7.3 FPS (Taxim) → 1631 FPS on 512 envs, over 200× speed-up.
• Force-field computation (10×10 grid): 3596 FPS (DefGraspSim) → 1.54M FPS on 32,768 envs, over 428× speed-up.
• At 100×100 grid resolution: 1.0×10⁵ FPS on 4096 envs, ≈ 46× speed-up. More than 70% of simulation step time is attributed to tactile computation, demonstrating the computational advantage of TacSL's GPU-centric approach (Akinola et al., 12 Aug 2024).

3. Sensor Models and Training Benchmarks

TacSL includes two distinct visuotactile sensor models based on GelSight-type devices:

• GelSight R1.5: Industrial format, producing $80\times60$ RGB images and $14\times10$ force grids.
• GelSight Mini: Compact, finger-shaped configuration.

Sensor models require volumetric elastomer meshes for collision, a thin surface mesh for rendering, and empirically calibrated contact parameters $(\kappa, c)$. Camera parameters and depth-to-RGB mappings are derived from physical calibration procedures.

Training Environments

Three contact-intensive, randomized manipulation tasks are implemented:

1. Peg Placement: Upright cylindrical peg placement without prior pose information.
2. Peg Insertion: Precision socket insertion.
3. Bolt-on-Nut Alignment: Challenging alignment and threading of bolt into a hexagonal nut.

Randomization bounds cover end-effector configuration, object position/orientation, contact properties, and joint damping, supporting robust sim-to-real transfer via domain randomization.

4. Learning Algorithms

TacSL incorporates both imitation-based and reinforcement learning frameworks, capitalizing on massive parallel data collection:

Policy Distillation (Offline and Online)

• Experts $\pi_e(a|s)$ are trained with privileged, low-dimensional state via PPO.
• Behavior cloning minimizes $$L_{BC} = \mathbb{E}_{(s,o)\sim D} [\|\pi_s(o) - \pi_e(s)\|^2]$$.
• Online DAgger mixes expert and student rollouts, with scheduling via $\beta_i$-decayed probability.

Asymmetric Actor-Critic (AAC)

• Actor $\pi_\theta(o)$ processes high-dimensional tactile and proprioceptive observations; critic $Q_\phi(s, a)$ uses low-dimensional privileged state.
• PPO surrogate loss optimizes the actor: $$L^{PPO}(\theta) = \mathbb{E}[\min(r_t(\theta)A_t, \mathrm{clip}(r_t, 1-\epsilon, 1+\epsilon)A_t)]$$, with $r_t$ and advantage $A_t$ computed from $Q_\phi$.

Asymmetric Actor-Critic Distillation (AACD)

A novel two-stage training protocol:

1. Stage 1: Expert $(\pi_e, Q_e)$ is trained on low-dimensional state space.
2. Stage 2: High-dimensional π_H(o) is initialized, with critic parameters $\phi_H \leftarrow \phi_e$, and joint policy/critic optimization proceeds. The critic is either frozen or lightly fine-tuned. Empirical results show AACD achieves up to 2× faster convergence and higher robustness under image augmentation versus standard AAC (Akinola et al., 12 Aug 2024).

5. Sim-to-Real Transfer and Empirical Evaluation

TacSL’s sim-to-real pipeline combines physics model randomization and image augmentation:

• Contact stiffness $\kappa\sim U[150,350]$ N/m, damping $c\sim U[0,1]$ N·s/m.
• Episodic RGB augmentations: geometric shifts, zoom, brightness/contrast/hue/saturation jitter, channel swaps.

Real-Robot Experiments

• Hardware: Franka Emika Panda arm, NVIDIA RTX workstation, dual GelSight sensors.
• Tasks: Peg Placement and Peg Insertion evaluated across 81 policy-trial pairs with variable socket locations and in-gripper offsets.

Zero-Shot Real-World Success Rates

Policies trained with color augmentation (ColorAug) or differential RGB (Diff+ColorAug) far surpass naïve baselines:

• Peg Placement: Vanilla 27.2%, ColorAug 87.7%, Diff+ColorAug 91.4%, Concat+ColorAug 77.9%
• Peg Insertion: ColorAug 82.7%

Policies exhibit resilience under significant physical perturbations and challenging industrial lighting scenarios, demonstrating practical sim-to-real transfer enabled by TacSL (Akinola et al., 12 Aug 2024).

6. Integration, Customization, and Extension

TacSL is distributed as a Python library for Isaac Gym. Adoption for new research involves: (1) integrating TacSL, (2) defining sensor meshes, contact parameters, and depth-to-RGB mappings, (3) selecting or constructing training tasks, (4) configuring training algorithms with appropriate randomization, (5) running parallel simulation for data collection and policy learning, and (6) directly deploying learned policies onto hardware without additional fine-tuning.

Potential Extensions

• Implementation of nonlinear contact laws (e.g., Hunt–Crossley) for improved physical accuracy.
• Learned, data-driven depth-to-RGB mappings via GANs or diffusion models.
• Augmentation with magneto-tactile or haptic-force modalities.
• Integration of advanced neural policy architectures (e.g., Transformers, diffusion-model priors).
• Expansion to dexterous multi-finger or bimanual manipulation tasks.

TacSL provides a platform for advancing tactile sensing, simulation, and manipulation policy learning—delivering end-to-end tools from sensor contact physics through robust sim-to-real deployment (Akinola et al., 12 Aug 2024).