TacEx: Modular Tactile Simulation

• TacEx is a modular tactile simulation framework that combines soft-body FEM and optical tactile modules to generate both mechanical deformation fields and GelSight-style images.
• It integrates specialized modules including GIPC, Taxim, and FOTS to simulate realistic gel deformation, RGB tactile imaging, and dense marker flow fields in contact-rich tasks.
• TacEx enables efficient RL training for contact-based manipulation tasks such as object pushing and lifting, offering robust performance benchmarks and insights into simulation fidelity and scalability.

TacEx is a modular tactile simulation framework designed for the simulation of the GelSight Mini visuotactile sensor in contact-rich robotic manipulation tasks. Developed within NVIDIA Isaac Sim and leveraging Isaac Lab for reinforcement learning (RL) environments, TacEx integrates high-fidelity soft-body and tactile vision modules to produce both mechanical and optical sensor data. The framework combines a soft-body finite element simulator (GIPC) and vision-based tactile simulators (Taxim and FOTS) to generate reliable gel deformation fields and plausible GelSight-style RGB images, enabling the training and evaluation of RL policies under dense, high-dimensional tactile feedback (Nguyen et al., 2024).

1. System Architecture and Module Integration

TacEx operates by tightly coupling three specialized modules within Isaac Sim’s physics and rendering engine:

• GIPC: Implements soft-body finite element modeling (FEM), solving the second-order equation $M \ddot{u} + C \dot{u} + K u = f$ for gelpad deformation under contact. It is interfaced via Python bindings, utilizes a tetrahedral mesh, and advances state via a Gauss–Newton IPC solver for inversion-free simulation under large deformations.
• Taxim: GPU-accelerated optical simulator that takes the gel surface geometry (height map) and transforms it into tactile RGB images. The process includes Gaussian pyramid smoothing, normal estimation, polynomial lookup-table mapping, and self-shadow overlays mimicking GelSight sensor imaging.
• FOTS: Computes dense marker displacement fields by applying analytical exponential-form models to the height map, accounting for normal, shear, and twist loads. The result is a 2D flow field representing marker motion.

PhysX (Isaac Sim’s built-in engine) handles rigid-body kinematics (robot, sensor housing, objects) and gelpad attachment points. The data flow synchronizes physics simulation, FEM deformation, optical rendering, and marker dynamics within each simulation timestep, outputting RGB tactile images, gel deformation fields, and updated physics states as observations for RL agents.

2. Soft-Body Gel Deformation Modeling

The core mechanical simulation in TacEx is governed by FEM as:

$M \ddot{u} + C \dot{u} + K u = f$

where $M$ is the mass matrix, $C$ the Rayleigh damping matrix, $K$ the stiffness matrix (tetrahedral elements), $u(t)$ the vertex displacement vector, and $f(t)$ external contact forces. Tetrahedral meshes are generated via Wildmeshing with tested resolutions spanning from $3,717$ vertices/$24$ tets to $66,563$ vertices/$221$ tets. Barrier-energy IPC ensures inversion-free stability during substantial deformations (validated under $>90^\circ$ twist). Material parameters—Young’s modulus $E$ (in the [$0.05$, $0.5$] MPa range) and Poisson’s ratio $\nu\approx0.45$—are selected to replicate silicone-like gels, with Rayleigh damping coefficients tuned to critical damping near contact frequencies. Domain randomization of gel parameters is employed for robust policy generalization.

3. Visuotactile Simulation Pipeline

The visuotactile simulation encompasses height-map acquisition, optical tactile rendering, and gel marker motion modeling:

• Height-Map Acquisition: Internal orthographic camera in the sensor case captures object protrusion through the semi-transparent gelpad, producing depth images translated into height maps $h(x,y)$.
• Optical Simulation (Taxim): Height maps are smoothed via Gaussian pyramids, normals $n(x,y)$ computed, and mapped through a polynomial LUT to yield RGB images $I(x,y)=P(n(x,y))$, with shadow overlays for realism.
• Marker Motion (FOTS): Indentation $\delta=h_0-h$ drives analytical exponential models for marker displacement $d(x,y)$, yielding dense flow field $F(x,y)$. Outputs are spatially projected to produce GelSight-style tactile data at marker resolution.

4. Reinforcement Learning Environment Design

TacEx provides a suite of RL environments for tactile-based manipulation:

• Task Definitions:
  • Object Pushing: Move a block to a target.
  • Object Lifting: Grasp and elevate an object using two GelSight-enabled fingertips.
  • Pole Balancing: Maintain a pole upright using tactile feedback.
• Observation Space:
  • RGB tactile image $I_{t} \in \mathbb{R}^{H \times W \times 3}$
  • Marker flow $F_{t} \in \mathbb{R}^{H \times W \times 2}$
  • Optional 3D positions/orientations of end-effector and object.
• Action Space: Continuous Cartesian velocity commands $$(\Delta x,\, \Delta y,\, \Delta z,\, \Delta \text{roll},\, \Delta \text{pitch},\, \Delta \text{yaw})$$.
• Reward Structure (Object Pushing example):

$r_t = -\|p_{obj}(t) - p_{goal}\|_2 - 0.1\|a_t\|_2$

with a sparse $+1$ bonus for goal proximity.

PPO (Proximal Policy Optimization) is used for RL training with GPU-parallelized rollouts. Domain randomization (lighting, gel materials, friction) and curriculum learning (progressive compliance increment) improve policy robustness.

5. Simulation Stability, Fidelity, and Performance Metrics

Comprehensive performance and stability benchmarks confirm the utility of TacEx for simulation-based RL:

• Soft-Body Stability: No mesh inversions under $>90^\circ$ beam twisting.
• Contact-Force Accuracy: $<$5% deviation between GIPC and PhysX penalty models for indentation.
• Visual Realism: Taxim-generated images achieve SSIM $>0.85$ against real GelSight datasets.

Performance timings (480$\times$640 RGB, 10$\times$10 markers/env) are summarized below.

Module	Time/frame	Hardware
Taxim+shadows	3–6 ms	RTX 3080 Ti
FOTS (markers)	2–5 ms	16 CPU cores
PhysX (rigid bodies)	0.01–4 ms	1–256 envs
PhysX (soft bodies)	0.05–0.45 ms	1–128 envs
GIPC (soft bodies)	25–220 ms	mesh-dependent

PPO policies typically converge within $\sim 1$M steps for pushing and balancing tasks. Sample efficiency improves by $20\%$ over rigid-only tactile baselines, attributed to richer tactile feedback. Quantitative sim-to-real transfer remains an open research direction.

6. Implementation and Workflow Integration

TacEx is distributed as an Isaac Lab extension compatible with NVIDIA Isaac Sim $\geq 2024.1$ and Isaac Lab $\geq 0.5$. After installation via pip, users instantiate the “tacex_sensor” in their USD scene and configure sensor parameters (gelpad mesh, camera intrinsics, Taxim LUT, FOTS settings) via YAML files. Taxim GPU kernels require PyTorch $1.13$, while GIPC interfaces use C++/CUDA bindings. Recommended hardware for high-throughput RL training is $\geq 12$ GB GPU memory for $>16$ parallel environments and $16$ CPU cores for marker simulation workloads.

Core repository, RL examples, and module sources are available at:

• https://sites.google.com/view/tacex
• https://git.ias.informatik.tu-darmstadt.de/tactile-sensing/taxim-gpu
• https://doi.org/10.1145/3643028

7. Limitations and Prospective Enhancements

TacEx exhibits several limitations:

• GIPC simulation scales poorly with high-resolution meshes, constraining RL batch sizes.
• Isaac Sim’s PhysX soft bodies lack static friction modeling, causing unreliable grasping in specific scenarios.
• TacEx experimental evaluation remains qualitative; rigorous sim-to-real verification is pending.

Proposed improvements include adaptive mesh refinement (coarsening far-field regions, refining near contact), incorporation of differentiable FEM and marker simulators for policy gradient optimization, extension to alternative tactile sensor modalities (e.g., GelStereo), and large-scale benchmarking to quantify the tactile domain gap.

By unifying high-fidelity soft-body, optical, and marker-based tactile simulation in a cohesive, modular pipeline, TacEx enables the prototyping, training, and evaluation of tactile-informed robotic control policies in contact-rich manipulation environments (Nguyen et al., 2024).