---
title: 'TacSL Benchmark: Visuotactile Simulation'
url: https://www.emergentmind.com/topics/tacsl-benchmark
type: topic
---

# TacSL Benchmark: Visuotactile Simulation

Searching arXiv for TacSL benchmark and closely related papers to ground the article.
TacSL is a GPU-based visuotactile sensor simulation and learning library that also functions as a benchmark suite for contact-rich robotic manipulation, particularly peg placement, peg insertion, and bolt-on-nut alignment tasks in Isaac Gym [2408.06506]. It standardizes sensor models, contact-intensive environments, learning setups, and evaluation procedures for studying tactile image policies, multimodal visuotactile policies, and sim-to-real transfer. In later work, TacSL is also used explicitly as a standard insertion benchmark for evaluating visuo-tactile fusion strategies under randomized, contact-rich conditions, with the same insertion task setup and success criteria adopted from the original framework [2602.13689].

## 1. Definition and benchmark scope

TacSL, pronounced “taxel,” is presented as a library for visuotactile sensor simulation and learning built on top of NVIDIA’s Isaac Gym [2408.06506]. Its benchmark is not limited to a single task or metric. Rather, it consists of a standardized combination of sensor models, contact-rich manipulation tasks, learning algorithms, and evaluation procedures. The stated goals are to make tactile simulation fast enough for large-scale on-policy RL and online distillation, to provide a standardized suite of tasks and sensors for comparing tactile learning methods, and to enable end-to-end tactile image policies for contact-rich manipulation to transfer from simulation to the real world.

The benchmark comprises four tightly coupled elements. First, it includes visuotactile sensor models, specifically GelSight-like sensors, with both tactile RGB images and normal/shear force fields. Second, it includes manipulation tasks that require fine contact reasoning: Peg Placement, Peg Insertion, and Bolt-on-Nut Alignment for Screwing. Third, it provides multiple learning setups, including behavior cloning, DAgger-style online distillation, asymmetric actor-critic RL, and Asymmetric Actor-Critic Distillation. Fourth, it defines an evaluation protocol centered on success rates in simulation, real-world success rates under zero-shot sim-to-real transfer, simulation speed, and modality and algorithm ablations [2408.06506].

A later paper frames TacSL more narrowly as a standard insertion benchmark for contact-rich manipulation with vision and touch, emphasizing its role as a source of environment configuration, sensor setup, evaluation criteria, and a baseline reference for fusion strategies [2602.13689]. In that usage, the benchmark is explicitly associated with a plug or peg insertion task, randomized poses and contact parameters, and comparative evaluation of wrist vision, tactile sensing, contact-force signals, and visuo-tactile fusion methods.

## 2. Task suite and randomized problem structure

TacSL’s core benchmark tasks are contact-rich assembly-style manipulations performed with a parallel-jaw gripper equipped with visuotactile sensing on each finger [2408.06506]. The main tasks are Peg Placement and Peg Insertion, with Bolt-on-Nut Alignment positioned as a challenge task.

Peg Placement requires the robot to place a cylindrical rod upright on a flat support plate so that the peg remains stable after release. Difficulty is induced by peg misalignment in the gripper, randomization of peg position, and randomization of the initial end-effector pose. The policy does not receive direct access to object pose in the gripper and must infer peg orientation and pose from contact geometry.

Peg Insertion requires a cylindrical peg to be inserted into a cylindrical socket fixed to a table. The initial peg pose in the gripper is randomized in position and orientation; the socket position is randomized in front of the robot; a noisy estimate of socket pose is provided; and physics parameters such as compliant stiffness, compliant damping, and robot joint damping are randomized. This produces a task in which visual localization is imprecise, occlusion occurs near contact, and tactile sensing becomes important for precise alignment and smooth insertion.

The Bolt-on-Nut Alignment task requires a bolt held in the gripper to be aligned with a threaded hole in a nut so that an automatic screwing primitive succeeds. The paper identifies it as more sensitive to rotational misalignment than Peg Insertion, and it is treated as a challenge task rather than the main quantitative focus [2408.06506].

For the insertion setting reused in later work, the environment is characterized as a robustified insertion benchmark rather than a single easy peg-in-hole configuration [2602.13689]. The reported randomization ranges include end-effector position with \(X \in [0.4, 0.6]\) m, \(Y \in [-0.1, 0.1]\) m, and \(Z \in [0.1, 0.2]\) m; end-effector Euler orientation with \(X \in [3.04, 3.24]\) rad, \(Y \in [-0.1, 0.1]\) rad, and \(Z \in [-1.0, 1.0]\) rad; socket position with \(X \in [0.4, 0.6]\) m, \(Y \in [-0.1, 0.1]\) m, and \(Z \in [0.0, 0.02]\) m; peg-in-gripper \(Z\) offset in \([-0.0125, 0.0125]\) m; peg rotation about the \(X\)-axis in \([-0.628, 0.628]\) rad; socket XYZ perceived-location noise in \([-0.005, 0.005]\) m on each coordinate; compliance stiffness in \([150, 350]\) N/m; compliance damping in \([0.0, 1.0]\) N/(m/s); and joint damping noise in \([-1.5, 1.5]\) N/(m/s).

These randomization schemes define TacSL’s benchmark difficulty. They enforce generalization across initial conditions, contact parameters, and observation uncertainty rather than permitting memorization of a fixed insertion geometry. A plausible implication is that TacSL measures not merely control quality but also robustness to uncertainty at the interface of vision, touch, and contact dynamics.

## 3. Sensor models, observations, and simulation machinery

TacSL’s benchmark is organized around visuotactile sensing rather than purely kinematic state estimation [2408.06506]. It supports GelSight R1.5 and GelSight Mini models, each specified by a visual mesh, a collision mesh, soft contact parameters, camera pose and intrinsics, and a calibrated lookup table mapping depth to RGB. The simulator outputs both tactile RGB images and tactile force fields, including normal and shear components at sampled surface points.

The underlying simulation is implemented inside Isaac Gym with almost all computation on the GPU. Physics uses a GPU-accelerated soft contact model; tactile images are generated via depth rendering followed by a GPU-side LUT; and tactile force fields are computed with GPU kernels using precomputed SDFs. TacSL’s soft contact model is based on a Kelvin-Voigt formulation, with normal force
\[
f = \max(-\kappa \epsilon - c \dot{\epsilon}, 0),
\]
and an implicit discrete-time formulation used for stability with stiff springs. Tactile images are produced by rendering a depth image and applying a calibrated mapping
\[
I_{\text{rgb}} = F(I_{\text{depth}}).
\]
Tactile force fields are computed from interpenetration and tangential motion using penalty-based expressions for normal and shear forces.

In the original benchmark, high-dimensional observation channels include tactile RGB images of size \(80 \times 60 \times 3\) per finger, tactile force fields of size \(14 \times 10 \times 3\) per finger, and a wrist RGB camera of size \(64 \times 64 \times 3\), alongside reduced-state and privileged-state features [2408.06506]. In the later insertion-fusion study, the TacSL instantiation uses a wrist-mounted RGB camera at \(64 \times 64 \times 3\), tactile arrays of size \(32 \times 32 \times 3\) per finger with channels \((f_x, f_y, f_z)\), and an optional privileged \(3 \times 1\) contact force vector [2602.13689]. The later paper also describes the visual encoder as a 3-layer CNN followed by Spatial SoftArgMax and describes the tactile preprocessing as residual-force computation relative to calibrated references or zero for symmetric objects.

TacSL’s simulation speed is a central benchmark attribute. For tactile image generation, TacSL reports 1631 FPS at 512 parallel environments versus 7.28 FPS for Taxim on the stated comparison hardware, corresponding to over \(200\times\) faster performance [2408.06506]. For tactile force fields, reported throughput reaches 1,541,043 FPS at 32768 parallel environments for a \(10 \times 10\) taxel grid, with a speedup of approximately \(428\times\) relative to the cited CPU baseline, and 103,493 FPS at 4096 environments for a \(100 \times 100\) grid, with a speedup of approximately \(46\times\). These figures define TacSL not only as a benchmark specification but also as an infrastructure benchmark for scalable tactile RL.

## 4. Learning formulations and policy training

TacSL’s benchmark is explicitly designed to compare multiple policy-learning paradigms under a shared task and sensor setup [2408.06506]. The principal methods are policy distillation, asymmetric actor-critic, and Asymmetric Actor-Critic Distillation.

In policy distillation, a teacher policy \(\pi_e(a|s)\) is trained on privileged low-dimensional state using PPO, and a student policy \(\pi_s(a|o)\) is trained from realistic observations using either offline behavior cloning or online DAgger-style aggregation. In asymmetric actor-critic, the actor consumes realistic or partial observations while the critic consumes privileged simulator state, providing value estimates that are more informative than those obtainable from partial observations alone. In AACD, training is staged: a low-dimensional actor-critic expert is first trained on privileged state; then a new high-dimensional actor is initialized for realistic observations while reusing the learned critic as a pretrained critic, either frozen or fine-tuned.

The benchmark’s action space is a 6-D relative end-effector pose command with maximum position displacement per axis of 0.01 m and maximum orientation displacement per axis of 0.05 rad, executed via a task-space impedance controller at 60 Hz [2408.06506]. PPO is used with standard hyperparameters, including clip ratio 0.2, \(\gamma = 0.99\), and \(\lambda = 0.95\). In the later TacSL insertion study, the policy is likewise a Gaussian policy with \(\mathbb{R}^6\) action outputs, trained with PPO using learning rate \(10^{-4}\), rollout length 512, batch size 512, PPO epochs 4, \(\gamma = 0.99\), \(\lambda_{\text{GAE}} = 0.95\), clip ratio 0.2, entropy coefficient 0, and value loss coefficient 2 [2602.13689].

The original benchmark also defines a task reward for peg tasks as
\[
R_{\text{task}} = r_{\text{keypoint}} - r_{\text{action}} - r_{\text{contact}},
\]
where the terms respectively penalize keypoint misalignment, action magnitude, and large contact forces [2408.06506]. By contrast, the later insertion-fusion study states that TacSL’s insertion success is modeled with a sparse reward,
\[
r_t =
\begin{cases}
1, & \text{if insertion is successful at time } t,\\
0, & \text{otherwise},
\end{cases}
\]
and that the policy maximizes the expected discounted sum of this reward while using TacSL’s success criteria [2602.13689]. This suggests that TacSL supports more than one reward formulation across task instantiations and research uses, while retaining a stable benchmark identity through task definitions, sensing assumptions, and success-based evaluation.

## 5. Evaluation protocol and empirical performance

TacSL’s primary benchmark metric is success rate [2408.06506]. In simulation, success rate is computed over 1024 randomized episodes per policy per condition; in real-world evaluation, success is reported over 81 episodes. For insertion in the later fusion study, success rate is defined as
\[
\text{Success rate} = \frac{\# \text{successful insertions}}{\# \text{total trials}} \times 100\%,
\]
and is reported as mean \(\pm\) standard deviation over three random seeds [2602.13689]. The later study also reports steps to success as an auxiliary efficiency metric.

The benchmark includes evaluation across sensing modalities and learning algorithms. On Peg Insertion in simulation, TacSL reports that privileged state with DAgger or AACD reaches approximately 95–97% success, reduced state alone is poor at approximately 5–9%, reduced state plus tactile images reaches approximately 82–91%, reduced state plus wrist reaches approximately 93–97%, and reduced state plus tactile and wrist reaches up to approximately 94% [2408.06506]. The paper interprets this as evidence that both touch and vision improve performance over reduced state, and that combined sensing tends to yield the best performance and sample efficiency.

Real-world zero-shot transfer is another core benchmark dimension. For Peg Placement, TacSL reports 27.2% success for a Vanilla Color tactile-image policy trained without augmentation, 87.7% for ColorAug, 91.4% for Diff+ColorAug, and 77.9% for Concat+ColorAug, all over 81 trials [2408.06506]. For Peg Insertion, a ColorAug policy trained in simulation achieves 82.7% success, corresponding to 67/81 successful trials. These results ground TacSL’s claim to be a sim-to-real benchmark rather than a simulation-only benchmark.

The later visuo-tactile fusion study uses TacSL’s insertion setup to compare observation configurations more finely [2602.13689]. The reported success rates are summarized below.

| Configuration | Success rate | Notes |
|---|---:|---|
| Privileged | 96.74 ± 1.63 % | Compact state only |
| Privileged + contact forces | 98.96 ± 0.83 % | Highest reported baseline |
| Tactile only | 91.41 ± 5.51 % | Reduced sensing |
| Wrist only | 93.23 ± 2.00 % | Vision only |
| Wrist + contact forces | 96.09 ± 1.41 % | Privileged visuo-force |
| Naïve Fusion | 92.97 ± 1.41 % | Simple concatenation |
| Gated Fusion | 94.53 ± 2.73 % | No symmetry regularization |
| CMT Fusion | 96.22 ± 0.98 % | No symmetry regularization |
| Gated Fusion + symmetry regularization | 95.05 ± 1.76 % | \(\lambda_{\text{sym}} = 1\) |
| CMT + symmetry regularization | 96.59 ± 2.11 % | Main reported model |

These results are used to support two conclusions about TacSL as a benchmark. First, tactile sensing supplies information that vision alone cannot recover reliably near contact, especially under occlusion and micro-misalignment. Second, the benchmark is sensitive to fusion design: naïve visuo-tactile concatenation yields negligible or negative gains, while structured self- and cross-attention with symmetry-based regularization approaches privileged performance [2602.13689]. A plausible implication is that TacSL is diagnostically useful not only for ranking policies but also for distinguishing between competing theories of multimodal fusion in contact-rich control.

## 6. Benchmark role, later extensions, and limitations

TacSL is positioned against prior tactile simulators such as Taxim, TACTO, and the force-field framework of Xu et al. (2023), as well as against broader visuomotor benchmarks such as RoboNet and RLBench [2408.06506]. Its distinguishing features are GPU-scale visuotactile simulation, support for arbitrary triangle meshes through SDF-based force computation, explicit integration of tactile images and force fields, contact-intensive assembly-style tasks, and demonstration of zero-shot sim-to-real transfer. The later visuo-tactile fusion paper further positions TacSL as the canonical testbed for insertion with vision and touch, especially for studying the gap between realistic sensing and privileged sensing [2602.13689].

The benchmark is also notable for the way later work inherits its evaluation framework. The fusion study states that it follows the evaluation framework established in TacSL and adopts the same insertion task setup and success criteria, while using the TacSL insertion task configuration in IsaacGymEnvs. In that study, TacSL is not redefined from scratch; instead, it serves as an established benchmark environment whose sensor modalities, randomization protocol, and success-based evaluation are already standardized. This continuity is a strong indicator of benchmark maturity.

Several limitations are explicitly stated in the original TacSL work [2408.06506]. The contact model is linear Kelvin-Voigt rather than a nonlinear alternative such as Hunt-Crossley. The depth-to-RGB mapping is a hand-calibrated polynomial LUT, with future work suggested around learned generative mappings. Sim-to-real experiments focus on tactile image modality rather than transferring force-field modality. The policy architectures are CNN+LSTM+MLP rather than more expressive architectures such as transformers or diffusion-based policies. The task set is contact-rich but limited primarily to peg and bolt assembly tasks, with future directions including dexterous in-hand manipulation, deformable object manipulation, and more complex industrial assembly sequences.

These limitations delimit the scope of the TacSL benchmark rather than undermining it. Within its intended domain, TacSL standardizes a reproducible and computationally scalable setting for evaluating visuotactile sensing, multimodal fusion, contact-rich policy learning, and sim-to-real transfer. Later results on symmetric plug insertion further suggest that TacSL is especially effective as a stress test for whether tactile information is used in a principled way, rather than merely appended to visual features [2602.13689].

Source: https://www.emergentmind.com/topics/tacsl-benchmark