SimShear: Tactile Sim-to-Real Pipeline

• SimShear is a sim-to-real tactile control pipeline that leverages learned shear-conditioned image translation to synthesize realistic shear deformations without explicit dynamic modeling.
• It integrates a shear-conditioned U-Net GAN (shPix2pix) with a Gaussian-Density Neural Network, achieving superior MAPE and SSIM metrics compared to conventional pix2pix methods.
• The pipeline bridges the gap between simulated tactile depth images and real tactile feedback, enabling robust tactile tracking and collaborative manipulation in diverse contact scenarios.

SimShear is a sim-to-real pipeline for tactile control that uses shear information without explicitly modeling shear dynamics in simulation. It was introduced for vision-based tactile servoing in settings where lateral contact deformation is essential but conventional rigid-body simulators do not represent it directly. The method combines simulated tactile depth images, an explicit shear representation, a shear-conditioned image-translation model called shPix2pix, and a downstream pose/shear estimator so that controllers can be trained in simulation and transferred to real robots (Freud et al., 28 Aug 2025).

1. Problem formulation and scope

The central problem addressed by SimShear is the mismatch between real tactile sensing and rigid-body simulation. In the reported setup, simulators such as PyBullet can represent contact depth or indentation but not the shear deformations that arise from lateral motion across a contact surface. This omission is consequential for tasks involving dynamic object interactions, slip prevention, or continuous surface following, because shear carries information about tangential loading and relative lateral displacement (Freud et al., 28 Aug 2025).

SimShear addresses this by replacing explicit shear physics with a learned sim-to-real translation stage. Rather than requiring a simulator to produce sheared tactile images natively, the pipeline renders tactile images without shear, augments them with a separate shear signal, and learns to synthesize tactile images that visually encode shear deformation. This is a distinct design choice: the pipeline is intended to enable true sim-to-real transfer while avoiding both expensive FEM-style modeling and real-to-sim translation at runtime (Freud et al., 28 Aug 2025).

The system was demonstrated on two tasks. The first is tactile tracking, in which a follower arm tracks a surface moved by a leader arm. The second is collaborative co-lifting, in which two arms jointly hold an object while the leader follows a prescribed trajectory. In both cases, the follower relies on tactile feedback that includes shear-sensitive information (Freud et al., 28 Aug 2025).

2. Data representation and simulation pipeline

The SimShear pipeline operates on tuples consisting of a real tactile image, a simulated tactile image, and a shear vector. The simulated tactile image is rendered from a rigid-body physics simulator and represents contact geometry without explicit shear deformation. The shear vector supplies the missing tangential information needed to disambiguate real tactile outcomes generated by different lateral motions (Freud et al., 28 Aug 2025).

Simulation is performed in Tactile Gym 2.0 built on PyBullet, where the tactile images are depth maps. Shear is captured separately as a 4D vector matching the 4-DoF robotic arm, encoding both positional and rotational displacements, $\vec{s} = [s_x, s_y, s_z, s_{R_z}]$. This representation makes the shear input explicit rather than implicit in image appearance alone (Freud et al., 28 Aug 2025).

The physical platform uses two Dobot MG400 4-axis desktop robotic arms. A leader arm moves the object, while a follower arm is equipped with a marker-based, vision-based TacTip sensor with 331 pins. Both arms execute smooth trajectories using low-level servo control at approximately $10\,\mathrm{Hz}$. The reported dataset contains 5000 training tuples and 2000 validation tuples; collection on the physical robot required 130 minutes, whereas the corresponding simulation data were gathered in 1 minute (Freud et al., 28 Aug 2025).

A key conceptual point is that the simulated image by itself is not sufficient. The same simulated depth image can correspond to many real tactile images with different shear deformations. This creates a one-to-many mapping for standard image-to-image translation, and SimShear resolves that ambiguity by conditioning the translation on the explicit shear vector (Freud et al., 28 Aug 2025).

3. shPix2pix and shear-aware state estimation

The generative core of SimShear is shPix2pix, described as a shear-conditioned U-Net GAN. The baseline reference is vanilla pix2pix, which uses a U-Net generator with skip connections and a convolutional discriminator. In shPix2pix, the shear vector is concatenated to the encoded latent representation at the U-Net bottleneck, and this augmented representation is passed through a fully connected layer with ReLU before decoding. The purpose of this conditioning is to make the mapping from simulated tactile image plus shear to real tactile image effectively one-to-one (Freud et al., 28 Aug 2025).

Training uses paired simulated image, shear, and real tactile image data for 100 epochs with batch size 16 and the Adam optimizer at learning rate $0.0001$, with early stopping. The objective combines a pixelwise term and adversarial supervision, $$L = \lambda_1 \cdot L_{\mathrm{MAPE}}(\mathrm{img}_{\mathrm{gen}}, \mathrm{img}_{\mathrm{real}}) + \lambda_2 \cdot L_{\mathrm{GAN}}$$. The reported metrics for image translation are Mean Average Pixel Error (MAPE) and Structural Similarity Index Measure (SSIM) (Freud et al., 28 Aug 2025).

Downstream tactile state estimation is performed by ShearNet, implemented as a Gaussian-Density Neural Network. This network is trained on shPix2pix-generated tactile images with ground-truth pose and shear labels. Rather than outputting only point estimates, it predicts distributions over pose and shear variables, and optimization uses the negative log likelihood of the true variables under the predicted distributions. The stated motivation is robustness under tactile perceptual aliasing and ambiguity (Freud et al., 28 Aug 2025).

This division of labor is central to the method. shPix2pix inserts visually realistic shear into tactile images generated from rigid-body simulation, and the GDNN then extracts the latent control variables needed for servoing. The pipeline therefore treats tactile image realism and state inference as separate but coupled learning problems (Freud et al., 28 Aug 2025).

4. Experimental behavior and reported performance

The image-translation results show a substantial gap between vanilla pix2pix and shPix2pix. On edge contacts, pix2pix obtained MAPE $0.21$ and SSIM $0.20$, whereas shPix2pix obtained MAPE $0.07$ and SSIM $0.63$. On surface contacts, pix2pix obtained MAPE $0.23$ and SSIM $0.14$, whereas shPix2pix obtained MAPE $10\,\mathrm{Hz}$0 and SSIM $10\,\mathrm{Hz}$1. Visual examples are reported to show that only shPix2pix realistically synthesizes shear deformations (Freud et al., 28 Aug 2025).

For pose and shear estimation, baseline pix2pix-trained models were reported to estimate pose with some success but to fail at shear prediction, with large MAE. By contrast, shPix2pix-trained GDNNs were reported to match both pose and shear on real data with error comparable to models fully trained on real tactile images. The example errors given for $10\,\mathrm{Hz}$2-shear and $10\,\mathrm{Hz}$3-shear are approximately $10\,\mathrm{Hz}$4–$10\,\mathrm{Hz}$5 MAE for shPix2pix/GDNN, compared with baseline MAE greater than $10\,\mathrm{Hz}$6 (Freud et al., 28 Aug 2025).

In tactile tracking, the leader manipulated an object along circle, square, spiral, and loop trajectories, and the follower used tactile pose and shear estimates to maintain contact while tracking the moving surface. The reported typical position error between tactile sensor and target was $10\,\mathrm{Hz}$7–$10\,\mathrm{Hz}$8, with no loss of continuous contact. In collaborative co-lifting, the leader followed circular-wave and star trajectories while the follower adapted grip and pose in real time from tactile feedback. The evaluated objects were a rigid square prism, a rigid egg, a soft brain, and a rubber ducky, and the sensor was tested in both vertical and horizontal orientations. The reported typical tracking errors were again $10\,\mathrm{Hz}$9–$0.0001$0 for all objects and trajectories (Freud et al., 28 Aug 2025).

These results support two narrower conclusions. First, the method outperformed baseline pix2pix approaches in simulating tactile images and in pose/shear prediction. Second, models trained only on simulation data from rigid, flat objects with a vertical sensor were nevertheless reported to generalize to curved, soft, and horizontally oriented contacts in the real system (Freud et al., 28 Aug 2025).

5. Position within tactile-shear research

SimShear occupies one end of a methodological spectrum in tactile robotics. Its defining choice is to avoid explicit shear dynamics in the simulator and instead learn a shear-conditioned sim-to-real image translation. A contrasting approach is HydroShear, a non-holonomic hydroelastic tactile simulator that explicitly models stick-slip transitions, path-dependent force and shear buildup, and full $0.0001$1 object-sensor interactions using signed distance functions and hydroelastic contact formulations. HydroShear is reported to more faithfully reproduce real tactile shear than prior methods and to enable zero-shot sim-to-real transfer across peg insertion, bin packing, book shelving for insertion, and drawer pulling, with a $0.0001$2 average success rate (Dang et al., 28 Feb 2026).

This contrast is technically important. SimShear assumes that rigid-body contact simulation plus learned shear-conditioned synthesis is sufficient for the targeted servoing problems. HydroShear instead treats shear as a physical field to be modeled directly, including recursive path dependence, contact-state tracking, and Coulomb-limited tangential force updates. A plausible implication is that the two methods represent different trade-offs between simulator fidelity and learning-mediated compensation (Freud et al., 28 Aug 2025).

Another useful comparison comes from work on semi-supervised disentanglement of tactile contact geometry from sliding-induced shear. That work removes global shear from soft optical tactile sensor images in order to recover unsheared contact geometry for shape reconstruction and pose estimation. It reports that model-generated unsheared images closely match vertically tapped references, with $0.0001$3 SSIM using only $0.0001$4 supervision, and that pose errors improve markedly relative to raw sheared images (Gupta et al., 2022). SimShear pursues the opposite operational goal: instead of suppressing shear to isolate geometry, it synthesizes shear so that control can exploit it. The two directions are therefore complementary rather than contradictory.

6. Misconceptions, limitations, and broader significance

A common misconception is that SimShear is a shear-physics simulator in the usual mechanistic sense. The method explicitly does not model shear dynamics in simulation; its novelty lies in showing that shear can still be used for sim-to-real tactile control by conditioning image synthesis on a separate shear representation. It is therefore better understood as a sim-to-real tactile pipeline than as a first-principles shear simulator (Freud et al., 28 Aug 2025).

A second misconception is that the approach eliminates all sim-to-real difficulties. The reported limitations are more specific. The experiments were limited to 4-DoF robots and light objects. For heavier or more complex tasks, including higher-DoF or more dynamic settings, further extension and data augmentation were identified as necessary. Generalization was reported to be good, but it was not yet demonstrated on highly slippery, sharp, or point-contact objects (Freud et al., 28 Aug 2025).

Within tactile robotics, the broader significance of SimShear is methodological. It shows that lateral contact effects can be incorporated into sim-to-real learning without requiring a rigid-body simulator to natively produce shear deformation fields. That result matters because shear is often the missing variable in tactile control pipelines built around inexpensive vision-based sensors and accessible simulation infrastructure. The method therefore opens a route to shear-aware tactile tracking and collaborative manipulation using low-cost hardware and simulation data alone, while leaving open the question of when explicit, path-dependent shear mechanics—as in hydroelastic approaches—become necessary for further gains in fidelity or task scope (Freud et al., 28 Aug 2025)