GelSight Mini Visuotactile Sensor

• The GelSight Mini sensor is a compact visuotactile device that captures both normal and shear force distributions via photometric stereo and embedded marker tracking.
• It features a thin, silicone gelpad with patterned markers and colored LED illumination, translating gel deformations into high-resolution RGB images.
• Learning-based force regression combined with FEM simulation enables real-time tactile inference and robust performance in contact-rich robotic manipulation.

The GelSight Mini is a commercially available, compact, high-resolution optical tactile sensor that enables three-dimensional force distribution inference on soft elastomeric pads through imaging of gel deformations. It achieves visuotactile sensing by leveraging photometric stereo effects from colored LED illumination combined with embedded marker tracking. This technology forms an integral component in contact-rich robotic manipulation research and has recently seen robust simulation and learning-based force inference pipelines. Below, the fundamental principles, modeling approaches, and research methodologies associated with the GelSight Mini are systematically detailed.

1. Sensor Design and Sensing Modality

The GelSight Mini sensor comprises a transparent acrylic shell supporting a ≈3 mm-thick silicone elastomeric gelpad. The inner surface of the gelpad is painted with a diffuse white layer and patterned with dark markers. An oblique ring of red, green, and blue LEDs surrounds the gel, with each color corresponding to a distinct illumination angle. A miniature color camera (240×320 pixels) is positioned to image the gel from behind.

During contact, object-induced deformations of the painted gel cause color-shading patterns—each related to the local gel surface slope via the colored illumination angles. These shading cues, together with the displacement of embedded markers (responsive to shearing motions), enable the observation of both out-of-plane normal and in-plane shear deformations. The sensor thus provides raw RGB images at 25 Hz encoding a nonlinear mapping from gel surface normal and marker slip to color intensities (Helmut et al., 2024).

2. Physical Simulation of GelSight Mini

High-fidelity simulation of the GelSight Mini visuotactile modality requires modeling both the elastomeric deformation and the camera's optical response. In the TacEx framework, the following physical modeling components are used (Nguyen et al., 2024):

• Gel Deformation Modeling: The GIPC solver is employed—a GPU-accelerated, inversion- and intersection-free finite element method (FEM) handling hyperelastic materials through an Incremental Potential Contact (IPC) formulation. For a mesh of n nodes at world-space positions $x \in \mathbb{R}^{3n}$, the total potential energy is

$$E(x) = \sum_{e} W_e(F_e(x)) + \sum_{contact} B(\phi(x))$$

where $F_e(x)$ is the deformation gradient, $W_e$ is the corotational or neo-Hookean energy, and $B(\phi)$ is a barrier potential to prevent interpenetration. Time integration proceeds via implicit minimization of the discrete update:

$$x^{k+1} = \underset{x}{\arg\min} \; \frac{1}{2} (x - x^k)^T M (x - x^k) + h^2 E(x)$$

with mass matrix $M$ and timestep $h$.

• Camera and Marker Rendering: Simulation includes a depth camera to generate a 480×640 height map $H(u,v)$ from the mesh, smoothed via a Gaussian pyramid. The Taxim renderer computes per-pixel normals and uses a calibrated polynomial lookup table to simulate photometric response. FOTS models marker displacement fields by generating flow fields $F(u,v)$ using exponential distributions parameterized by estimated load types, mapped to the camera view.
• Parameterization: Gelpad mesh resolution is user-configurable. Tetrahedral resolutions from 24 tets (3,717 vertices) to 221k tets (66,563 vertices) yield GIPC simulation steps from ≈25 ms up to 222 ms on an RTX 3080 Ti, with 16 Gauss–Newton iterations per frame. Gel stiffness and interface geometry are set via calibration but not explicitly detailed.
• Stability: The simulation stably handles rolling contact, object lifting via frictional grasping, and beam twisting, with intersection-free deformation and plausible physical effects.

3. Force Distribution Estimation: FEA-Guided Learning

Direct inference of distributed normal and shear forces from GelSight Mini images is addressed by learning-based methods that amortize the inverse mechanics problem. The FEATS framework (Helmut et al., 2024) proceeds as follows:

• Ground-Truth Generation with FEA: Twelve 3D-printed indenters probe the gelpad using a CNC mill under controlled conditions (±0.25 µm position, up to 40 N normal, ±5 N shear). The same procedure is simulated in CalculiX, modeling the gel as a compressible Neo-Hookean solid with density

$$U(\mathbf{F}) = C_{10}(\bar{I}_1 - 3) + \frac{1}{D_1} (J-1)^2$$

with $C_{10}$ (shear modulus) ≈ 0.0792 and standard $D_1$. The stress field is projected into the camera image plane, and force elements are aggregated into $24 \times 32$ grid bins for $f_x$, $f_y$, $f_z$.

• Learning Pipeline: A U-Net maps 240×320×3 raw images to $24\times32\times3$ force maps. The encoder and decoder consist of four down/up-sampling stages, with skip connections, convolutional layers, and ReLU activations. Loss is the per-pixel mean-squared error over all force components,

$$\mathcal{L}(\mathbf{f},\hat{\mathbf{f}}) = \frac{1}{3WH} \sum_i^W \sum_j^H \| \mathbf{f}^{(i,j)} - \hat{\mathbf{f}}^{(i,j)} \|_2^2$$

Training employs 5,173 samples with random color and noise augmentation, using Adam optimizer.

• Performance: On test data, per-pixel MAEs are $0.0006 \pm 0.0006$ N ($f_x$), $0.0005 \pm 0.0003$ N ($f_y$), and $0.0015 \pm 0.0010$ N ($f_z$). Summed over the tactile map, total-force errors are $0.224 \pm 0.401$ N ($F_x$), $0.093 \pm 0.136$ N ($F_y$), $0.372 \pm 0.473$ N ($F_z$). Inference operates at $4.17 \pm 1.34$ ms per frame (real-time at 25 Hz), contrasting with FEA runtimes of 10–120 minutes. The model generalizes to other GelSight Mini units with minor spatial shifts attributed to mounting variability. Ablation shows the 24×32×3 output U-Net balances spatial resolution and accuracy.

4. Tactile Observations in Simulation and RL Environments

TacEx deploys the full visuotactile pipeline in Isaac Lab reinforcement learning environments (Nguyen et al., 2024):

• Object Pushing: A GelSight Mini finger pushes a cube toward targets; observations include 2D flow field $F(u,v)$ ($480\times640\times2$) and proprioceptive robot state.
• Object Lifting: Dual pads grasp primitives with two marker flow fields and pose/height features.
• Pole Balancing: The end-effector must maintain a pole upright, with marker flow and angle inputs.

Tactile readings are formatted as tensors $T \in \mathbb{R}^{H \times W \times C}$ ($C=2$ for flow, $C=3$ for RGB), normalized to $[-1,1]$ and concatenated with proprioceptive data. These are provided to a proximal policy optimization (PPO) policy. The simulation streams high-dimensional, gel-deformation or marker-based signals amenable to deep RL, enabling study and development of contact-rich manipulation behaviors.

5. Evaluated Strengths, Limitations, and Open Challenges

The GelSight Mini's appeal lies in full spatial resolution of both shear and normal force distributions from a single, commercial, and unmodified sensor. Learning-based force regression achieves real-time speeds and generalizes across objects and sensor units, given calibration. In simulation, the GIPC-based models supply robust, frictional, and intersection-free contact dynamics, and the unified pipeline supports parallelized training environments for tactile RL.

Noted limitations arise from domain shift between sensor units, which motivates domain adaptation research. The force estimator's accuracy for normal components may benefit from more sophisticated constitutive models if further improvement beyond ≈0.4 N MAE is required. Training is presently based on a relatively narrow set of indenter geometries; generalization to more diverse contacts and dynamic events (slip, rolling) is an open area, as is modeling of long-term gel fatigue or viscoelastic effects. No direct, quantitative image-based sim-to-real validation has yet been performed in TacEx. Real-time operation, however, is demonstrated for all core components (Helmut et al., 2024, Nguyen et al., 2024).

6. Summary of Computational and Experimental Benchmarks

Aspect	Metric / Result	Source
Camera Resolution	240×320 (real), 480×640 (sim height map)	(Helmut et al., 2024, Nguyen et al., 2024)
Force Distribution Grid	24×32×3	(Helmut et al., 2024)
Deformation Solver	GIPC/FEM, 16 Gauss–Newton iterations/frame	(Nguyen et al., 2024)
Simulation Step (GIPC)	25–222 ms (mesh-dependent)	(Nguyen et al., 2024)
Learning Inference	4.17 ± 1.34 ms/frame	(Helmut et al., 2024)
Test Error (total force)	≤0.4 N normal, ≤0.2 N shear	(Helmut et al., 2024)

With its hardware simplicity, learning-based inverse mechanics, and simulation-embeddable tactile modality, the GelSight Mini provides a research platform for closed-loop, force-aware, and contact-rich robotic tasks, bridging real and simulated environments with physically grounded data-driven tactile inference.