PneuGelSight: Soft Optical Tactile Sensor

• PneuGelSight is a vision-based soft robotic sensor that integrates a pneumatic manipulator with an embedded wide-angle camera for simultaneous proprioception and tactile sensing.
• It employs a coupled inference strategy, using contour-based image analysis for global shape estimation and local photometric cues for detailed contact geometry reconstruction.
• The system leverages a simulation-driven pipeline to optimize optical fiber illumination and dynamic training, achieving real-time, precise deformation and tactile measurements.

Searching arXiv for PneuGelSight and closely related GelSight-family work. PneuGelSight is a soft robotic vision-based sensing system built around a pneumatic manipulator with an embedded camera that simultaneously supports high-resolution proprioception and tactile sensing. In the reported implementation, a single internal wide-angle camera observes the deformation of the finger’s inner reflective sensing surface, so the same image stream contains both global deformation cues for whole-finger shape estimation and local photometric cues for contact-geometry reconstruction (Zhang et al., 25 Aug 2025). The system was proposed to address a persistent difficulty in soft robotics: soft pneumatic manipulators are compliant and flexible, but their high-dimensional, pressure-dependent deformation makes both proprioception and tactile feedback difficult to obtain with conventional embedded sensors (Zhang et al., 25 Aug 2025). Within the broader GelSight lineage, PneuGelSight extends the vision-based tactile paradigm from rigid or semi-rigid tactile pads to a fully deformable pneumatic finger, while also incorporating a sim-to-real pipeline for optical design and proprioceptive learning (Zhang et al., 25 Aug 2025).

1. Historical and Technical Context

PneuGelSight belongs to the GelSight family of vision-based tactile sensors, which use an internal optical system and embedded camera to observe deformation of a soft reflective sensing surface. In this family, deformation images can encode geometry, texture, force-related effects, and slip-related phenomena, but most classical designs assume a relatively fixed optical geometry and a mechanically stable support (Li et al., 2018). That assumption is relaxed in several nonplanar and compliant descendants, including rounded GelSight-like fingertips for dexterous manipulation, soft-finger embodiments such as GelFlex, and compliant Fin Ray integrations, all of which expose the optical complications introduced when the sensor body itself deforms (Romero et al., 2020, She et al., 2019, Liu et al., 2022).

The specific challenge that motivates PneuGelSight is more severe than in rigid GelSight or even curved fingertip variants. A soft pneumatic finger has a high-dimensional continuous state; deformation is distributed over the entire body, nonlinear, and pressure-dependent. As a result, the internal image changes not only because of local contact, but also because the whole optical geometry, illumination distribution, and reflective surface deform together (Zhang et al., 25 Aug 2025). This makes conventional rigid-base GelSight assumptions inapplicable without modification.

Earlier GelSight-family work had already identified several relevant limitations. Standard GelSight can struggle under very light contact, with smooth or textureless objects, and when compliance-induced deformation creates image motion that is ambiguous with slip (Li et al., 2018). Soft or compliant embodiments further require explicit separation of global body deformation from local tactile effects; this was already a central issue in GelFlex and GelSight Fin Ray, where embedded vision had to support both proprioceptive and tactile interpretation inside a deformable finger (She et al., 2019, Liu et al., 2022). PneuGelSight can be understood as a pneumatic realization of this broader trajectory: a fully deformable vision-based soft manipulator in which one sensing stack must support both body-state estimation and local contact reconstruction (Zhang et al., 25 Aug 2025).

2. Mechanical Structure and Optical Hardware

PneuGelSight is implemented as a pneumatic soft finger inspired by prior asymmetric soft actuators. The finger combines a 3D-printed bellow-shaped back or backbone with an inner cast silicone slab that serves as the contact surface (Zhang et al., 25 Aug 2025). The robot was sized with inspiration from a human palm, with a reported length of 110 mm, a semi-circular cross-section, and a 55 mm diameter (Zhang et al., 25 Aug 2025).

The asymmetry of the structure is mechanically important. The slab is thicker and harder than the bellows, so under internal pressure the bellow structure elongates more easily while the slab resists extension and bends inward. This produces grasping-oriented inward bending (Zhang et al., 25 Aug 2025). The backbone is 3D printed using Silicone 40A, while the slab is a multilayer silicone structure with both optical and mechanical roles (Zhang et al., 25 Aug 2025).

The slab includes several layers. The sides use an opaque silicone diffuser made from Smooth-On EcoFlex 00-30. The transparent main body uses Silicone Inc. XP565, with a 7:1 ratio for a harder layer and a 14:1 ratio for a softer layer intended to improve tactile sensitivity (Zhang et al., 25 Aug 2025). The sensing surface is coated with semi-specular aluminum powder, followed by a protective final XP565, 14:1 silicone layer (Zhang et al., 25 Aug 2025). The entire inner side of the finger functions as the contact surface, which distinguishes PneuGelSight from localized tactile pads (Zhang et al., 25 Aug 2025).

The optical system is built around an embedded Arducam Wide Angle Camera with approximate 160° field of view, mounted in the back of the soft finger and looking toward the inner reflective slab through the hollow cavity (Zhang et al., 25 Aug 2025). Illumination is delivered through 0.75 mm optical fibers, chosen because they are lightweight, flexible, and easier to integrate into a soft body than rigid LED boards (Zhang et al., 25 Aug 2025). Light is provided from a remote source and guided into the finger through these fibers. The reported arrangement uses 6 green, 7 blue, 7 red, and 4 additional blue fibers, for 24 lights in total, arranged in clockwise order (Zhang et al., 25 Aug 2025).

The fabrication procedure is explicitly layered. A two-part mold is designed and 3D printed; diffusive silicone, hard transparent silicone, and soft transparent silicone are sequentially poured and cured; aluminum powder is applied as a semi-specular coating; a protective silicone layer is added; the soft backbone is printed and UV cured; the slab is sealed to the backbone using an alignment feature; optical fibers are inserted and glued; the embedded camera is sealed; and additional silicone glue is applied for airtightness (Zhang et al., 25 Aug 2025). The paper notes that the hard silicone height is slightly larger than the sealed depth, leaving some side area exposed to external light in order to improve visibility of the contact-surface contour in the internal camera view (Zhang et al., 25 Aug 2025).

3. Unified Sensing Principle

The central sensing principle is that the same internal image contains information at two spatial scales. First, the visible contour and geometry of the inner surface vary with whole-finger deformation and therefore encode proprioception. Second, local contact deforms the reflective elastomer surface, altering local surface normals and reflected color patterns in a GelSight-like manner, which encodes tactile information (Zhang et al., 25 Aug 2025).

For proprioception, the reported pipeline primarily uses contour features extracted from a binarized image. Rather than relying on explicit markers, the method uses intrinsic visual structure and learns a mapping from the camera-derived contour image to a 3D point cloud representation of finger shape (Zhang et al., 25 Aug 2025). This differs from earlier soft-finger vision systems such as GelFlex, which used engineered sidewall visual features and reflective tactile coatings within the same finger body (She et al., 2019).

For tactile sensing, the system uses local photometric variation produced by directional colored illumination reflecting from the deformed semi-specular surface. Because the global finger state changes the optical field, tactile sensing is formulated not as direct image-to-shape reconstruction, but as reconstruction relative to a deformation-conditioned background (Zhang et al., 25 Aug 2025). This is conceptually related to other compliant GelSight-family systems that must distinguish local contact effects from broader sensor-body deformation, although PneuGelSight addresses this within a fully pneumatic body rather than an exoskeleton-covered or Fin Ray structure (Liu et al., 2022, She et al., 2019).

The contact region is localized through a proposal-and-scoring pipeline. The image is divided into 36 proposals. For each proposal, the method estimates a no-contact background under the current deformation state, computes a color-difference score,

$$\Delta C = \sum |I_{\text{image}} - I_{\text{background}}|,$$

and selects the proposal with maximal discrepancy as the contact area (Zhang et al., 25 Aug 2025). After region selection, a reconstruction network predicts per-pixel surface normals $(N_x, N_y, N_z)$, which are then integrated using Poisson integration to recover local contact geometry (Zhang et al., 25 Aug 2025).

This suggests that tactile interpretation in PneuGelSight is inherently conditional on global body state rather than separable from it. A plausible implication is that, in fully deformable optical tactile systems, background estimation is not just preprocessing but part of the sensing model itself.

4. Learning Architecture for Proprioception and Tactile Reconstruction

The proprioception pipeline is built in two stages. First, a PointNet-style autoencoder is trained on point clouds sampled from robot meshes, learning a latent prior over plausible finger shapes. The finger shape is represented as a point cloud

$p \in \mathbb{R}^{N \times 3},$

with $N = 4096$ in the main experiments and up to $N = 8192$ in additional tests (Zhang et al., 25 Aug 2025). The reconstruction loss is the Chamfer distance,

$L_{\text{recon}} = \text{CD}(p, p_{\text{recon}})$

$$= \frac{1}{|p|} \sum_{x \in p} \min_{y \in p_{\text{recon}}} \|x-y\|^2 + \frac{1}{|p_{\text{recon}}|} \sum_{y \in p_{\text{recon}}} \min_{x \in p} \|y-x\|^2.$$

This stage learns a compact shape manifold for the soft finger (Zhang et al., 25 Aug 2025).

Second, the conditional proprioception network, termed ProprioNet in the technical summary, combines the learned shape prior with an image encoder operating on the binary contour mask. The image feature is spatially repeated and fused with reference point and global features by element-wise summation, and the decoder outputs the deformed point cloud (Zhang et al., 25 Aug 2025). Training uses a combined objective,

$$L = L_{\text{recon}} + \frac{1}{N}\|g - g_{\text{pre-trained}}\|^2,$$

where $g$ is the global feature from the multimodal network and $g_{\text{pre-trained}}$ is the feature extracted by the pre-trained autoencoder (Zhang et al., 25 Aug 2025). This architecture is intended to predict deformation relative to a geometric prior rather than synthesizing soft-robot shape directly from image alone.

The tactile branch uses a conditional MLP with encoder-decoder structure. The reported input for each sampled pixel consists of a $(N_x, N_y, N_z)$0 neighborhood of RGB values, giving 27 values, together with pixel coordinates $(N_x, N_y, N_z)$1, giving a total feature dimension of 29. The per-iteration tensor is

$(N_x, N_y, N_z)$2

and the network predicts surface normals for points both inside and outside the contact region, with non-contact points labeled $(N_x, N_y, N_z)$3 (Zhang et al., 25 Aug 2025). The global feature from the pre-trained proprioception branch is incorporated into tactile reconstruction, so the tactile estimator is deformation-aware (Zhang et al., 25 Aug 2025).

This interconnected design is one of the defining characteristics of PneuGelSight. Unlike classical GelSight, where local tactile reconstruction can often be treated independently of global sensor shape, PneuGelSight requires a coupled inference strategy because the entire sensing geometry is state-dependent (Zhang et al., 25 Aug 2025).

5. Optical and Dynamic Simulation Pipeline

A major part of PneuGelSight is its simulation framework, which serves two distinct purposes: optical design optimization and proprioceptive data generation (Zhang et al., 25 Aug 2025). This dual use differentiates it from earlier GelSight simulation efforts that primarily targeted optical image synthesis for rigid or non-pneumatic sensors (Gomes et al., 2023, Nguyen et al., 2024).

For optical simulation, the fibers and diffuser layer are jointly approximated as point lights; the transparent silicone gel layer is modeled with a rough dielectric model; the reflective coating is modeled with a surface diffusive model; and the camera is modeled as a perspective camera with given field of view and resolution (Zhang et al., 25 Aug 2025). The soft body walls are treated as black or absorbing, since they are painted black to suppress stray light (Zhang et al., 25 Aug 2025).

The tactile background model uses per-light intensity terms: $(N_x, N_y, N_z)$4 with total intensity

$(N_x, N_y, N_z)$5

where $(N_x, N_y, N_z)$6 is the distance from pixel $(N_x, N_y, N_z)$7 to light $(N_x, N_y, N_z)$8, $(N_x, N_y, N_z)$9 is the angle between the surface normal and light direction, $p \in \mathbb{R}^{N \times 3},$0 is the in-plane angular displacement, and $p \in \mathbb{R}^{N \times 3},$1 is a directional falloff exponent (Zhang et al., 25 Aug 2025). In linearized color space, the model is written

$p \in \mathbb{R}^{N \times 3},$2

where $p \in \mathbb{R}^{N \times 3},$3 is the linearized RGB vector, $p \in \mathbb{R}^{N \times 3},$4 is the coefficient matrix, and $p \in \mathbb{R}^{N \times 3},$5 is the light-intensity matrix. The inverse estimate is approximated as $p \in \mathbb{R}^{N \times 3},$6, and the background in a new region is projected as $p \in \mathbb{R}^{N \times 3},$7 (Zhang et al., 25 Aug 2025).

Color-space conversion is also modeled explicitly: $p \in \mathbb{R}^{N \times 3},$8 and

$p \in \mathbb{R}^{N \times 3},$9

(Zhang et al., 25 Aug 2025).

The optical design variable optimized in simulation is the color arrangement of the 24 fibers. The metric used is contact-region color variance under sphere indentation: $N = 4096$0 This metric is averaged across bending scenarios and indentation positions, and the arrangement maximizing it is selected (Zhang et al., 25 Aug 2025). Because naive search would require $N = 4096$1 combinations, the pipeline exploits superposition by rendering one LED at a time and combining outputs, reducing the rendering burden to $N = 4096$2 images plus combinatorial recombination (Zhang et al., 25 Aug 2025).

For dynamic simulation, the authors generate proprioception training data from randomized FEM scenes involving varying pneumatic pressures and external contacts. The reported dataset contains 26 scenes and 3,000 images, including interactions with a wall, a cube, a cylinder, and a moving plane approaching from multiple directions (Zhang et al., 25 Aug 2025). The stated result is zero-shot transfer from simulation to the real robot for proprioception, achieved through physically informed simulation, augmentation, and contour-based inference rather than explicit domain adaptation (Zhang et al., 25 Aug 2025).

6. Experimental Performance, Limitations, and Significance

The proprioception evaluation uses five representative deformation scenarios: neutral pose, natural bending, forward bending with external load, backward bending with external load, and lateral bending with external load. Ground truth is obtained using an RGBD camera, and the reported Chamfer distances are 2.21 mm, 6.76 mm, 8.76 mm, 2.12 mm, and 7.09 mm, with an overall mean of 5.35 mm (Zhang et al., 25 Aug 2025). The paper explicitly notes that this improves on the authors’ previous 8.85 mm result (Zhang et al., 25 Aug 2025). On an NVIDIA RTX 4070 GPU, inference time remains below 0.05 s even at $N = 4096$3, indicating real-time feasibility (Zhang et al., 25 Aug 2025).

The tactile evaluation uses a 3D printed six-faced pyramid indenter with 8 mm diameter and 2 mm height, tested at 6 indentation locations and 11 pressure levels from 1 atm to 1.35 atm. Quantitative metrics are Chamfer distance between indenter geometry and reconstructed mesh, and maximum indentation depth error (Zhang et al., 25 Aug 2025). The reported best average Chamfer distance is 0.18 mm, and the maximum depth error remains below 0.2 mm in the central region across all bending angles (Zhang et al., 25 Aug 2025). Performance is strongest in the central sensing area, weaker near the tip and borders, and generally better at lower internal pressures (Zhang et al., 25 Aug 2025).

Tactile sensitivity is measured with a UR5e robot arm, a NRS-6050-D80 force-torque sensor, and two spherical indenters of 5 mm and 35 mm diameter. At 1.35 atm, the minimum detectable force ranges from approximately 0.2 N to 0.95 N, depending on indenter size, and sensitivity decreases as internal pressure increases (Zhang et al., 25 Aug 2025). This directly reflects the stiffness-sensitivity tradeoff induced by pneumatic pressurization.

A real-world demonstration reconstructs the shape and texture of an avocado from multiple touches using a UR5e-mounted PneuGelSight finger. At each contact, proprioception estimates the finger deformation and the tactile branch reconstructs local texture; the partial reconstructions are then stitched into an aggregate estimate (Zhang et al., 25 Aug 2025). This suggests that the combined sensing stack can support active perception over distributed contact sequences, rather than only instantaneous local touch.

Several limitations are explicit. Sensing quality is spatially nonuniform, with reduced performance near the borders and tip; tactile sensitivity decreases at higher pressure; large bending alters illumination and lowers sensing quality; lateral bending is mechanically constrained and potentially harmful; and the simulation-real match is good but not exact, with edge distortions near light sources (Zhang et al., 25 Aug 2025). The paper does not provide an extended durability study, so wear of the reflective coating, contamination sensitivity, and long-term pressure cycling remain open issues. Related work on polyurethane-based optical tactile skins suggests that resilience of the tactile interface can be a major deployment bottleneck in vision-based tactile sensors, particularly under repeated loading, shear, and abrasion (Davis et al., 11 Nov 2025). This suggests that future PneuGelSight variants may need materials co-design in addition to optical and learning improvements.

In the broader research landscape, PneuGelSight occupies a distinctive position. Compared with rigid or rounded GelSight fingertips, it extends optical tactile sensing into a fully deformable pneumatic body (Romero et al., 2020). Compared with soft-finger systems such as GelFlex and GelSight Fin Ray, it couples tactile sensing with pneumatic actuation and simulation-driven zero-shot proprioception (She et al., 2019, Liu et al., 2022). Compared with simulation frameworks for GelSight Mini and other complex morphologies, it adds dynamic deformation simulation and state-dependent tactile interpretation specific to a soft pneumatic manipulator (Nguyen et al., 2024, Gomes et al., 2023). The result is a sensorized soft finger in which embodied deformation is not a nuisance variable but a first-class signal.

Overall, PneuGelSight establishes that embedded vision can serve as a dense sensing modality for soft pneumatic robots, enabling both whole-body proprioception and local tactile geometry reconstruction from a single internal camera stream (Zhang et al., 25 Aug 2025). Its technical significance lies not only in the hardware, but in the coupled inference strategy and the use of simulation for both optical design and learning.