TacThru: Transparent Tactile-Visual Sensing

• TacThru is a see-through-skin sensor system that combines a transparent elastomer with keyline marker tracking to capture simultaneous tactile and visual data.
• It employs persistent illumination and Kalman filter-based real-time tracking of 64 markers, enabling fine-grained detection of deformations during manipulation.
• TacThru-UMI integrates these multimodal signals using a Transformer-based Diffusion Policy, achieving state-of-the-art performance in complex robotic tasks.

TacThru is a see-through-skin (STS) sensor system for robotic manipulation that enables high-fidelity, simultaneous tactile and visual perception, robust tactile tracking, and effective integration into modern learning-based manipulation pipelines. The TacThru architecture pairs a transparent elastomer sensor with persistent illumination and keyline marker tracking, and incorporates these signals through a Transformer-based Diffusion Policy for manipulation. The TacThru-UMI variant demonstrates state-of-the-art performance in complex real-world tasks, establishing significant benchmarks in multimodal robot learning and sensor design (Li et al., 10 Dec 2025).

1. Sensor Architecture: Hardware, Optics, and Materials

TacThru’s sensing unit consists of a fully transparent elastomer skin, fabricated using methods compatible with GelSight-style sensors. The elastomer is cast in a 40 mm × 40 mm window directly over a camera lens. While exact parameters such as thickness or refractive index are not specified, typical GelSight elastomers are PDMS with $n \approx 1.41$ and $d \approx 2$–$3$ mm.

Light propagation within the elastomer adheres to Snell’s law:

$$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$

Under non-contact (transmissive) conditions, incident rays pass through. When the elastomer is deformed by contact, altered local surface normals result in reflection or scattering, revealing precise contact locations and geometry.

Persistent illumination is provided by a ring of white LEDs (400–700 nm) positioned behind the sensor, delivering uniform illumination without the need for alternating between tactile and visual imaging modes. This hardware design enables simultaneous acquisition of visual information from the world beyond the sensor and tactile information through changes in elastomer reflection and marker displacement.

2. Keyline Markers: Geometry, Tracking, and Tactile Encoding

TacThru introduces a marker scheme with 64 concentric “keyline” markers, applied using laser-cut masks. Each marker consists of a black paint inner circle ($r_\text{in} = 0.6$ mm) and a white paint outer circle ($r_\text{out} = 1.0$ mm), arranged on a 3.5-mm grid spanning the sensing window. The high-contrast black/white “keyline” edge remains visible even against complex scene backgrounds, and marker spacing is chosen to prevent cross-matching under maximal deformation.

Tactile deformation is encoded by tracking each marker’s 2D displacement $\Delta x_t$, which registers local elastomer stretch and shear. Each marker’s pixel-level movement under deformation is mapped to local strain $\epsilon \approx \Delta x/d$; with the elastomer modulus $E$, the resulting stress is approximated as $\sigma = E \epsilon$. This mapping operates under the assumption of small deformation.

Keyline detection on each frame involves grayscale conversion with global threshold $d \approx 2$0 for background suppression, blob detection to localize markers, nearest-neighbor assignment with respect to prior marker positions, and temporal smoothing using independent Kalman filters per marker:

• Prediction: $d \approx 2$1, $d \approx 2$2
• Update: $d \approx 2$3, $d \approx 2$4 with $d \approx 2$5 for random walk.

The full tracking pipeline achieves robust, real-time performance at 6.08 ms per frame ($d \approx 2$6 Hz), stably tracking all 64 markers with negligible false positives.

3. TacThru-UMI: Multimodal Learning Framework

TacThru-UMI is an imitation learning system leveraging simultaneous tactile-visual data streams within a Transformer-based Diffusion Policy architecture. At each timestep $d \approx 2$7, the following inputs are provided:

• Wrist-camera frames $d \approx 2$8
• Sensor frames $d \approx 2$9
• Marker deviations $3$0
• Proprioceptive state $3$1 (gripper width, end-effector pose)

Tokens are embedded per modality:

• Visual: $3$2, $3$3
• Markers: $3$4
• Proprio: $3$5

All modality tokens are concatenated (early fusion) and input to the Transformer, which predicts a sequence of actions $3$6 via a standard denoising diffusion process. The action chunk is then executed via a low-level Cartesian controller.

Training employs a combination of behavior cloning loss ($3$7) and diffusion (denoising) loss ($3$8), with total objective $3$9. AdamW optimizer is used with a one-cycle schedule and 150 training epochs. Reported observation horizons are $$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$0, $$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$1, $$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$2; action chunk length is $$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$3.

4. Experimental Tasks and Quantitative Results

TacThru-UMI was evaluated on five real-world manipulation tasks of increasing complexity:

1. PickBottle: Grasp a plastic bottle and place it in a bowl.
2. PullTissue: Extract a single tissue from a pack of thin, soft paper.
3. SortBolt: Identify and sort one of three M12×25 bolts into designated bowls.
4. HangScissors: Grasp scissors and hang them on a hook.
5. InsertCap: Insert a small cap onto a mount with $$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$41 mm clearance.

Average success rates for each policy variant:

Policy Variant	Success Rate (%)
TacThru-UMI (TT-M)	85.5
Alternating Tactile-Visual (TT)	66.3
Vision-only (Wrist)	55.4

Critical scenario analysis and ablations revealed that TacThru-UMI substantially outperforms both the GelSight+markers (GS-M) and vision-only policies, particularly in:

• PullTissue: TT-M ($$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$580%), GelSight ($$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$60%), wrist camera ($$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$75%).
• SortBolt: TT-M (85%), GS-M (45%), wrist only (17%); marker-based tactile encodings resolve ambiguous geometries, corroborated by t-SNE of DINO embeddings.
• HangScissors: TT-M and GS-M ($$n_{\text{elastomer}} \sin \theta_i = n_{\text{air}} \sin \theta_t$$880%), wrist only (35%).
• InsertCap: TT-M (90%), GS-M (70%), wrist only (40%); TT-M enables adaptive vision-servoing plus tactile fallback.

Sensor robustness was tested under challenging lighting and background conditions. Solid markers led to frequent detection failures with dark backgrounds. The keyline marker + Kalman filter configuration tracked all markers reliably at 6.08 ms/frame (Li et al., 10 Dec 2025).

5. Analysis of Limitations and Prospective Developments

TacThru’s transparent elastomer trades off direct depth-from-contact capability and lacks embedded depth estimation. Demonstrations are limited to two-finger gripper grasps and a relatively small object set. Transformer-based policy learning requires careful temporal synchronization across modalities.

Suggested future directions include:

• Expansion to large-scale imitation datasets and simulated tactile data for pre-training specialized encoders.
• Extension to multi-fingered hands and more dexterous in-hand manipulation tasks.
• Integration of slip detection, dynamic force control, and hybrid reinforcement learning + imitation learning policies.
• Deployment in applications such as assembly, fragile object handling, and surgical robotics where simultaneous pre-contact vision and tactile feedback are crucial.

6. Contextual Significance in Multimodal Robotic Manipulation

TacThru demonstrates the utility of simultaneous, co-located tactile and visual perception in high-precision, contact-rich manipulation tasks. The system’s multimodal integration—transparent elastomer, persistent illumination, and robust marker tracking—enables state-of-the-art success rates, substantially surpassing unimodal or alternately switched baselines. Key methodological advances include marker design, real-time tactile tracking, and diffusion-based policy learning with early-fused multimodal tokens. This suggests new directions for scalable, adaptable, and dexterous manipulation systems in both research and industry settings (Li et al., 10 Dec 2025).