More with LESS -- Local Scene Representations for Tactile Imaging

Abstract: Tactile imaging seeks to reconstruct the internal structure of soft objects through touch sensing, with applications in medical diagnosis and robotic manipulation. Recent self-supervised learning approaches have shown promising results, but rely on global, unstructured representations and robot-controlled sensing, limiting generalization and practical use. We propose Local Encoder for Spatial Sensing (LESS), an object-centric tactile representation that exploits the local nature of touch. The tactile scene is modeled as a grid of recurrent encoders with local receptive fields, whose states are fused to reconstruct 2D or 3D images of internal structure. This compositional design enables strong generalization: models trained on single-inclusion phantoms accurately image objects with multiple inclusions and varying sizes. The local structure further supports spatial uncertainty estimation. In addition, we enable hand-held tactile imaging via external pose tracking and human-like palpation data, and extend tactile imaging to full 3D reconstruction.

• The paper introduces the LESS architecture, which leverages localized neural representations to reconstruct tactile images and enable zero-shot compositional generalization.
• It employs a grid of recurrent particles processed via GRU-based encoders and patch-based decoders to reconstruct 2D/3D subsurface structures.
• Experimental results demonstrate high F1 scores, effective uncertainty estimation, and robustness to pose errors in both robotic and manual operations.

Local Scene Representations for Tactile Imaging: The LESS Architecture

Introduction and Motivation

Tactile imaging aims to reconstruct the internal structure of soft objects through touch—holding substantial relevance for applications in medical diagnosis and robotic manipulation. Traditional approaches rely on global, unstructured scene representations and robot-controlled, highly structured exploration—leading to compromised generalization and real-world practicality, especially for variable human-in-the-loop operation. To address these limitations, "More with LESS -- Local Scene Representations for Tactile Imaging" (2606.14344) proposes the Localized Scene Representation (LESS) architecture: a compositional, spatially localized neural representation inspired by the physically local nature of touch. LESS models tactile scenes as grids of recurrent "particles," each encoding touch feedback within a local receptive field and integrating this information to reconstruct 2D or 3D images of subsurface structure. This design affords zero-shot compositional generalization, spatially localized uncertainty estimation, operator-guided sampling, and robust operation in both robotic and unstructured hand-held use cases.

The LESS Model: Localized Representation Learning

Unlike previous methods employing a single global latent vector, LESS replaces this design with a structured grid of localized representations—each "particle" covering a spatial receptive field in the $x$-$y$ plane (Figure 1). Each particle receives, as input, time sequences of tactile readings and pose information only within its receptive field, centered with respect to the particle's location. Measurement pairs are encoded via sinusoidal positional encoding and MLPs, then processed by GRU-based encoders to produce particle-specific latent states. Weight sharing across all particles enables locality-aware learning with strong sample efficiency.

Figure 1: LESS representation learning: each particle encodes local tactile sequences within its spatial receptive field using a GRU-based encoder-decoder, with all particles sharing weights.

To decode forces at arbitrary locations, each particle’s representation—joint with centered relative position—is fed into an MLP-based decoder. The entire tactile scene is thus modeled as an ensemble of local predictors, each governing a spatial region and collectively tiling the region of interest.

Image Reconstruction: Patch-Based Generative Modeling

The scene image reconstruction module receives the set of local particle latent states and reconstructs patches of the final tactile image independently for each particle (Figure 2). Each latent representation is decoded by a transposed convolutional neural network to produce logits for a local patch. To assemble the global tactile image, overlapping logits from all patches are summed and softmaxed, yielding pixel- or voxel-wise class probabilities and supporting uncertainty quantification.

Figure 2: Local latent representations are mapped to image patches, and all patches are stitched together by averaging logits and applying a softmax, resulting in the aggregated tactile image.

This patch-based generative mechanism extends readily to 3D, supporting volumetric reconstruction for richer visualization and improved metric performance even on 2D slices, as the volumetric prior enforces geometric consistency.

Experimental Results

Zero-Shot Compositional Generalization

LESS demonstrates strong zero-shot compositional generalization. When trained exclusively on single-inclusion phantoms, LESS accurately reconstructs tactile images of objects with multiple, spatially varying inclusions, multi-connected structures, and size variations (Figure 3). The localized nature of the model—combined with relative pose centering—renders the architecture agnostic to the global configuration, enabling recombination of learned local features in novel test-time contexts.

Figure 3: LESS reconstructs intermediate tactile images for test sequences; the model generalizes to multiple or complex inclusions despite only being trained on single inclusions.

For out-of-distribution samples—such as multi-inclusion and enlarged phantoms—LESS shows high F1 scores (71.1% and 71.3%, respectively), with significantly lower error in area and center-of-mass estimation compared to global representation baselines. The global approach fails to reconstruct multiple inclusions or adapt to objects of novel extents, while LESS exhibits minimal degradation relative to in-distribution performance.

Figure 4: LESS achieves accurate reconstructions for both in-distribution and out-of-distribution structures compared to a global representation baseline that fails on compositional and scale variation.

Uncertainty Estimation and Receptive Field Ablation

LESS natively provides spatially localized uncertainty estimates by computing pixel-wise entropy of softmax class probabilities. Unlike global latent models, which often prematurely saturate confidence, LESS yields uncertainty maps that closely track sensor coverage and can guide operator re-sampling by highlighting undersampled regions.

Local receptive field size controls the trade-off between compositionality and context aggregation. As field width increases, single-object accuracy improves (due to richer context), but generalization to multi-inclusion objects degrades, as spatial independence is relaxed. The optimal field size is empirically found to balance these competing priorities (Figure 5).

Figure 5: Model performance as a function of receptive field size, showing improved compositionality for smaller fields and improved in-distribution accuracy for larger ones.

Robustness to Pose Estimation Noise and Hand-Held Operation

Robust real-world operation necessitates resilience to pose inaccuracies. LESS exhibits stable imaging up to $\sim$1 mm positional and 0.1 radian orientation error (Figure 6), which matches state-of-the-art fiducial-based tracking system capabilities.

Figure 6: LESS performance (F1 score) as a function of injected pose noise, validating robustness at practical noise magnitudes.

Training on teleoperated motion primitives in addition to robot-generated pokes mitigates distribution shift, enabling high-fidelity, real-time image reconstruction in manual, hand-held operation. Visualization in Figure 7 shows the operator’s display during palpation: prediction and spatial uncertainty maps are updated in real-time and compared against MRI-derived ground truth.

Figure 7: Operator interface showing the tactile prediction, uncertainty map, and ground-truth MRI during real-time hand-held imaging.

Technical Contributions Beyond Model Architecture

The study introduces several engineering advances to support comprehensive evaluation and practical deployment:

• Dataset: Large-scale tactile interaction dataset ($\sim$800 hours), spanning diverse phantoms with variable inclusions and MRI-captured ground truth.
• Phantom Fabrication: A robust, repeatable fabrication pipeline supporting multi-inclusion and variable-scale inserts (see Figures 10, 11, and 16).
• Automated Data Collection: Motorized insert handling (scissor lift in Figure 8) enabling 24/7 unsupervised tactile data collection.
• Pose Calibration: High-accuracy, fiducial-based pose estimation and robot-to-camera calibration (Figure 9).

Implications and Future Outlook

The LESS framework sets a foundation for robust, generalizable tactile perception in both autonomous and operator-driven scenarios. Practical implications are significant for point-of-care, operator-independent diagnostic devices and advanced robotic manipulators in unstructured environments. The compositional, localized scene representation approach offers an alternative to prevalent global/holistic latent approaches, with superior generalization grounded in physical localism.

Future directions include scaling LESS to more complex anatomical real-world datasets (moving from synthetic phantoms to human tissue), integrating with multimodal sensory streams (e.g., combining vision and tactile images), and exploiting its compositionality in downstream tasks such as action selection, affordance reasoning, and dexterous object manipulation. The architectural motifs here—namely spatially structured, weight-sharing latent grids with restricted receptive fields—are likely to find broad relevance in spatio-temporal sensor fusion domains.

Conclusion

LESS introduces a principled, scalable local representation framework for tactile imaging, supporting zero-shot compositional generalization, spatially precise uncertainty estimation, and robust operation in both structured and unstructured settings. By leveraging the physical locality of touch, the model outperforms global latent representations in reconstructing complex internal structures and enables real-time, operator-guided tactile exploration for clinical and robotic applications (2606.14344).