GelSLAM: Tactile 3D SLAM

• GelSLAM is a tactile-only 3D SLAM system that uses vision-based tactile sensors to estimate object pose and reconstruct shape with submillimeter accuracy.
• It harnesses differential geometry by converting tactile readings into surface normals and curvature, enabling robust registration, loop closure, and drift correction.
• The system achieves high precision in in-hand manipulation and scanning applications, significantly outperforming traditional methods in low-texture and occluded environments.

GelSLAM is a tactile-only, real-time 3D SLAM system designed to estimate long-horizon $6\mathrm{DoF}$ object pose and reconstruct object shape with high fidelity using vision-based tactile sensors from the GelSight family. Its central methodological move is to treat tactile observations not as shallow point clouds alone, but as measurements of differential geometry—specifically surface normals and curvature—which are then used for registration, loop closure, and globally consistent mapping. In the reported formulation, GelSLAM tracks object motion in real time with low error and minimal drift, reconstructs shapes with submillimeter accuracy, and operates robustly on low-texture objects such as wooden tools, thereby extending tactile sensing from strictly local contact estimation to global, long-horizon spatial perception (Huang et al., 21 Aug 2025).

1. Problem setting and operational scope

GelSLAM is motivated by a standard limitation of vision-based pose and shape perception: accuracy can degrade under occlusion, with specular or transparent materials, in cramped workspaces, and during in-hand manipulation where cameras cannot directly observe the contact region. Tactile sensing addresses a different part of the sensing problem. At contact, it directly measures local surface geometry with high precision, is immune to visual occlusion, is less sensitive to ambient lighting and material optical properties, and provides a strong local signal even on low-depth surfaces.

The target domain is therefore not generic scene-level SLAM, but object-centric SLAM under physical interaction. The system estimates the sensor pose relative to the object, with $\mathbf{T}_i \in SE(3)$ at time $i$ and $\mathbf{T}_0 = \mathbf{I}$ initially. The formulation is especially aimed at in-hand manipulation and dexterous tasks requiring precise pose tracking under occlusion, but the same sensing regime is also applied to high-precision scanning for AR/VR content, biology, archaeology, geology, dentistry, handheld “in-the-wild” scanning, and manipulation settings where accurate object pose must be maintained while contact is ongoing.

A common misconception is that tactile sensing is intrinsically restricted to local contact with no meaningful global structure. GelSLAM directly contests that view by demonstrating loop closures and robust relocalization using only touch, which in turn enables a global object map despite the inherently local support of each tactile observation (Huang et al., 21 Aug 2025).

2. Sensor model, calibration, and geometric representation

The primary sensor configuration uses the GelSight Mini, a vision-based tactile sensor that observes deformations in a compliant elastomer. In the reported implementation it is resized to $320 \times 240$ px for processing, has an effective sensing area of $20 \text{ mm} \times 15 \text{ mm}$, runs at $25$ Hz, and supports a maximum indentation of approximately $2$ mm. A second sensor, GelBelt, is a continuous belt-based vision tactile sensor with a $60 \text{ mm} \times 40 \text{ mm}$ sensing area processed at $216 \times 288$ px, and is demonstrated on a large object in the form of a tree trunk.

Surface normal estimation is derived from photometric stereo. The calibration procedure is data-driven and is described as inspired by Wang et al. (2021). A three-layer MLP with architecture $\mathbf{T}_i \in SE(3)$0 maps per-pixel inputs $\mathbf{T}_i \in SE(3)$1 to surface gradients $\mathbf{T}_i \in SE(3)$2. Ground-truth gradients are obtained by pressing a calibrated metal ball of diameter $\mathbf{T}_i \in SE(3)$3 mm at multiple locations across $\mathbf{T}_i \in SE(3)$4 images and analytically computing surface gradients over the induced circular contacts.

From these gradients, GelSLAM constructs a normal map $\mathbf{T}_i \in SE(3)$5, a height map $\mathbf{T}_i \in SE(3)$6, a contact mask $\mathbf{T}_i \in SE(3)$7, and a curvature map $\mathbf{T}_i \in SE(3)$8. The normal map is obtained from

$\mathbf{T}_i \in SE(3)$9

The height map is integrated from gradients using a fast Poisson solver and is used both to derive a shallow point cloud and to support contact masking. The sensor-relative 3D point associated with pixel $i$0 in frame $i$1 is

$i$2

The curvature map is defined as the Laplacian of the height map approximated from gradients,

$i$3

and then smoothed by a $i$4 Gaussian filter. The paper emphasizes that curvature is invariant to rigid transformations; in GelSLAM this is exploited for robust appearance-based feature extraction via SIFT and for saliency weighting during registration.

This representation differentiates GelSLAM from point cloud-centric tactile pipelines. The system assumes rigid body transformations between contact patches and does not include an explicit force model. Instead, the compliant elastomer is treated as inducing small deformations, on the order of less than a few millimeters, sufficient for normal estimation and shallow height integration. The map itself is maintained in two forms: an online fusion process over coverage keyframes and an offline watertight reconstruction based on Screened Poisson surface reconstruction (Huang et al., 21 Aug 2025).

3. Front-end tracking, keyframing, and loop closure

The front-end processes each incoming tactile frame $i$5 by computing its normal map $i$6, curvature map $i$7, height map $i$8, and contact mask $i$9. Registration to the latest keyframe is then performed by NormalFlow (NF), initialized from the previous frame’s estimate to that keyframe.

For reference frame $\mathbf{T}_0 = \mathbf{I}$0 and target frame $\mathbf{T}_0 = \mathbf{I}$1, the NormalFlow objective is

$\mathbf{T}_0 = \mathbf{I}$2

where $\mathbf{T}_0 = \mathbf{I}$3 denotes the shared contact region and the warping function is

$\mathbf{T}_0 = \mathbf{I}$4

Optimization is performed with Gauss–Newton and is reported as fast, approximately $\mathbf{T}_0 = \mathbf{I}$5 ms per estimate. For robustness, NF uses the top $\mathbf{T}_0 = \mathbf{I}$6 pixels within the shared contact region having the highest curvature, replacing random subsampling with curvature-salient selection.

Tracking quality is evaluated with two curvature-based diagnostics. Curvature Cosine Similarity is

$\mathbf{T}_0 = \mathbf{I}$7

where $\mathbf{T}_0 = \mathbf{I}$8 is the curvature map of frame $\mathbf{T}_0 = \mathbf{I}$9 warped to frame $320 \times 240$0. Shared Curvature Ratio is

$320 \times 240$1

For online tracking, the acceptance thresholds are $320 \times 240$2 and $320 \times 240$3; for offline reconstruction they are $320 \times 240$4 and $320 \times 240$5.

Keyframe selection is deferred until NF no longer tracks reliably. When NF fails according to CCS or SCR, frame $320 \times 240$6 becomes a new keyframe and a tracking edge is inserted between consecutive keyframes. If the latest keyframe is already $320 \times 240$7 and NF from $320 \times 240$8 still fails, tracking is declared lost and a new tracking session starts with $320 \times 240$9 as the first keyframe. The reported typical keyframe interval is approximately $20 \text{ mm} \times 15 \text{ mm}$0 frames. This sparse anchoring is intended to reduce both drift and computation.

Loop closure is built around a dynamically maintained coverage set $20 \text{ mm} \times 15 \text{ mm}$1. When a new keyframe $20 \text{ mm} \times 15 \text{ mm}$2 is added, its non-redundant new contact area relative to the existing coverage set is computed; if that area exceeds $20 \text{ mm} \times 15 \text{ mm}$3, the keyframe is inserted. Coverage keyframes whose unique area becomes less than $20 \text{ mm} \times 15 \text{ mm}$4 are pruned.

Loop detection proceeds in two stages. First, SIFT features are extracted on curvature maps, the new keyframe is matched against coverage keyframes, and a planar 2D warp is estimated with least squares and RANSAC. Pairs with at least $20 \text{ mm} \times 15 \text{ mm}$5 inliers are retained as candidates; this stage is reported at approximately $20 \text{ mm} \times 15 \text{ mm}$6 ms per pair. Second, the candidate warp initializes NormalFlow, which refines the estimate to a full $20 \text{ mm} \times 15 \text{ mm}$7 relative pose $20 \text{ mm} \times 15 \text{ mm}$8. The loop edge is accepted only if CCS and SCR again exceed their thresholds. This architecture makes curvature the common signal for both feature-based candidate generation and geometry-based registration refinement (Huang et al., 21 Aug 2025).

4. Back-end optimization and map reconstruction

The back-end maintains a pose graph over keyframes $20 \text{ mm} \times 15 \text{ mm}$9 with an edge set $25$0 containing both tracking edges and loop edges. For each $25$1, with NormalFlow-estimated relative pose $25$2, the residual is

$25$3

The optimization problem is

$25$4

with fixed covariance $25$5 providing information weighting. The solver is Levenberg–Marquardt in GTSAM. For repetitive patterns, the paper also reports an optional robustification using GNC to reject outlier edges.

Reconstruction is split into an online path and an offline path. The online method performs fast fusion over coverage keyframes. Each coverage keyframe is reprojected into overlapping coverage keyframes; points within the corresponding contact masks are averaged using weights derived from a sigmoid function of distance to the contact boundary, so that border regions are down-weighted. The averaged points are triangulated per keyframe and the resulting meshes are merged. The stated outcome is smooth, visually coherent surfaces in real time.

The offline method applies Screened Poisson surface reconstruction to fused oriented points, with normals taken from the normal maps $25$6. This produces a watertight mesh and is the high-fidelity reconstruction mode reported in the quantitative reconstruction experiments.

The implementation stack comprises ROS2 for streaming and process orchestration, GTSAM for pose graph optimization, SIFT on curvature maps, Gauss–Newton for NF, and Screened Poisson reconstruction for the offline mesh. Online execution may skip some keyframes for loop closure when the loop module is busy, whereas the offline mode includes all detected loops. This distinction explains why online and offline results are close but not identical (Huang et al., 21 Aug 2025).

5. Quantitative performance, reconstruction fidelity, and ablations

The reported long-horizon tracking benchmark contains $25$7 episodes over $25$8 objects, with mean absolute error evaluated after excluding the perfectly smooth symmetric Ball and Cylinder. Average cumulative motion per episode is approximately $25$9 in rotation and $2$0 mm in translation. Ground truth is provided by MoCap for tracking and by CAD for printed reconstruction objects.

Method	Rotation MAE $2$1	Translation MAE $2$2
GS-Online	$2$3	$2$4 mm
GS-Offline	$2$5	$2$6 mm
NF	$2$7	$2$8 mm
ICP (point-to-plane)	$2$9	$60 \text{ mm} \times 40 \text{ mm}$0 mm
FilterReg	$60 \text{ mm} \times 40 \text{ mm}$1	$60 \text{ mm} \times 40 \text{ mm}$2 mm
FPFH+RI	$60 \text{ mm} \times 40 \text{ mm}$3	$60 \text{ mm} \times 40 \text{ mm}$4 mm

Within that comparison, GS-Online is reported to reduce rotation MAE by approximately $60 \text{ mm} \times 40 \text{ mm}$5 and translation MAE by approximately $60 \text{ mm} \times 40 \text{ mm}$6 relative to NF without loops, indicating that loop closure is the principal mechanism for drift correction. Tac2Structure, the full SLAM baseline considered in the study, is described as exhibiting failure modes similar to FPFH+RI on low-texture objects, with poor accuracy attributed to unreliable features and outlier loops.

For reconstruction, the evaluation covers $60 \text{ mm} \times 40 \text{ mm}$7 printed objects using Chamfer Distance (CD) and Normal Cosine Distance (NCD). The average CD is approximately $60 \text{ mm} \times 40 \text{ mm}$8 mm across objects of average size approximately $60 \text{ mm} \times 40 \text{ mm}$9 mm. Reported examples include Dice with CD $216 \times 288$0 mm and NCD $216 \times 288$1 for small/large variants, Almond with CD $216 \times 288$2 mm and NCD $216 \times 288$3, Seed with CD $216 \times 288$4 mm and NCD $216 \times 288$5, and Lime with CD $216 \times 288$6 mm and NCD $216 \times 288$7. The average NCD over simulated local normal maps is approximately $216 \times 288$8, based on $216 \times 288$9 touches per object with normals aligned to the local surface. The textures are reported to closely match CAD, with slight loss of sharp microdetails due to limited resolution and small residual misalignments.

Runtime is measured on an AMD Ryzen 7 PRO 7840U with $\mathbf{T}_i \in SE(3)$00 cores at $\mathbf{T}_i \in SE(3)$01 GHz. Mean per-frame geometric property computation $\mathbf{T}_i \in SE(3)$02 takes $\mathbf{T}_i \in SE(3)$03 ms, NormalFlow tracking takes $\mathbf{T}_i \in SE(3)$04 ms, and total tracking time is $\mathbf{T}_i \in SE(3)$05 ms; the paper notes that operation below the $\mathbf{T}_i \in SE(3)$06 Hz camera rate is achievable with pipelining and asynchrony. Per-keyframe loop closure requires $\mathbf{T}_i \in SE(3)$07 ms for loop detection, $\mathbf{T}_i \in SE(3)$08 ms for pose graph optimization, and $\mathbf{T}_i \in SE(3)$09 ms for coverage set update, for a total of $\mathbf{T}_i \in SE(3)$10 ms.

The ablation studies emphasize three design choices. First, NF-KF outperforms NF without keyframes by reducing drift and computation, while maintaining similar accuracy to NF-OrigKF but running twice as fast because it uses one NF solve per frame instead of two. Second, GS-OnlySIFT performs poorly because of occasional false loops, while GS-OnlyNF is better but much slower and can miss valid loops without a good initialization. Third, curvature-based SIFT is stronger than SIFT on raw tactile images: GS-ImageSIFT is weaker because raw appearance lacks invariance, whereas curvature SIFT yields more reliable initializations and leads to the best results in GS-Offline and GS-Online (Huang et al., 21 Aug 2025).

6. Failure modes, practical use, and research directions

The reported limitations are specific and technically consequential. Extremely low-texture surfaces provide few saliency cues because curvature is weak, which makes loop detection difficult and limits drift correction; Block and Can are given as examples. Repetitive textures pose the opposite problem: they increase the risk of false loop closures. Standard least-squares pose graph optimization is described as sensitive to these outliers, and while GNC robust PGO mitigates the issue, it may also reject some true loops and create local disconnections. Rapid motion or contact loss can interrupt tracking entirely, in which case GelSLAM starts a new session and depends on later loop closures for reconnection. Because sensing remains local, substantial surface coverage and time are required for full reconstructions; the GelBelt trunk demonstration shows slight global distortions, including minor cylindrical shrinkage at the ends. Contact-mask thresholds may also require manual tuning, suggesting a need for adaptive thresholding.

These limitations also clarify what GelSLAM is and is not. It is not an explicit contact-mechanics estimator, does not model slip or force in its current form, and does not rely on multimodal fusion. Its strength lies instead in a geometry-first tactile representation that uses normals and curvature as stable cues for registration and relocalization. A plausible implication is that the method is best suited to tasks where persistent contact is acceptable and geometric precision at contact is more valuable than broad field-of-view coverage.

The application profile follows directly from that design. The system tracks the sensor’s $\mathbf{T}_i \in SE(3)$11 pose relative to the object during contact, which is directly relevant to in-hand manipulation, tool handling, and assembly under occlusion. The tracking module runs at near-real-time rates, loop closures operate asynchronously per keyframe, and the pose graph produces drift-corrected trajectories that are described as suitable for feedback control. Online fast fusion additionally provides immediate coverage feedback for human operators. Reported demonstrations include small seeds and fossils reconstructed with submillimeter CD, accurate long-horizon tracking on everyday tools, and large-scale trunk scanning with GelBelt.

The future work identified in the paper extends along five axes: multi-sensor fusion of vision and tactile sensing to reduce required touch coverage and accelerate reconstruction; dynamic object modeling incorporating contact mechanics, slip, and force cues; stronger outlier rejection for repetitive textures and loop validation that combines geometry with learned semantics; active exploration policies that minimize redundancy and optimize coverage or uncertainty reduction; and scalability through multiple tactile sensors, larger arrays, and improved real-time meshing and streaming. In that sense, GelSLAM is best understood not as a general replacement for vision-based SLAM, but as a specialized tactile SLAM framework for manipulation-centered spatial perception in regimes where contact geometry is the dominant source of reliable information (Huang et al., 21 Aug 2025).