Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces (2403.20275v1)

Abstract: Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. Our method optimises 3D Gaussian primitives to accurately model the object's geometry at points of contact. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. Touch is particularly useful when considering non-Lambertian objects (e.g. shiny or reflective surfaces) since contemporary methods tend to fail to reconstruct with fidelity specular highlights. By combining vision and tactile sensing, we achieve more accurate geometry reconstructions with fewer images than prior methods. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach, offering significant improvements in reconstruction quality.

• The paper presents a novel method that fuses tactile sensing and multi-view vision data via 3D Gaussian Splatting for improved surface reconstruction.
• It employs advanced regularization techniques, including 3D transmittance and edge-aware smoothness, to optimize geometry around touch locations.
• The approach outperforms visual-only methods in minimal view scenarios, significantly enhancing reconstruction of glossy and reflective surfaces.

Tactile-Informed 3D Gaussian Splatting for Enhanced Surface Reconstruction

Introduction

Reconstructing the 3D geometry of objects, particularly those with challenging surfaces such as glossy or reflective materials, continues to be a significant undertaking in the field of computer vision and robotics. Current methods mainly rely on visual data which, while effective in many scenarios, show limitations when confronted with non-Lambertian surfaces or when the number of available views is restricted. This research introduces a novel approach, Tactile-Informed 3D Gaussian Splatting (3DGS), which integrates tactile sensing with multi-view vision data to achieve superior surface reconstruction and novel view synthesis.

Literature Review

Tactile Sensing for 3D Object Reconstruction

The exploration of tactile sensing for 3D object reconstruction introduces high-resolution, optical-based tactile sensors, enabling direct object surface interaction for detailed geometric information acquisition. Recent advancements have focused on generating 3D shapes from tactile data, showcasing potential in tactile-only shape reconstruction. However, the challenge remains to effectively combine tactile with visual information for comprehensive 3D modelling, especially for objects exhibiting non-Lambertian characteristics.

Novel-View Synthesis on Reflective Surfaces

Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) methodologies have significantly advanced the field of novel-view synthesis. Despite their success, both encounter difficulties accurately modelling specular and glossy surfaces due to the inherent nature of volumetric rendering. Efforts to overcome these challenges include the modification of radiance parameterisation, as seen in Ref-NeRF, or the separate modelling of direct and indirect illumination seen in NeRO. Nonetheless, these approaches require extensive computational resources and struggle with minimal view settings.

Methodology

The core of our approach, Tactile-Informed 3DGS, relies on the integration of tactile data (local depth maps) and multi-view vision data within a 3D Gaussian Splatting framework. The process involves Gaussian initialisation and optimisation through tactile and vision data integration, accompanied by a series of regularisation techniques devised to enhance the reconstruction quality.

Regularisation Techniques

• 3D Transmittance: A novel regularisation that guides the optimisation of 3D Gaussians around touch locations, improving modelling of object geometry.
• Edge-Aware Smoothness with Proximity-Based Masking: This regularisation modulates edge-aware smoothness loss according to proximity to touch locations, refining reconstruction further away from touched surfaces.

Performance Evaluation

The effectiveness of Tactile-Informed 3DGS was evaluated on datasets featuring objects with glossy and reflective surfaces. This method demonstrated a significant improvement in geometry reconstruction, particularly in minimal view scenarios, where it outperformed existing 3D reconstruction methods based on visual data alone. Furthermore, the integration of tactile data allowed for a more robust reconstruction process, observable in both synthetic and real-world datasets.

Limitations and Future Directions

While the current methodology marks an advancement in tactile-informed object reconstruction, it does acknowledge limitations, particularly regarding the efficiency of tactile data acquisition. Future research directions include the development of adaptive tactile sampling strategies and the exploration of this multimodal approach for transparent object reconstruction.

Conclusion

This work presents a significant step forward in the reconstruction of challenging surface objects by highlighting the potential of integrating tactile information with visual data. The proposed Tactile-Informed 3DGS method not only achieves superior surface reconstruction and novel view synthesis but also demonstrates a promising direction for future research in the field of multimodal sensory integration in 3D object reconstruction. Through continued exploration and refinement, this approach has the potential to broaden the applicability and effectiveness of 3D reconstruction methodologies, particularly for applications in robotics, virtual reality, and 3D modeling.