VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation (2107.13824v2)

Abstract: In recent years, sparse voxel-based methods have become the state-of-the-arts for 3D semantic segmentation of indoor scenes, thanks to the powerful 3D CNNs. Nevertheless, being oblivious to the underlying geometry, voxel-based methods suffer from ambiguous features on spatially close objects and struggle with handling complex and irregular geometries due to the lack of geodesic information. In view of this, we present Voxel-Mesh Network (VMNet), a novel 3D deep architecture that operates on the voxel and mesh representations leveraging both the Euclidean and geodesic information. Intuitively, the Euclidean information extracted from voxels can offer contextual cues representing interactions between nearby objects, while the geodesic information extracted from meshes can help separate objects that are spatially close but have disconnected surfaces. To incorporate such information from the two domains, we design an intra-domain attentive module for effective feature aggregation and an inter-domain attentive module for adaptive feature fusion. Experimental results validate the effectiveness of VMNet: specifically, on the challenging ScanNet dataset for large-scale segmentation of indoor scenes, it outperforms the state-of-the-art SparseConvNet and MinkowskiNet (74.6% vs 72.5% and 73.6% in mIoU) with a simpler network structure (17M vs 30M and 38M parameters). Code release: https://github.com/hzykent/VMNet

• The paper introduces VMNet, a novel hybrid network that integrates voxel and mesh data to enhance geodesic-aware 3D semantic segmentation.
• It employs intra- and inter-domain attentive modules to effectively combine Euclidean and geodesic features, improving accuracy while reducing network complexity.
• Empirical results on the ScanNet dataset show VMNet achieving 74.6% mIoU, outperforming SparseConvNet and MinkowskiNet with only 17M parameters.

Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation of Indoor Scenes

The paper presents a sophisticated approach to 3D semantic segmentation tailored for the complexities of indoor scenes. Leveraging the limitations of traditional voxel-based segmentation methods, the authors introduce the Voxel-Mesh Network (VMNet), which ingeniously combines voxel and mesh representations to enhance segmentation accuracy by incorporating both Euclidean and geodesic information.

The challenges inherent in voxel-based methods arise largely from their insensitivity to the intrinsic geometric correlations in 3D data. Such methods often blur features of spatially proximal objects and grapple with complex geometries due to the absence of geodesic awareness. VMNet addresses these issues by simultaneously operating on both voxelized data, which efficiently captures Euclidean spatial contexts, and mesh data, which maintains critical surface continuity via geodesic information.

A central feature of the proposed system is the integration of two novel modules designed for this dual-domain approach: the intra-domain attentive module and the inter-domain attentive module. The intra-domain module enhances feature aggregation within each domain, while the inter-domain module facilitates the adaptive fusion of multi-domain features, thus allowing VMNet to draw semantic insights from both geometric structures and spatial layouts.

The paper presents compelling experimental results, particularly on the ScanNet dataset, demonstrating the advantages of VMNet over established SparseConvNet and MinkowskiNet architectures. VMNet achieves a segmentation accuracy of 74.6% mean Intersection over Union (mIoU), which marks a notable improvement compared to the 72.5% and 73.6% scores of SparseConvNet and MinkowskiNet, respectively. This benchmark is reached with a more streamlined network structure, comprising fewer parameters (17M compared to 30M and 38M), underscoring the efficiency of the proposed architecture.

A detailed discussion on the practical implications of VMNet indicates its potential for application in large-scale 3D scene understanding tasks where accurate semantic segmentation is critical, such as in robotics, augmented reality, and autonomous navigation. Theoretically, the introduction of concurrent handling of dual-domain information sets a precedent for future developments in multi-modal neural networks, encouraging further research into harmonizing disparate data representations.

The paper's contribution is significant in addressing the persistent challenges faced by voxel-based segmentation methods in 3D scene understanding. By innovatively integrating voxel and mesh representations, VMNet not only advances the field of 3D semantic segmentation but also paves the path for exploring new architectures that leverage the semantic synergy of multiple geometric data forms.