Incremental Semantics-Aided Meshing from LiDAR-Inertial Odometry and RGB Direct Label Transfer

Abstract: Geometric high-fidelity mesh reconstruction from LiDAR-inertial scans remains challenging in large, complex indoor environments -- such as cultural buildings -- where point cloud sparsity, geometric drift, and fixed fusion parameters produce holes, over-smoothing, and spurious surfaces at structural boundaries. We propose a modular, incremental RGB+LiDAR pipeline that generates incremental semantics-aided high-quality meshes from indoor scans through scan frame-based direct label transfer. A vision foundation model labels each incoming RGB frame; labels are incrementally projected and fused onto a LiDAR-inertial odometry map; and an incremental semantics-aware Truncated Signed Distance Function (TSDF) fusion step produces the final mesh via marching cubes. This frame-level fusion strategy preserves the geometric fidelity of LiDAR while leveraging rich visual semantics to resolve geometric ambiguities at reconstruction boundaries caused by LiDAR point-cloud sparsity and geometric drift. We demonstrate that semantic guidance improves geometric reconstruction quality; quantitative evaluation is therefore performed using geometric metrics on the Oxford Spires dataset, while results from the NTU VIRAL dataset are analyzed qualitatively. The proposed method outperforms state-of-the-art geometric baselines ImMesh and Voxblox, demonstrating the benefit of semantics-aided fusion for geometric mesh quality. The resulting semantically labelled meshes are of value when reconstructing Universal Scene Description (USD) assets, offering a path from indoor LiDAR scanning to XR and digital modeling.

• The paper presents an end-to-end pipeline that fuses high-resolution semantic labels with LiDAR-inertial odometry to generate accurate, semantically-rich 3D meshes.
• It employs class-aware volumetric fusion and explicit uncertainty decomposition, improving surface fidelity and enabling precise error attribution.
• Quantitative results on Oxford Spires show reduced RMSE and enhanced completeness compared to traditional geometry-only methods.

Incremental Semantics-Aided Meshing from LiDAR-Inertial Odometry and RGB Direct Label Transfer

Problem Motivation and Background

The accurate reconstruction of semantically-rich 3D meshes from indoor environments remains a longstanding challenge due to the inherent limitations of LiDAR-based methods, such as point-cloud sparsity, geometric drift, and fixed-parameter volumetric fusion. Traditional geometry-only pipelines, e.g., ImMesh and Voxblox, lack semantic awareness, resulting in artifacts at structural boundaries, holes, and over-smoothed or incomplete surfaces, especially in complex indoor and heritage environments. Conventional semantic-3D reconstruction approaches primarily utilize RGB-D streams, which require dense and co-registered depth data—unattainable for large scenes where LiDAR excels. The presented work addresses this gap by introducing an end-to-end incremental meshing framework that fuses semantic labels from vision foundation model (VFM) segmentations with LiDAR-inertial odometry (LIO) data.

Methodology

The proposed pipeline operates at the LiDAR frame rate and involves three principal stages: segmentation-driven label transfer, tightly-coupled LIO, and class-aware volumetric fusion.

After preprocessing with FAST-LIO2 for accurate pose estimation and deskewing, high-resolution semantic panoptic labels are predicted in every RGB frame using OneFormer. These 2D segmentation masks are incrementally projected and fused into the LiDAR-inertial odometry map via ego-motion-compensated transformation, addressing temporal misalignments and sampling sparsity inherent to cross-modal datasets. Robustness in label transfer is maintained using boundary-aware filtering, morphological erosion, outlier rejection, and depth discontinuity checks.

Label-aware TSDF fusion modifies classical TSDF updates by incorporating per-voxel semantic histograms, temporally consolidated "Dirichlet-like" label evidence, and dynamic, class- and range-conditioned truncation distances. Volumetric fusion weights are adaptively modulated using empirical priors for each semantic class, leading to improved surface capturing (e.g., thin railings versus broad walls).

Figure 1: Flowchart illustrating the modular pipeline from semantic segmentation to class-aware volumetric fusion and mesh extraction.

Following integration, the volumetric field is meshed via marching cubes, with semantic class and instance identifiers propagated from voxels to mesh triangles, establishing the basis for Universal Scene Description (USD) asset export.

Uncertainty Analysis Framework

A distinguishing aspect of this work is the explicit quantification and partitioning of uncertainty in the final mesh. Semantic and geometric uncertainties are separately estimated per voxel using a combination of statistical consensus (via label histograms), neighbor label disagreement, TSDF gradient excess, and principal direction variability from raw points. Persistent scores for each uncertainty channel are maintained using exponential moving averages, enabling the diagnostic separation of errors arising from semantic transfer (e.g., segmentation or projection artifacts) and geometric inconsistencies (e.g., registration drift, fusion errors).

Figure 2: Visualization of per-voxel semantic and geometric uncertainty in the NTU VIRAL dataset, highlighting their distinct spatial characteristics.

The framework enables a voxel-wise decomposition into four regimes: confident, geometry-uncertain, semantics-uncertain, and joint uncertainty, which in turn supports robust analysis and error attribution in the context of multi-modal fusion.

Quantitative and Qualitative Results

Empirical evaluation focuses on both geometric and uncertainty metrics. The Oxford Spires (Christ Church College Scene) dataset is used for quantitative assessment, while the NTU VIRAL (NYA01 Scene) provides qualitative insights.

On Oxford Spires, the proposed pipeline outperforms baseline geometry-only methods:

• Accuracy (RMSE): 14.54 cm (Ours) vs. 17.24 cm (Voxblox), 17.40 cm (ImMesh)
• Completeness: 98.55% (Ours) vs. 97.17% (Voxblox), 95.99% (ImMesh)
• F1 Score: 98.58% (Ours) vs. 96.98% (Voxblox), 96.31% (ImMesh)

Both the lowest accuracy error and highest completeness were achieved by the semantics-aware method, substantiating the claim that semantic guidance improves both local surface fitting and global coverage.

Figure 3: Three-way comparison of reconstructed meshes on the Oxford Spires dataset—ImMesh and Voxblox omit fine details and structural continuity captured by the semantics-aided pipeline.

Figure 4: Qualitative comparison on the NTU VIRAL dataset—semantic fusion yields cleaner, more consistent meshes in regions with structural ambiguity and sparse data.

Uncertainty decomposition demonstrates a clear spatial separation: geometric uncertainty localizes along depth discontinuities and high-curvature features, while semantic uncertainty is concentrated at class transition boundaries and regions with ambiguous or inconsistent segmentation evidence. The limited overlap between these channels highlights the complementarity of the two error sources.

Implications and Future Directions

This methodology demonstrates that leveraging semantic priors from generalist vision models in fusion with LiDAR and inertial data substantially enhances mesh reconstruction quality without the need for dense, depth-synchronous data streams. The result is not only higher fidelity geometry but also fully attributed scene assets compatible with modern XR and digital twin pipelines via USD export.

Practically, this approach enables more robust and reusable digital reconstructions of large-scale indoor environments, particularly those with architectural complexity or sparse visual texture where LiDAR excels but geometric-only fusion flounders. The modular and incremental design supports extensibility to emerging segmentation backbones, additional sensor modalities, and the integration of more sophisticated semantic regularization.

Theoretically, the explicit modeling and partitioning of uncertainty in fused multi-modal reconstructions sets a precedent for principled diagnostic analysis in scene modeling pipelines, potentially informing active sensor planning, confidence-aware downstream perception, and automated error correction.

Future research should address residual limitations, including drift on extended trajectories, calibration dependence, temporal and spatial mismatches, and the challenges of reconstructing thin structures, specular surfaces, and heavy indoor clutter. Further integration of semantic priors and geometric optimization is warranted.

Conclusion

This work establishes a systematic pipeline for incremental 3D mesh reconstruction by integrating VFM-derived semantic labels with LIO-registered LiDAR data through class-aware volumetric fusion. Quantitative and qualitative results demonstrate superior geometric fidelity and scene coverage compared to state-of-the-art geometric baselines. Explicit decomposition of semantic and geometric uncertainty facilitates robust error attribution. The produced USD-compatible assets directly serve applications in XR, digital twins, and architectural documentation, while also motivating future advances in semantics-driven multi-modal 3D mapping (2604.09478).