Consistent Scene Understanding in 3D Gaussian Splatting via Multi-Cue Mask Refinement

Abstract: Reliable instance-level scene understanding is a fundamental prerequisite for object-level interactions and high-fidelity 3D representations. While current methods often leverage 2D foundation segmentation models to obtain these priors, their 2D-centric design typically yields fragmented masks and inconsistent predictions across different views. To address these issues, we propose a novel framework that produces consistent 2D instance masks to guide the optimization of 3D Gaussian Splatting (3DGS) feature fields. Our framework consists of three main stages. (1) Multi-Cue Extraction that generates synergistic semantic, geometric, and structural priors from input images. (2) Multi-Cue-Guided Mask Merging process that consolidates fragmented masks using a composite merge score derived from semantic, depth, and edge cues. (3) Cross-View Mask Matching that establishes globally consistent identity assignments across all viewpoints. By transforming viewpoint-specific segments into coherent 3D primitives, our approach enables stable 3D instance segmentation and effective downstream editing tasks. Experiments demonstrate that our method significantly improves cross-view consistency and segmentation stability over existing baselines while maintaining high-fidelity photometric reconstruction.

• The paper introduces a multi-cue mask refinement approach that integrates semantic, geometric, and structural cues to transform fragmented 2D segmentation into consistent 3D representations.
• It refines over-segmented outputs from methods like SAM by merging masks using multi-view clustering and composite affinity scores, significantly reducing mask count and improving mIoU.
• Experimental results on LERF and Replica datasets demonstrate improved boundary accuracy, photometric fidelity, and overall object-level consistency in 3D instance segmentation.

Consistent Scene Understanding in 3D Gaussian Splatting via Multi-Cue Mask Refinement

Introduction

Instance-level scene understanding in 3D is central to object-based interactions, semantic navigation, and high-fidelity scene editing. While 3D Gaussian Splatting (3DGS) provides an explicit scene representation enabling real-time radiance field rendering, its integration with 2D segmentation priors such as the Segment Anything Model (SAM) is severely encumbered by the intrinsic 2D-centric fragmentation and view inconsistency of the latter. The paper "Consistent Scene Understanding in 3D Gaussian Splatting via Multi-Cue Mask Refinement" (2607.01708) addresses the core limitations of using 2D foundation models for 3D object instance segmentation, introducing a comprehensive multi-cue mask refinement pipeline that produces compact, view-consistent object-level feature fields for 3DGS.

Figure 1: Cross-view inconsistency from SAM over-segmentation; automatic masks from SAM demonstrate fragmented identities and erroneous merges across views, impeding 3D scene understanding.

Methodology

Multi-Cue Mask Refinement Pipeline

The proposed pipeline operates in three key stages: (1) Multi-cue feature extraction from multi-view images; (2) Multi-cue-guided mask merging (MCM) utilizing semantic, geometric, and structural cues; and (3) cross-view mask matching and feature lifting into 3D Gaussian primitives.

Figure 2: Overview of the pipeline: multi-view images are processed to extract SAM masks and multi-cue priors, which are refined with composite merge scores and merged for global mask consistency, before being lifted into 3D.

Multi-Cue Feature Extraction

• Semantic Cue: Dense DINOv2 embeddings provide robust object-level descriptors, allowing grouping of perceptually similar fragments disregarding minor local variations.
• Geometric Cue: Monocular depth maps from DepthAnythingV2 inject geometric priors, enforcing hard constraints to prevent merges across significant scene-scale depth discontinuities.
• Structural Cue: Laplacian-of-Gaussian (LoG) edge maps supply continuous boundary evidence, preserving salient structural discontinuities even in textureless or appearance-aliased regions.

  Figure 3: Visualization of multi-cue features—semantic (DINOv2), geometric (depth), and structural (LoG edge), cooperatively resolve over-segmentation, aligning region boundaries with physical object extents.

Multi-Cue-Guided Mask Merging (MCM)

Mask merging is framed as an adjacency-constrained clustering problem, admitting candidate merges strictly by:

• Hard depth threshold ($|\bar{d}_i - \bar{d}_j| < \tau_{depth}$)
• Composite affinity score, combining cosine similarity of semantic features, soft depth affinity, and penalties derived from the maximum boundary gradients in both depth and LoG edges

The merge process greedily consolidates over-segmented 2D fragments into semantically coherent regions, iteratively updating adjacency and merge scores until convergence.

Cross-View Mask Matching and 3D Feature Lifting

After per-view MCM, global identity consistency is achieved by cross-view clustering of the merged mask descriptors. Masks from different views are considered equivalent if their mean semantic feature similarity exceeds threshold $\tau_{match}$, enforced via global affinity graph clustering.

Projection of Gaussians into visible camera views enables the assignment of object identity via majority voting, and a variance-based filter discards Gaussians with inconsistent multiview feature support, suppressing single-view and occlusion-induced noise.

Joint Optimization

The entire 3DGS and feature field are then jointly optimized:

• Photometric loss ($\mathcal{L}_{render}$): preserves appearance fidelity;
• Semantic loss ($\mathcal{L}_{semantic}$): penalizes feature discrepancy between lifted 2D priors and learnable 3D attributes;
• 3D regularization ($\mathcal{L}_{3D-reg}$): enforces local spatial smoothness via k-NN consistency.

A two-stage schedule stabilizes early geometric convergence by freezing object features before introducing semantic and structural refinement.

Experimental Results

Quantitative Performance

Substantial improvements are reported on LERF and Replica datasets:

• Segmentation: mIoU increases to 0.728 (from 0.685 in GaussianGrouping and 0.712 in SAGA); boundary mIoU achieves 0.677, surpassing all baselines.
• Over-segmentation resolution: Mask count is reduced from 9,627 (GaussianGrouping) to 67, indicating a dramatic consolidation of fragmented 2D proposals into actionable 3D primitives.
• Photometric fidelity: Rendering quality (PSNR: 28.6, SSIM: 0.89) aligns with or exceeds vanilla 3DGS and leading 3D segmentation methods, indicating that semantic field refinement does not compromise radiance reconstruction.

Qualitative Analysis

Figure 4: Qualitative result—faithful 3D segmentation with consistent identities and boundaries across views.

Figure 5: Comparative results: baseline methods show identity flicker and boundary ambiguity, whereas the proposed approach maintains global mask consistency under significant view changes.

Figure 6: Cross-view mask matching ablation. Without global association, identity inconsistency persists. Full pipeline yields view-stable object representations.

Additional results on real-world and diverse synthetic scenes are presented, highlighting the robustness and generalization of the method. The integration of multi-cue priors is ablated to quantify the incremental accuracy and compactness improvements.

Figure 7: Additional qualitative results—globally coherent 3D instance masks across diverse scenes.

Model and Feature Backbone Selection

Systematic analysis demonstrates the superiority of DINOv2 (ViT-B/14) for semantic affinity (intra-object similarity = 0.89, inter-object = 0.21), and highlights the capability of DepthAnythingV2 to disambiguate appearance-aliased but spatially distinct regions ("white-on-white dilemma"). LoG is shown to be robust for boundary detection without entangling texture noise, outperforming Canny and Sobel in continuous-edge scenarios.

Limitations and Future Directions

While dramatically reducing over-segmentation and enforcing global consistency, the method assumes SAM provides an over-complete initial set. Severe under-segmentation or missing boundaries cannot be rectified by this refinement pipeline. Furthermore, occlusion-heavy or highly dynamic contexts may yield localized inconsistencies in per-view refinement.

Future goals include:

• Deploying consistent 3D masks as editing indices for generative and manipulation tasks in 3DGS
• Extension to dynamic scene understanding and temporally stable semantic tracking
• Enriching feature regularization with language-driven modules (e.g., CLIP, LangSplat) for open-vocabulary and text-driven 3D interaction

Conclusion

This work establishes a robust framework for transforming fragmented, inconsistent 2D foundation model outputs into unified, physically grounded, view-consistent object primitives in 3DGS. By synthesizing semantic, geometric, and structural cues at the mask merging and cross-view matching stages, and rigorously filtering labels via multiview consensus, the approach delivers high-fidelity 3D instance segmentation that is object-level actionable for downstream editing and manipulation without sacrificing photometric fidelity (2607.01708).

Figure 8: AI-generated illustrations deployed for effective visualization of the main pipeline.

Implications: The methodology advances practical 3DGS scene editing and interactive applications, and provides a blueprint for future extensions toward open-vocabulary object-centric 3D content creation and manipulation.