Multi-Cue-Guided Mask Merging

Updated 6 July 2026

The paper introduces a multi-cue merging technique that consolidates fragmented masks using semantic, depth, and structural cues for robust 3D scene understanding.
It applies a composite scoring system combining cosine similarity, depth affinity, and boundary penalties to iteratively merge masks per view for consistent cross-view matching.
The method improves segmentation quality by reducing mask fragmentation and enhancing 3D lifting and feature-field optimization while addressing common failure modes of single-cue approaches.

Searching arXiv for the cited papers to ground the article. Multi-Cue-Guided Mask Merging denotes a class of mask consolidation procedures in which fragmented masks are merged by jointly evaluating heterogeneous evidence rather than a single cue. In the formulation introduced for 3D Gaussian Splatting scene understanding, it is the per-view stage that converts raw SAM automatic mask generation outputs into larger, coherent instance masks by combining semantic similarity, depth affinity, and structural boundary evidence; these refined masks then support cross-view identity assignment, 3D lifting, and feature-field supervision (Park et al., 2 Jul 2026). More broadly, related work uses analogous multi-cue mask-merging principles in online zero-shot 3D segmentation, multi-view mask matching, controllable diffusion, fusion, model merging, and harmonization, but the canonical technical instantiation is the 3DGS-oriented mask refinement framework of (Park et al., 2 Jul 2026).

1. Position within the 3DGS scene-understanding pipeline

In "Consistent Scene Understanding in 3D Gaussian Splatting via Multi-Cue Mask Refinement" (Park et al., 2 Jul 2026), Multi-Cue-Guided Mask Merging is the second stage of a four-stage pipeline. The complete pipeline comprises Multi-Cue Feature Extraction, Multi-Cue-Guided Mask Merging, Cross-View Mask Matching and 3D Lifting, and 3DGS Feature-Field Optimization. The first stage extracts DINOv2 semantic features, DepthAnythingV2 monocular depth, and Laplacian-of-Gaussian edge maps from each input image; the second stage operates within each view and merges fragmented masks; the third stage establishes globally consistent IDs across views; the fourth stage supervises learnable Gaussian features with lifted 3D features.

Within that architecture, the merging stage functions as the bridge between raw per-view SAM masks and globally consistent 3D instances. Its outputs are a reduced set of per-view masks, each equipped with a geometric extent, a refined semantic descriptor computed as a DINO-v2 average over the merged region, and refined depth statistics. These outputs are the units passed to cross-view mask matching and then projected onto 3D Gaussians via majority voting and feature fusion.

The method is motivated by a specific failure mode of 2D foundation segmentation priors in multiview 3D reconstruction: their 2D-centric design yields fragmented masks and inconsistent predictions across views. Multi-Cue-Guided Mask Merging addresses the first part of that problem by consolidating viewpoint-specific fragments before cross-view clustering. This suggests that the method is not merely a post-processing heuristic, but a conditioning stage that stabilizes every downstream component relying on mask identity, geometric support, and semantic coherence.

2. Cue extraction and mask representation

The merging stage consumes three cue families: semantic, geometric, and structural cues (Park et al., 2 Jul 2026). For each image view, SAM provides an initial mask set

$\mathcal{M}=\{m_i\}_{i=1}^K,\qquad m_i \subset \{1,\dots,H\}\times\{1,\dots,W\}.$

Each mask is represented by its binary region $m_i$ , its bounding box $\text{bbox}(m_i)$ , its boundary pixel set $\partial m_i$ , a semantic descriptor $f_i\in\mathbb{R}^{768}$ , and depth statistics including mean depth $\bar d_i$ and boundary depth gradients sampled from the depth-gradient map.

The semantic cue is derived from DINOv2 ViT-B/14. The image is resized to $518\times 518$ , producing per-patch 768-d features that are assigned or upsampled to pixels. For a mask $m_i$ , the semantic descriptor is the mean DINO feature over the mask: $f_i=\frac{1}{|m_i|}\sum_{p\in m_i} f(p),$ followed by L2 normalization,

$f_i\leftarrow \frac{f_i}{\|f_i\|_2},$

so that cosine similarity is numerically stable.

The geometric cue is derived from DepthAnythingV2. The model predicts a normalized depth map

$m_i$ 0

For each mask, the mean depth is

$m_i$ 1

Depth is used in three distinct ways: as a hard constraint for candidate rejection, as a soft affinity term through a Gaussian kernel on mean-depth difference, and as a boundary penalty through the depth-gradient magnitude $m_i$ 2 evaluated along mask boundaries.

The structural cue is a LoG-based edge-strength map $m_i$ 3, where $m_i$ 4 encodes continuous edge-response magnitude. Along candidate boundaries, strong LoG responses discourage merging across object-object or object-background discontinuities.

A central design feature is that all per-mask attributes are updated after every merge. A merged mask therefore becomes not only a larger region $m_i$ 5, but also a new semantic unit with recomputed DINO descriptor, mean depth, and boundary statistics. This update rule is critical because the algorithm is greedy and iterative rather than single-pass.

3. Composite merge score and greedy iterative procedure

The core decision rule is a composite merge score that fuses semantic affinity, depth compatibility, and boundary penalties (Park et al., 2 Jul 2026): $m_i$ 6 where $m_i$ 7. The global constants used in experiments are

$m_i$ 8

The semantic term

$m_i$ 9

captures appearance-level affinity. The depth-affinity term

$\text{bbox}(m_i)$ 0

rewards similar depth layers. The depth-boundary and edge penalties are defined by the maximum boundary gradient magnitudes, so large discontinuities suppress merging even when semantic similarity is high.

Candidate generation is restricted to adjacent masks. Two masks are considered adjacent only if all three geometric criteria hold: their bounding boxes are within $\text{bbox}(m_i)$ 1 pixels, their dilated masks intersect, and their contact ratio exceeds $\text{bbox}(m_i)$ 2. This step discards over 80% of mask pairs. A hard depth constraint is then applied: $\text{bbox}(m_i)$ 3 Pairs that fail this constraint are never merged, even when their semantic similarity is high. The paper identifies this as critical for visually similar but depth-separated regions, including the “white-on-white” case.

If a pair passes adjacency and hard depth filtering, the algorithm evaluates the composite score and merges when

$\text{bbox}(m_i)$ 4

The procedure is greedy and iterative, performed per image view, with a cap of 1000 iterations. After each accepted merge, the new region is formed by union, its semantic descriptor is recomputed as the mean DINO feature over the union, and its depth and boundary statistics are updated. Because scores are recomputed after every merge, conflicts among multiple potential merges are resolved implicitly by the greedy update schedule.

At convergence, each view contains a refined set of masks $\text{bbox}(m_i)$ 5 that is larger and less fragmented than the raw SAM output. A common misconception is that such consolidation is equivalent to purely semantic agglomeration. The method explicitly rejects that interpretation: semantics provides only one term in a composite criterion, and both depth discontinuities and structural edges can veto semantically attractive merges.

4. Cross-view matching, 3D lifting, and feature-field supervision

After per-view merging, the refined masks are still viewpoint-specific. Cross-view consistency is imposed in the next stage by comparing refined descriptors across views (Park et al., 2 Jul 2026). For each merged mask $\text{bbox}(m_i)$ 6, the representative descriptor $\text{bbox}(m_i)$ 7 is the mean DINO feature over the merged region, and two masks are assigned the same global ID if

$\text{bbox}(m_i)$ 8

These global IDs are then projected onto Gaussians. For a Gaussian $\text{bbox}(m_i)$ 9 visible in views $\partial m_i$ 0, the fused semantic feature is

$\partial m_i$ 1

where $\partial m_i$ 2 is the DINO descriptor of the merged mask containing the projection of $\partial m_i$ 3, and $\partial m_i$ 4 are view weights. The Gaussian ID is assigned by majority voting: $\partial m_i$ 5 The practical effect is that less fragmented and more geometrically coherent masks produce more stable majority votes and less ID flickering.

Feature-field optimization attaches a learnable semantic feature $\partial m_i$ 6 to each Gaussian and supervises it with the lifted feature $\partial m_i$ 7. The semantic cohesion loss is

$\partial m_i$ 8

where $\partial m_i$ 9 are Gaussians with low multi-view feature variance. A 3D geometric regularization term enforces neighborhood smoothness: $f_i\in\mathbb{R}^{768}$ 0 The total optimization objective is

$f_i\in\mathbb{R}^{768}$ 1

with $f_i\in\mathbb{R}^{768}$ 2 and $f_i\in\mathbb{R}^{768}$ 3.

The significance of the merging stage is therefore downstream rather than local. Cleaner per-view masks improve cross-view clustering, which improves 3D Gaussian supervision, which in turn improves 3D instance segmentation quality, cross-view identity consistency, and downstream editing in which one ID should correspond to one object rather than many fragments.

5. Empirical behavior, design rationale, and ablation evidence

The design rationale is organized around three failure modes of single-cue merging: over-segmentation from texture variation, erroneous merging of visually similar but depth-separated objects, and lack of boundary awareness (Park et al., 2 Jul 2026). Semantics consolidates fragments, depth separates layers and supplies a hard rejection criterion, and structural cues localize boundaries.

The ablation on the LERF Figurine dataset makes the complementarity explicit.

Configuration	mIoU	#Masks
Baseline (GaussianGrouping, no multi-cue refinement)	0.685	9,627
Ours (1) – DINO only	0.721	245
Ours (2) – DINO + LoG (semantics + edges)	0.729	112
Ours (Full) – DINO + LoG + DepthAnythingV2	0.741	67

Semantic similarity alone drastically reduces fragments but still leaves hundreds of masks and some incorrect merges. Adding LoG edges improves boundaries and further reduces fragmentation. Adding depth yields the best segmentation metrics and the smallest number of masks, while also slightly improving rendering metrics such as PSNR and SSIM. Additional hyperparameter ablation shows that balancing semantic and geometric cues gives the best trade-off; semantic-biased or structure-biased settings degrade mIoU or induce over- or under-merging.

Several implementation details are fixed across datasets. DINOv2 ViT-B/14 is used with $f_i\in\mathbb{R}^{768}$ 4 inputs and 768-d features, DepthAnythingV2 produces relative depth normalized to $f_i\in\mathbb{R}^{768}$ 5, and the LoG edge detector uses $f_i\in\mathbb{R}^{768}$ 6. The same merge-score weights, $f_i\in\mathbb{R}^{768}$ 7, $f_i\in\mathbb{R}^{768}$ 8, and the 1000-iteration cap are used for indoor scenes such as Replica and LERF as well as real-world scenes, which suggests cross-domain robustness under a fixed parameterization.

The empirical message is not that more cues are always beneficial in the abstract, but that the particular hard-plus-soft factorization matters. Depth is both a hard rejection rule and a soft affinity term; edges and depth gradients act as penalties rather than affinities. This asymmetry is a defining property of the method.

The method has several stated limitations (Park et al., 2 Jul 2026). It assumes that SAM provides an over-segmented decomposition, so severe under-segmentation, missing small objects, or incorrect large masks cannot be corrected by merging alone. Merging is performed per view before global clustering, so extreme occlusion or drastic lighting changes can yield view-specific merge discrepancies that must later be corrected by cross-view matching. Aggressive thresholds, such as lower $f_i\in\mathbb{R}^{768}$ 9 or larger $\bar d_i$ 0, can cause distinct adjacent objects to be merged. Noisy depth can both block correct merges through false hard-constraint failures and introduce erroneous depth-boundary penalties.

These limitations also delimit common misconceptions. Multi-Cue-Guided Mask Merging is not a general replacement for mask generation, because it cannot recover objects omitted by the initial segmentation prior. It is not a purely geometric routine, because DINO semantics is the dominant positive term in the merge score. It is not a purely semantic agglomerator, because hard depth filtering and structural penalties explicitly override semantics. Its scope is consolidation, not universal correction.

Within 3D scene understanding, related zero-shot methods implement analogous multi-cue mask-merging logics. OnlineAnySeg merges VFM-generated 2D masks into 3D instances with spatial overlap in 3D, semantic similarity from CLIP, geometric similarity from FCGF, third-view support, and mask weights, while reducing overlap-query complexity from $\bar d_i$ 1 to $\bar d_i$ 2 through voxel hashing (Tang et al., 3 Mar 2025). MV3DIS uses coarse 3D segments as a common reference, builds 3D coverage distributions for 2D masks, and adds a depth consistency weighting scheme to suppress occlusion ambiguity during multi-view mask matching (Zhao et al., 10 Apr 2026). In both cases, the operative principle is again that mask association improves when multiple cues are fused rather than when each frame is processed independently with 2D-only scores.

The expression also appears in broader senses outside 3DGS. In training-free diffusion, Mask-guided Prompt Following uses the union of object masks to gate ControlNet features and combines text, visual control, object masks, and attention maps through mask-based gating and attention-based guidance (Chen et al., 2024). In model merging, CALM uses consensus-aware localized binary masks guided by global task consensus, class-balanced entropy-minimized pseudo-data, and sparsity regularization to merge task vectors conflict-free (Yan et al., 16 Jun 2025). In controllable infrared-visible image fusion, CtrlFuse merges infrared and visible features, SAM-based segmentation priors, and mask prompts through a Reference Prompt Encoder and Prompt-Semantic Fusion Module (Sun et al., 12 Jan 2026). In semantic-guided multi-mask image harmonization, operator masks for LAB-channel scale and shift functions provide a decomposed, editable alternative to direct RGB regression (Ren et al., 2022). These formulations are heterogeneous, but they share an identifiable pattern: masks are not treated as isolated binary supports, but as decision variables or gating structures whose consolidation depends on several jointly evaluated signals.

In that broader sense, Multi-Cue-Guided Mask Merging names both a specific per-view algorithm for refining SAM masks in 3D Gaussian Splatting and a more general design principle for consolidating mask hypotheses under heterogeneous evidence. The 3DGS formulation of (Park et al., 2 Jul 2026) is its clearest canonical instance: a greedy iterative merger whose composite score integrates semantics, depth, and structure before cross-view identity assignment and 3D feature lifting.