Seg-Wild: Interactive 3D Segmentation

• Seg-Wild is an interactive 3D segmentation method that employs 3D Gaussian Splatting to jointly reconstruct and segment in-the-wild photo collections.
• It integrates multi-dimensional affinity features, a Scale-Adaptive Segmentation Masking module, and a Spiky 3D Gaussian Cutter to enhance segmentation consistency across views.
• Evaluations on landmark and smartphone datasets demonstrate improved IoU and pixel accuracy over traditional 3D segmentation pipelines.

Searching arXiv for the primary Seg-Wild paper and a small set of directly related 3DGS segmentation/reconstruction works to ground citations. Seg-Wild is an interactive segmentation method for unconstrained image collections that is built on 3D Gaussian Splatting and targets joint 3D reconstruction and segmentation in in-the-wild scenes. It is designed for Internet-scale or casually captured photo sets with different cameras, highly varying viewpoints and sparsity, inconsistent lighting and appearance, and transient occluders such as people, cars, and scaffolding. Within this setting, Seg-Wild combines a GS-W reconstruction backbone, multi-dimensional feature embeddings for each 3D Gaussian, a Scale-Adaptive Segmentation Masking module, and a Spiky 3D Gaussian Cutter to obtain 3D segments that can be rendered consistently across views and from novel viewpoints (Bao et al., 10 Jul 2025).

1. Problem setting and scope

Seg-Wild addresses joint 3D reconstruction and interactive segmentation from unconstrained photo collections. The input is a set of images

$I = \{ I_1, I_2, \ldots, I_N \},$

with different cameras and intrinsic parameters, highly varying viewpoints and sparsity, inconsistent lighting and appearance, transient occluders, and post-processing artifacts such as HDR and JPG compression (Bao et al., 10 Jul 2025).

The method is positioned against a failure mode common to prior segmentation pipelines on reconstructed 3D scenes. Classical multi-view segmentation typically assumes controlled capture, while NeRF- and 3DGS-based segmentation methods such as SAGA, Feature 3DGS, and SA3D are described as assuming clean, well-captured multi-view sequences with consistent lighting and no transient occlusions. In such in-the-wild settings, photometric inconsistencies induce noisy optimization and ghost geometry, Gaussians may mix static structure with transient clutter, and semantic consistency across views deteriorates (Bao et al., 10 Jul 2025).

A second part of the motivation concerns the insufficiency of purely 2D segmentation. SAM and SAM2 can segment individual images, but the resulting masks are view-specific, and mapping them into 3D is non-trivial when geometry is inconsistent. The data also state that fixed-scale SAM masks are inconsistent across images with different camera distances. Seg-Wild therefore targets a regime in which reconstruction robustness, feature lifting into 3D, interactive selection, and post-processing for unconstrained reconstructions must be handled within a single pipeline (Bao et al., 10 Jul 2025).

2. Representation and reconstruction backbone

Seg-Wild uses a 3D Gaussian Splatting scene representation

$$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$

where each Gaussian has a center $p_i \in \mathbb{R}^3$, covariance $\Sigma_i \in \mathbb{R}^{3 \times 3}$, opacity $\alpha_i$, and color represented by zero-order and higher-order spherical harmonics coefficients. The covariance is parameterized as

$\Sigma = R S S^T R^T,$

and the Gaussian density is

$$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$

Under projection with view transform $W$ and Jacobian $J$, the covariance becomes

$\Sigma' = J W \Sigma W^T J^T,$

and the projected center is

$$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$0

Rendering uses ordered alpha compositing in screen space: $$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$1 These definitions are central because Seg-Wild inherits the editability and explicitness of 3DGS while attaching segmentation-oriented features to each Gaussian (Bao et al., 10 Jul 2025).

The reconstruction backbone is GS-W. In the supplied description, GS-W adds a UNet that extracts feature maps from reference images to model dynamic appearance, together with a per-image transient occlusion mask $$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$2 taking values in $$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$3. The mask is regularized by

$$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$4

and the color loss is

$$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$5

with $$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$6 weights. This backbone is described as explicitly accounting for transient occluders and modeling appearance variations through learned appearance features, thereby improving robustness to inconsistent lighting before segmentation-specific objectives are introduced (Bao et al., 10 Jul 2025).

3. Feature field and training objective

Seg-Wild extends each Gaussian to

$$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$7

where $$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$8 denotes a static or intrinsic appearance feature, $$\mathcal{G} = \{ G_i(p_i, \Sigma_i, c_i, \alpha_i)\}_{i=1}^N,$$9 a dynamic appearance feature, and $p_i \in \mathbb{R}^3$0 an affinity feature used for segmentation. The affinity feature is the key embedding used for semantic similarity during interactive segmentation (Bao et al., 10 Jul 2025).

For each image, the SAM image encoder provides a feature map

$p_i \in \mathbb{R}^3$1

To reduce memory and computation, the features are compressed by PCA from 256 dimensions to

$p_i \in \mathbb{R}^3$2

yielding $p_i \in \mathbb{R}^3$3. A $p_i \in \mathbb{R}^3$4 convolution later decompresses them back to 256 dimensions for alignment, while the stored per-Gaussian affinity features remain 64-dimensional (Bao et al., 10 Jul 2025).

Feature rendering parallels color rendering: $p_i \in \mathbb{R}^3$5 A per-pixel feature alignment loss ties the 3D feature field to SAM’s feature space: $p_i \in \mathbb{R}^3$6

The method further introduces a compactness loss guided by SAM masks. For each pixel $p_i \in \mathbb{R}^3$7, SAM yields a mask $p_i \in \mathbb{R}^3$8, from which an IoU-based similarity is computed: $p_i \in \mathbb{R}^3$9 with $\Sigma_i \in \mathbb{R}^{3 \times 3}$0. Cosine similarity between rendered features is defined as

$\Sigma_i \in \mathbb{R}^{3 \times 3}$1

and the compactness objective is

$\Sigma_i \in \mathbb{R}^{3 \times 3}$2

The full training objective is

$\Sigma_i \in \mathbb{R}^{3 \times 3}$3

with $\Sigma_i \in \mathbb{R}^{3 \times 3}$4 and $\Sigma_i \in \mathbb{R}^{3 \times 3}$5 (Bao et al., 10 Jul 2025).

The intended effect of this design is explicit in the supplied material: $\Sigma_i \in \mathbb{R}^{3 \times 3}$6 is learned so that projected Gaussian features match SAM features for a given view, while Gaussians corresponding to the same SAM region share similar affinity features. This produces what the paper describes as a compact 3D feature field, which is then used as the substrate for interactive segmentation (Bao et al., 10 Jul 2025).

4. Interactive segmentation and segmentation-specific modules

Interactive segmentation is driven by prompt points on a reference image. If the prompt set is

$\Sigma_i \in \mathbb{R}^{3 \times 3}$7

the rendered feature map $\Sigma_i \in \mathbb{R}^{3 \times 3}$8 is sampled at these prompt locations. For each prompt and each Gaussian, Seg-Wild computes cosine similarity

$\Sigma_i \in \mathbb{R}^{3 \times 3}$9

where $\alpha_i$0 is the stacked matrix of all Gaussian affinity features (Bao et al., 10 Jul 2025).

Multiple prompt similarities are fused by a softmax-weighted combination,

$\alpha_i$1

and SAM is also queried with the same prompt points to produce a prompt-conditioned 2D mask

$\alpha_i$2

Each Gaussian center $\alpha_i$3 is projected to the image by

$\alpha_i$4

and a Gaussian is selected if its fused similarity exceeds a threshold $\alpha_i$5 and its projected center lies inside the 2D mask: $\alpha_i$6 The default threshold is $\alpha_i$7 (Bao et al., 10 Jul 2025).

A distinct component, the Scale-Adaptive Segmentation Masking module, adapts SAM’s global segmentation to each view. The module uses average scene depth for the camera, depth distribution of projected 3D Gaussians, and a sky mask from DeepLabv3+ to avoid sky dominating depth statistics. In the implementation details, the segmentation scale is computed as

$\alpha_i$8

The same description lists filtered depth maps $\alpha_i$9, grid-wise average depths $\Sigma = R S S^T R^T,$0, and per-grid numbers of prompt points $\Sigma = R S S^T R^T,$1 as part of the SASM procedure (Bao et al., 10 Jul 2025). This suggests that Seg-Wild treats 2D mask generation not as a fixed preprocessing step, but as a view-conditioned regularizer synchronized with reconstruction geometry.

5. Spiky 3D Gaussian Cutter and boundary refinement

The Spiky 3D Gaussian Cutter is introduced to address a characteristic artifact of in-the-wild reconstructions: slender Gaussians whose long axes extend beyond the true object boundary. The data emphasize that simply deleting such Gaussians can remove useful geometry, since much of the Gaussian mass may still lie inside the target region (Bao et al., 10 Jul 2025).

For a spiky Gaussian $\Sigma = R S S^T R^T,$2 with covariance $\Sigma = R S S^T R^T,$3, projection to 2D yields

$\Sigma = R S S^T R^T,$4

Let $\Sigma = R S S^T R^T,$5 and $\Sigma = R S S^T R^T,$6 be the principal eigenpair of $\Sigma = R S S^T R^T,$7, and let $\Sigma = R S S^T R^T,$8 be the projected center. The two endpoints of the long axis are defined as

$\Sigma = R S S^T R^T,$9

Intermediate pixels are sampled along this axis, and the fraction of sampled points lying inside the 2D segmentation mask is

$$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$0

This ratio is interpreted in the supplied description as the portion of the Gaussian’s long axis covered by the 2D segmentation (Bao et al., 10 Jul 2025).

The Gaussian is then shifted and shrunk: $$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$1 The updated 2D center is back-projected to 3D, and the scale parameters in $$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$2 are updated accordingly. The stated effect is to trim the protruding part while retaining the portion inside the object, thereby smoothing the segmentation boundary without removing valuable coverage (Bao et al., 10 Jul 2025).

The ablation summary in the supplied details attributes a marked contribution to SGC: removing it reduces IoU by approximately $$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$3 and Acc by approximately $$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$4 on average, and the visual comparison is described as showing visibly spikier boundaries without it (Bao et al., 10 Jul 2025).

6. Evaluation, ablations, and limitations

Seg-Wild is evaluated on two datasets. The first is Photo Tourism, comprising Internet photos of landmarks such as Trevi Fountain, Taj Mahal, and Brandenburg Gate, with 800–1500 unconstrained images per scene. The second is NeRF-On-the-go, comprising handheld smartphone videos with 200–300 frames per sequence (Bao et al., 10 Jul 2025).

The paper also introduces a benchmark for in-the-wild segmentation based on the Photo Tourism dataset. The benchmark includes Brandenburg Gate, Taj Mahal, and Trevi Fountain, with multiple segmentation targets per scene and manually refined 2D ground-truth masks generated using ISAT together with SAM. The evaluation metrics are Intersection over Union and Pixel Accuracy, and segmented 3D Gaussians are rendered back into reference images for comparison with the ground-truth masks (Bao et al., 10 Jul 2025).

The quantitative comparison includes Feature 3DGS, SAGA, and GS-W + Projection as baselines. The supplied details state that, in every metric and target, Seg-Wild outperforms the baselines. Representative values given in the data include IoU ranges of $$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$5–$$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$6 on Brandenburg Gate, $$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$7–$$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$8 on Trevi Fountain, and $$G(x) = \exp\left( -\frac{1}{2} (x - p)^T \Sigma^{-1} (x - p) \right).$$9–$W$0 on Taj Mahal, with Pixel Accuracy reaching approximately $W$1–$W$2 on Brandenburg Gate and up to $W$3 on Taj Mahal (Bao et al., 10 Jul 2025).

The training configuration reported in the data uses PyTorch with Adam, $W$4 iterations on a single Nvidia RTX 4060 GPU, and on-the-fly feature extraction with SAM mask generation taking approximately $W$5 hours. Gaussian parameters, appearance features, affinity features, the UNet, and PCA-related modules are all optimized jointly (Bao et al., 10 Jul 2025).

Ablation studies identify the main contributors. Removing $W$6 reduces IoU from $W$7 to $W$8 on Brandenburg Gate, from $W$9 to $J$0 on Taj Mahal, and from $J$1 to $J$2 on Trevi Fountain. Removing SASM causes a further drop, removing the sky mask causes a slight drop, and removing SGC causes an approximately $J$3–$J$4 IoU reduction. Regarding affinity dimensionality, 32D is worse, while 64D and 128D are similar; 64D is therefore used as the default tradeoff. Threshold values $J$5, $J$6, and $J$7 show moderate variation, with $J$8 reported to work well (Bao et al., 10 Jul 2025).

The stated limitations are also tied closely to the segmentation prior. Segmentation quality depends partly on SAM’s 2D masks, and the supplied failure example involves the “left hippocampus” statue at Trevi Fountain, where SAM mis-segments the winged horse and the resulting 3D segmentation becomes incomplete. The details further indicate that with very sparse views, feature lifting and compactness constraints may be insufficient, that large scenes increase memory and training time, and that SGC may still leave artifacts under extreme geometry distortions or occlusion patterns (Bao et al., 10 Jul 2025).

7. Relation to surrounding research

Within the supplied related-work framing, Seg-Wild is situated among 3DGS and NeRF segmentation methods such as ISRF, SA3D, Feature 3DGS, SAGA, Click-Gaussian, and Gaussian Grouping, and among in-the-wild reconstruction systems such as NeRF-W, K-planes, GS-W, WE-GS, Wild-GS, WildGaussians, and SLS (Bao et al., 10 Jul 2025). The description distinguishes Seg-Wild from these lines of work by emphasizing its specialization to unconstrained photo collections and its integration of segmentation-specific losses and modules with a reconstruction backbone already designed for transient occlusions and variable appearance (Bao et al., 10 Jul 2025).

Three components define that positioning. The first is the 3D affinity feature field optimized against SAM embeddings with compactness regularization. The second is the Scale-Adaptive Segmentation Masking module, which adapts global SAM segmentation to camera depth and projected Gaussian statistics. The third is the Spiky 3D Gaussian Cutter, which repairs boundary artifacts associated with unconstrained 3DGS reconstructions (Bao et al., 10 Jul 2025).

A plausible implication is that the method’s contribution is as much architectural as it is algorithmic: rather than replacing 3DGS or SAM, it inserts a geometry-aware feature-learning and refinement layer between them. In this reading, Seg-Wild can be understood as a system for making 2D foundation-model priors operational in unstable 3D reconstructions, rather than a purely new segmentation objective.

The paper’s own summary is consistent with that interpretation. It presents Seg-Wild as a full pipeline for interactive 3D segmentation on 3D Gaussian splats reconstructed from unconstrained photo collections, combining GS-W’s appearance and transient modeling with a 3D affinity feature field, SASM, and SGC, and showing improved segmentation results and reconstruction quality on in-the-wild data (Bao et al., 10 Jul 2025).