SDI-GS: Segmentation-Driven Gaussian Splatting

• The paper introduces SDI-GS, a method that leverages region segmentation for initializing Gaussians from cross-view consistent clusters, reducing point counts by up to 75%.
• It utilizes MDBSCAN for fast, unsupervised 2D segmentation and stratified sampling to achieve compact representations with minimal losses in SSIM and PSNR.
• Joint optimization of Gaussians and camera poses via photometric loss results in improved memory efficiency and faster training and rendering speeds.

Segmentation-Driven Initialization for Gaussian Splatting (SDI-GS) is a sparse-view 3D Gaussian Splatting (3DGS) method designed for SfM-free novel-view synthesis under limited observations. It replaces uniform pixel-wise lifting with a region-based initialization procedure that identifies “structurally significant” regions, groups 3D points by cross-view region consistency, and then performs stratified downsampling within each group before Gaussian optimization. In the reported formulation, MASt3R provides poses and dense depth, MDBSCAN supplies unsupervised 2D region segmentation in RGB space, and the resulting compact Gaussian set is optimized jointly with camera poses under a photometric loss. The stated objective is to preserve scene fidelity while substantially reducing Gaussian count, memory footprint, and training cost in genuinely sparse-view settings (Li et al., 15 Sep 2025).

1. Problem setting and motivation

Sparse-view synthesis remains difficult because accurate geometry and appearance must be recovered from limited observations. The SDI-GS formulation identifies two limitations in existing 3DGS pipelines. First, SfM-based initialization depends on Structure-from-Motion, such as COLMAP, for both poses and point clouds; feature matching is reported to fail once views drop below approximately $6\!-\!8$, producing noisy poses and collapsed geometry. Second, SfM-free pipelines that use monocular depth or learned stereo often lift every pixel into 3D, generating uniformly dense point clouds with tens or hundreds of millions of 3D points, which in turn produces high memory use and slow training and rendering (Li et al., 15 Sep 2025).

Within this framing, SDI-GS is not a semantic segmentation method in the open-vocabulary sense. Its segmentation step is instead a region-based mechanism for controlling initialization density. The key insight is to identify “structurally significant” regions via 2D region segmentation, group 3D points by cross-view region consistency, and then stratified-downsample uniformly within each group. The reported result is a compact, geometry-aware Gaussian set that retains detail while cutting $30\!-\!75\%$ of points (Li et al., 15 Sep 2025).

The paper also argues that some purported sparse-view evaluations are not genuinely sparse because SfM is first run on a dense video and only later restricted to a few views. In that argument, SDI-GS is positioned as a method intended for a stricter SfM-free protocol, rather than a post hoc reduction of a dense-view reconstruction (Li et al., 15 Sep 2025).

2. Region segmentation and structural clustering

The region segmentation stage uses MDBSCAN (Modified DBSCAN) in RGB space. The summary characterizes MDBSCAN as an unsupervised clustering method based on local color similarity, with no fixed superpixel size constraint: large flat areas form large clusters, whereas textured areas form fine clusters. Its reported runtime is approximately $4$ ms per view, compared with approximately $150$ ms for deep networks such as SAM, making the segmentation overhead negligible in the claimed pipeline (Li et al., 15 Sep 2025).

For each input image $I_i$, MDBSCAN assigns a per-pixel region label

$\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$

These 2D labels are then lifted into a cross-view structural descriptor. For a point $\mathbf{x}\in X_i$, where $X_i$ is the dense 3D point set obtained from MASt3R for view $i$, the method projects $\mathbf{x}$ into the two adjacent views $30\!-\!75\%$0 using projection operators $30\!-\!75\%$1, and defines

$30\!-\!75\%$2

Points sharing the same label vector $30\!-\!75\%$3 form a structural cluster

$30\!-\!75\%$4

where

$30\!-\!75\%$5

This construction operationalizes the paper’s notion of structural significance: segments that remain coherent when projected into adjacent views indicate stable geometry and retain more samples; flat or low-frequency segments, such as sky or walls, yield few large clusters and therefore fewer retained points; high-frequency regions generate many small clusters, each of which retains up to $30\!-\!75\%$6 samples (Li et al., 15 Sep 2025).

A recurring misconception is to read the segmentation stage as an object-level parser. In SDI-GS, the segmentation signal is region-based and driven by local color similarity rather than language supervision or category labels. Its role is to organize dense points into cross-view consistent clusters for sampling, not to assign semantic identities (Li et al., 15 Sep 2025).

3. Downsampling, Gaussian initialization, and optimization

After structural clusters are built, SDI-GS performs stratified sampling within each cluster. Given a cluster $30\!-\!75\%$7 of size $30\!-\!75\%$8, the sampled subset is

$30\!-\!75\%$9

The final downsampled point cloud is the union of these sampled subsets,

$4$0

The hyperparameter $4$1 controls the maximum number of retained points per segment. Smaller values increase compression and may lose detail, while larger values reduce compression. The reported typical range is $4$2, and the paper states that a 3-view label vector is optimal relative to higher-dimensional alternatives using $4$3 views, because additional views fragment segments and increase file size (Li et al., 15 Sep 2025).

The initialization pipeline then creates one Gaussian per sampled point. In the provided pseudocode, each Gaussian is initialized as

$4$4

Here, $4$5 is the Gaussian mean, $4$6 the initial covariance, $4$7 the color sampled at the point, and $4$8 the opacity. The process therefore uses segmentation only at initialization time; it does not introduce a separate segmentation loss or semantic field (Li et al., 15 Sep 2025).

Optimization proceeds jointly over Gaussians and camera poses. At each iteration, the current Gaussian set $4$9 is rendered under the current poses $150$0 to produce reconstructed images $150$1, and the photometric objective is

$150$2

The pseudocode specifies the following sequence: MASt3R estimates $150$3 and $150$4; MDBSCAN produces $150$5; cross-view cluster keys are built from source and adjacent-view labels; cluster-wise sampling forms $150$6; Gaussians are initialized from $150$7; and joint optimization updates $150$8 together with $150$9 (Li et al., 15 Sep 2025).

4. Reported empirical performance

The reported experiments span Tanks & Temples, Mip-NeRF 360, and MVImgNet under an SfM-free protocol. The central empirical claim is that SDI-GS achieves major compression and speed gains while maintaining comparable rendering quality in PSNR and SSIM, with only marginal degradation in LPIPS (Li et al., 15 Sep 2025).

Benchmark and setting	Baseline	SDI-GS
Tanks & Temples, 3-view	SSIM 0.768, LPIPS 0.175, 98.6 MB	SSIM 0.754, LPIPS 0.245, 21.8 MB
Mip-NeRF 360, 3-view	SSIM 0.317, LPIPS 0.534, 118.9 MB	SSIM 0.336, LPIPS 0.569, 20.5 MB
MVImgNet, 3-view	SSIM 0.554, LPIPS 0.386, 102.7 MB	SSIM 0.550, LPIPS 0.438, 25.5 MB

These numbers illustrate the paper’s stated trade-off profile. On Tanks & Temples, file size decreases from $I_i$0 MB to $I_i$1 MB, while SSIM changes from $I_i$2 to $I_i$3 and LPIPS from $I_i$4 to $I_i$5. On Mip-NeRF 360, SSIM improves from $I_i$6 to $I_i$7 while file size decreases from $I_i$8 MB to $I_i$9 MB, with LPIPS changing from $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$0 to $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$1. On MVImgNet, SSIM is nearly unchanged, $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$2 versus $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$3, while file size decreases from $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$4 MB to $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$5 MB and LPIPS changes from $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$6 to $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$7 (Li et al., 15 Sep 2025).

The paper summarizes these comparisons by stating that the SSIM drop is at most $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$8 absolute, LPIPS degradation is at most $\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.$9, and PSNR remains comparable within $\mathbf{x}\in X_i$0 dB. It further reports that as the number of views increases, redundancy also increases, enabling up to $\mathbf{x}\in X_i$1 size reduction at $\mathbf{x}\in X_i$2 views with a flat PSNR curve (Li et al., 15 Sep 2025).

5. Efficiency profile, hyperparameters, and limitations

The compression effect is reflected in both storage and runtime. For 3-view input, SDI-GS is reported to produce approximately $\mathbf{x}\in X_i$3 fewer points than InstantSplat; on Tanks & Temples, file size decreases from $\mathbf{x}\in X_i$4 MB to $\mathbf{x}\in X_i$5 MB, described as approximately $\mathbf{x}\in X_i$6 savings. For 12-view input, the file size decreases from $\mathbf{x}\in X_i$7 MB to $\mathbf{x}\in X_i$8 MB, described as approximately $\mathbf{x}\in X_i$9 savings. GPU memory is reported to be roughly proportional to Gaussian count, with the example that $X_i$0 fewer Gaussians corresponds to at most $X_i$1 of peak memory (Li et al., 15 Sep 2025).

Training and rendering also become faster. On Tanks & Temples in the 3-view setting, training time decreases from $X_i$2 s to $X_i$3 s; in the 12-view setting, it decreases from $X_i$4 s to $X_i$5 s. Rendering speed on an A100 is reported to improve from $X_i$6 FPS to $X_i$7 FPS for 3-view input and from $X_i$8 FPS to $X_i$9 FPS for 12-view input (Li et al., 15 Sep 2025).

The principal control variable is $i$0, the maximum number of samples per structural cluster. The paper describes this as a direct compression-versus-detail trade-off: smaller $i$1 values produce more aggressive reduction and possible detail loss; values in the range $i$2 are reported as robust. Another ablation concerns the dimensionality of the cluster label vector. Using labels from three views is stated to be optimal, whereas higher-dimensional label vectors based on $i$3 views fragment segments and enlarge the final model (Li et al., 15 Sep 2025).

The stated failure modes are correspondingly geometric. Unobserved or sparsely overlapped regions may remain as holes because the method does not use pseudo-view densification or a generative prior. Very fine details may be lost when per-cluster sampling is too aggressive. The paper’s listed extensions include learned adaptive segmentation to capture geometry cues beyond color, hierarchical sampling for global shape and silhouettes, progressive densification after initial SDI-GS, and semantic saliency maps to prioritize foreground objects (Li et al., 15 Sep 2025).

6. Position within segmentation-conditioned Gaussian splatting

SDI-GS belongs to a broader 2025 line of work in which segmentation affects Gaussian representations before or during optimization rather than being applied only after reconstruction. A closely related but task-distinct example is “Segment then Splat,” which addresses 3D open-vocabulary segmentation in static and dynamic scenes. That method divides Gaussians into distinct object sets before reconstruction using multi-view 2D masks from SAM and SAM 2, optimizes the scene under both photometric and per-object losses, and assigns a single CLIP embedding to each object for querying after optimization. Its reported gains include $i$4 mIoU in $i$5 minutes on 3DOVS versus OpenGaussian’s $i$6 mIoU in $i$7 minutes, and $i$8 mIoU in $i$9 minutes on HyperNeRF versus DGD’s $\mathbf{x}$0 mIoU in $\mathbf{x}$1 minutes (Lu et al., 28 Mar 2025).

A second adjacent example is “Contour Information Aware 2D Gaussian Splatting for Image Representation,” which operates in 2D rather than sparse-view 3D reconstruction. In that framework, each Gaussian is assigned to exactly one segmentation region by sampling the mask at its center, rasterization is restricted by the integer comparison $\mathbf{x}$2, and a warm-up scheme refreshes region assignments every $\mathbf{x}$3 iterations during the first half of training $\mathbf{x}$4. Reported gains include higher reconstruction quality around object edges on synthetic color charts and DAVIS, for example improving DAVIS full-image PSNR from $\mathbf{x}$5 to $\mathbf{x}$6 and edge-PSNR from $\mathbf{x}$7 to $\mathbf{x}$8 at $\mathbf{x}$9 Gaussians (Takabe et al., 29 Dec 2025).

These comparisons clarify what is specific to SDI-GS. Unlike “Segment then Splat,” it does not aim at object-level 3D segmentation or CLIP-based retrieval. Unlike contour-aware 2DGS, it does not enforce region constraints during rasterization. Its segmentation signal is used to compactify initialization for sparse-view 3D reconstruction by retaining structurally significant regions and downsampling dense point clouds in a geometry-aware manner. This suggests a broader design pattern in Gaussian-based representations: segmentation priors can be most consequential when they intervene at the level of primitive allocation, sampling density, or rasterization support, rather than only as a downstream labeling stage.