---
title: 'SDI-GS: Segmentation-Driven Gaussian Splatting'
url: https://www.emergentmind.com/topics/segmentation-driven-initialization-for-gaussian-splatting-sdi-gs
type: topic
---

# SDI-GS: Segmentation-Driven Gaussian Splatting

Segmentation-Driven Initialization for Gaussian Splatting (SDI-GS) is a sparse-view 3D Gaussian Splatting (3DGS) method designed for SfM-free novel-view synthesis under limited observations. It replaces uniform pixel-wise lifting with a region-based initialization procedure that identifies “structurally significant” regions, groups 3D points by cross-view region consistency, and then performs stratified downsampling within each group before Gaussian optimization. In the reported formulation, MASt3R provides poses and dense depth, MDBSCAN supplies unsupervised 2D region segmentation in RGB space, and the resulting compact Gaussian set is optimized jointly with camera poses under a photometric loss. The stated objective is to preserve scene fidelity while substantially reducing Gaussian count, memory footprint, and training cost in genuinely sparse-view settings [2509.11853].

## 1. Problem setting and motivation

Sparse-view synthesis remains difficult because accurate geometry and appearance must be recovered from limited observations. The SDI-GS formulation identifies two limitations in existing 3DGS pipelines. First, SfM-based initialization depends on Structure-from-Motion, such as COLMAP, for both poses and point clouds; feature matching is reported to fail once views drop below approximately \(6\!-\!8\), producing noisy poses and collapsed geometry. Second, SfM-free pipelines that use monocular depth or learned stereo often lift every pixel into 3D, generating uniformly dense point clouds with tens or hundreds of millions of 3D points, which in turn produces high memory use and slow training and rendering [2509.11853].

Within this framing, SDI-GS is not a semantic segmentation method in the open-vocabulary sense. Its segmentation step is instead a region-based mechanism for controlling initialization density. The key insight is to identify “structurally significant” regions via 2D region segmentation, group 3D points by cross-view region consistency, and then stratified-downsample uniformly within each group. The reported result is a compact, geometry-aware Gaussian set that retains detail while cutting \(30\!-\!75\%\) of points [2509.11853].

The paper also argues that some purported sparse-view evaluations are not genuinely sparse because SfM is first run on a dense video and only later restricted to a few views. In that argument, SDI-GS is positioned as a method intended for a stricter SfM-free protocol, rather than a post hoc reduction of a dense-view reconstruction [2509.11853].

## 2. Region segmentation and structural clustering

The region segmentation stage uses MDBSCAN (Modified DBSCAN) in RGB space. The summary characterizes MDBSCAN as an unsupervised clustering method based on local color similarity, with no fixed superpixel size constraint: large flat areas form large clusters, whereas textured areas form fine clusters. Its reported runtime is approximately \(4\) ms per view, compared with approximately \(150\) ms for deep networks such as SAM, making the segmentation overhead negligible in the claimed pipeline [2509.11853].

For each input image \(I_i\), MDBSCAN assigns a per-pixel region label
\[
\ell_i^{2D}(u,v)\in\{1,\dots,M_i\}.
\]
These 2D labels are then lifted into a cross-view structural descriptor. For a point \(\mathbf{x}\in X_i\), where \(X_i\) is the dense 3D point set obtained from MASt3R for view \(i\), the method projects \(\mathbf{x}\) into the two adjacent views \(\{i-1,i+1\}\) using projection operators \(p_k:\mathbb{R}^3\to\mathbb{R}^2\), and defines
\[
\ell^{3D}(\mathbf{x})
=\bigl[\ell^{2D}_{i}(u,v),\;
\ell^{2D}_{i-1}(p_{i-1}(\mathbf{x})),\;
\ell^{2D}_{i+1}(p_{i+1}(\mathbf{x}))\bigr].
\]
Points sharing the same label vector \(\ell_j^{3D}\) form a structural cluster
\[
C_j=\{\mathbf{x}\in\mathcal{P}\mid \ell^{3D}(\mathbf{x})=\ell_j^{3D}\},
\]
where
\[
\mathcal{P}=\bigcup_{i=1}^N X_i,\qquad X_i=\{\mathbf{x}_i(u,v)\mid (u,v)\in I_i\}.
\]
This construction operationalizes the paper’s notion of structural significance: segments that remain coherent when projected into adjacent views indicate stable geometry and retain more samples; flat or low-frequency segments, such as sky or walls, yield few large clusters and therefore fewer retained points; high-frequency regions generate many small clusters, each of which retains up to \(N_{\max}\) samples [2509.11853].

A recurring misconception is to read the segmentation stage as an object-level parser. In SDI-GS, the segmentation signal is region-based and driven by local color similarity rather than language supervision or category labels. Its role is to organize dense points into cross-view consistent clusters for sampling, not to assign semantic identities [2509.11853].

## 3. Downsampling, Gaussian initialization, and optimization

After structural clusters are built, SDI-GS performs stratified sampling within each cluster. Given a cluster \(C_j\) of size \(|C_j|\), the sampled subset is
\[
\hat C_j=\mathrm{RandomSample}\bigl(C_j,\;\min(N_{\max},\,|C_j|)\bigr).
\]
The final downsampled point cloud is the union of these sampled subsets,
\[
\hat{\mathcal{P}}=\bigcup_j \hat C_j.
\]
The hyperparameter \(N_{\max}\) controls the maximum number of retained points per segment. Smaller values increase compression and may lose detail, while larger values reduce compression. The reported typical range is \(30\!-\!100\), and the paper states that a 3-view label vector is optimal relative to higher-dimensional alternatives using \(6\!-\!12\) views, because additional views fragment segments and increase file size [2509.11853].

The initialization pipeline then creates one Gaussian per sampled point. In the provided pseudocode, each Gaussian is initialized as
\[
G_i=(\mu=x,\Sigma=\Sigma_0,c=\mathrm{color}(x),\alpha=\alpha_0).
\]
Here, \(\mu\) is the Gaussian mean, \(\Sigma\) the initial covariance, \(c\) the color sampled at the point, and \(\alpha\) the opacity. The process therefore uses segmentation only at initialization time; it does not introduce a separate segmentation loss or semantic field [2509.11853].

Optimization proceeds jointly over Gaussians and camera poses. At each iteration, the current Gaussian set \(G\) is rendered under the current poses \((R_i,t_i)\) to produce reconstructed images \(\hat I_i\), and the photometric objective is
\[
L_{\mathrm{photo}}=\sum_i \|I_i-\hat I_i\|_2^2.
\]
The pseudocode specifies the following sequence: MASt3R estimates \((R_i,t_i)\) and \(X_i\); MDBSCAN produces \(L_i^{2D}\); cross-view cluster keys are built from source and adjacent-view labels; cluster-wise sampling forms \(\hat{\mathcal{P}}\); Gaussians are initialized from \(\hat{\mathcal{P}}\); and joint optimization updates \(G\) together with \((R_i,t_i)\) [2509.11853].

## 4. Reported empirical performance

The reported experiments span Tanks & Temples, Mip-NeRF 360, and MVImgNet under an SfM-free protocol. The central empirical claim is that SDI-GS achieves major compression and speed gains while maintaining comparable rendering quality in PSNR and SSIM, with only marginal degradation in LPIPS [2509.11853].

| Benchmark and setting | Baseline | SDI-GS |
|---|---|---|
| Tanks & Temples, 3-view | SSIM 0.768, LPIPS 0.175, 98.6 MB | SSIM 0.754, LPIPS 0.245, 21.8 MB |
| Mip-NeRF 360, 3-view | SSIM 0.317, LPIPS 0.534, 118.9 MB | SSIM 0.336, LPIPS 0.569, 20.5 MB |
| MVImgNet, 3-view | SSIM 0.554, LPIPS 0.386, 102.7 MB | SSIM 0.550, LPIPS 0.438, 25.5 MB |

These numbers illustrate the paper’s stated trade-off profile. On Tanks & Temples, file size decreases from \(98.6\) MB to \(21.8\) MB, while SSIM changes from \(0.768\) to \(0.754\) and LPIPS from \(0.175\) to \(0.245\). On Mip-NeRF 360, SSIM improves from \(0.317\) to \(0.336\) while file size decreases from \(118.9\) MB to \(20.5\) MB, with LPIPS changing from \(0.534\) to \(0.569\). On MVImgNet, SSIM is nearly unchanged, \(0.554\) versus \(0.550\), while file size decreases from \(102.7\) MB to \(25.5\) MB and LPIPS changes from \(0.386\) to \(0.438\) [2509.11853].

The paper summarizes these comparisons by stating that the SSIM drop is at most \(2\%\) absolute, LPIPS degradation is at most \(0.1\), and PSNR remains comparable within \(\pm 0.2\) dB. It further reports that as the number of views increases, redundancy also increases, enabling up to \(75\%\) size reduction at \(12\) views with a flat PSNR curve [2509.11853].

## 5. Efficiency profile, hyperparameters, and limitations

The compression effect is reflected in both storage and runtime. For 3-view input, SDI-GS is reported to produce approximately \(50\%\) fewer points than InstantSplat; on Tanks & Temples, file size decreases from \(98.6\) MB to \(21.8\) MB, described as approximately \(78\%\) savings. For 12-view input, the file size decreases from \(278\) MB to \(85\) MB, described as approximately \(69\%\) savings. GPU memory is reported to be roughly proportional to Gaussian count, with the example that \(80\%\) fewer Gaussians corresponds to at most \(20\%\) of peak memory [2509.11853].

Training and rendering also become faster. On Tanks & Temples in the 3-view setting, training time decreases from \(7.12\) s to \(6.50\) s; in the 12-view setting, it decreases from \(12.88\) s to \(10.00\) s. Rendering speed on an A100 is reported to improve from \(152\) FPS to \(190\) FPS for 3-view input and from \(105\) FPS to \(148\) FPS for 12-view input [2509.11853].

The principal control variable is \(N_{\max}\), the maximum number of samples per structural cluster. The paper describes this as a direct compression-versus-detail trade-off: smaller \(N_{\max}\) values produce more aggressive reduction and possible detail loss; values in the range \(30\!-\!100\) are reported as robust. Another ablation concerns the dimensionality of the cluster label vector. Using labels from three views is stated to be optimal, whereas higher-dimensional label vectors based on \(6\!-\!12\) views fragment segments and enlarge the final model [2509.11853].

The stated failure modes are correspondingly geometric. Unobserved or sparsely overlapped regions may remain as holes because the method does not use pseudo-view densification or a generative prior. Very fine details may be lost when per-cluster sampling is too aggressive. The paper’s listed extensions include learned adaptive segmentation to capture geometry cues beyond color, hierarchical sampling for global shape and silhouettes, progressive densification after initial SDI-GS, and semantic saliency maps to prioritize foreground objects [2509.11853].

## 6. Position within segmentation-conditioned Gaussian splatting

SDI-GS belongs to a broader 2025 line of work in which segmentation affects Gaussian representations before or during optimization rather than being applied only after reconstruction. A closely related but task-distinct example is “Segment then Splat,” which addresses 3D open-vocabulary segmentation in static and dynamic scenes. That method divides Gaussians into distinct object sets before reconstruction using multi-view 2D masks from SAM and SAM 2, optimizes the scene under both photometric and per-object losses, and assigns a single CLIP embedding to each object for querying after optimization. Its reported gains include \(88.53\) mIoU in \(9.4\) minutes on 3DOVS versus OpenGaussian’s \(31.00\) mIoU in \(59.4\) minutes, and \(69.48\) mIoU in \(218\) minutes on HyperNeRF versus DGD’s \(7.83\) mIoU in \(1564.5\) minutes [2503.22204].

A second adjacent example is “Contour Information Aware 2D Gaussian Splatting for Image Representation,” which operates in 2D rather than sparse-view 3D reconstruction. In that framework, each Gaussian is assigned to exactly one segmentation region by sampling the mask at its center, rasterization is restricted by the integer comparison \(r_i=r_{xy}\), and a warm-up scheme refreshes region assignments every \(K=1000\) iterations during the first half of training \((\tau=0.5)\). Reported gains include higher reconstruction quality around object edges on synthetic color charts and DAVIS, for example improving DAVIS full-image PSNR from \(27.1\) to \(28.2\) and edge-PSNR from \(21.0\) to \(22.5\) at \(1250\) Gaussians [2512.23255].

These comparisons clarify what is specific to SDI-GS. Unlike “Segment then Splat,” it does not aim at object-level 3D segmentation or CLIP-based retrieval. Unlike contour-aware 2DGS, it does not enforce region constraints during rasterization. Its segmentation signal is used to compactify initialization for sparse-view 3D reconstruction by retaining structurally significant regions and downsampling dense point clouds in a geometry-aware manner. This suggests a broader design pattern in Gaussian-based representations: segmentation priors can be most consequential when they intervene at the level of primitive allocation, sampling density, or rasterization support, rather than only as a downstream labeling stage.

Source: https://www.emergentmind.com/topics/segmentation-driven-initialization-for-gaussian-splatting-sdi-gs