SplatSSC: Monocular 3D Semantic Scene Completion

• SplatSSC is a monocular 3D semantic scene completion framework that predicts dense 3D voxel grids of occupancy and semantic labels from a single RGB image.
• It employs depth-guided Gaussian primitive initialization and a Group-wise Multi-scale Fusion module to efficiently refine features for accurate geometry estimation.
• The decoupled Gaussian Aggregator separates occupancy from semantic prediction, effectively eliminating outlier-induced artifacts and achieving state-of-the-art performance on Occ-ScanNet.

Searching arXiv for the SplatSSC paper and closely related Gaussian-splatting work to support the article. Searching arXiv for "SplatSSC Decoupled Depth-Guided Gaussian Splatting for Semantic Scene Completion". Searching arXiv for related terms: "SplatSSC", "3D Student Splatting and Scooping", "Depth-Anything-V2", "MonoScene", "GaussianFormer", "EmbodiedOcc", "RoboOcc". SplatSSC is a monocular 3D Semantic Scene Completion (SSC) framework that predicts a dense 3D voxel grid of occupancy and semantic labels from a single RGB image by combining depth-guided Gaussian primitive initialization with a decoupled Gaussian-to-voxel aggregation mechanism. It is formulated to address two failure modes identified in Gaussian-splatting SSC pipelines: inefficient primitive initialization caused by randomly scattering thousands of Gaussian primitives through the volume, and outlier-induced “floaters” caused by aggregation rules that allow low-confidence primitives to spuriously activate local voxels. The method introduces a dedicated depth branch with a Group-wise Multi-scale Fusion (GMF) module, a Decoupled Gaussian Aggregator (DGA), and a Probability Scale Loss, and reports state-of-the-art performance on Occ-ScanNet with improvements of +6.35 percentage points in IoU and +4.16 percentage points in mIoU over the prior best result (Qian et al., 4 Aug 2025).

1. Problem formulation and design rationale

Monocular 3D SSC in SplatSSC is defined as the task of predicting a dense 3D voxel grid $V$ of occupancy and $C$ semantic labels, together with a sparse set of continuous 3D primitives $G$, from a single RGB image $I_{\mathrm{rgb}}$ (Qian et al., 4 Aug 2025). The framework is positioned against object-centric Gaussian-splatting approaches such as GaussianFormer and its variants, which improve efficiency through flexible 3D Gaussian primitives but still rely heavily on a large number of randomly initialized primitives.

Two deficiencies motivate the architecture. The first is inefficient primitive initialization: without depth cues, prior methods randomly scatter thousands of Gaussian primitives throughout the 3D volume, so many lie in empty space, wasting capacity and slowing convergence and inference. The second is outlier-induced “floaters”: in aggregation schemes such as probabilistic Gaussian superposition, an isolated outlier Gaussian with small learned opacity $a_i$ can still dominate the local posterior and spuriously activate voxels near its mean, producing floating semantic artifacts. SplatSSC addresses these with depth-guided initialization of a compact yet representative Gaussian set and with a decoupled aggregator that separates geometry from semantics.

This problem framing places geometry estimation and semantic estimation under distinct but coupled constraints. Geometry determines whether a voxel is occupied at all, whereas semantics should only be assigned conditionally on occupancy. A plausible implication is that the method’s central contribution is not merely improved Gaussian parameterization, but a restructuring of the inference pipeline so that semantic evidence cannot bypass geometric uncertainty.

2. Depth-guided Gaussian initialization

The front end consists of an image branch and a depth branch. The image branch uses EfficientNet-B7 + FPN and outputs multi-scale features $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$. The depth branch uses a frozen Depth-Anything-V2 network to extract latent depth features $F_d$ and a coarse depth map $I_d^0$ (Qian et al., 4 Aug 2025).

The depth branch is refined by the Group-wise Multi-scale Fusion (GMF) module. SplatSSC samples $K$ reference points in the image using a $30\times 40$ regular grid, so $C$0. It extracts sampled depth features $C$1 and multi-scale image features $C$2. Channels are split into $C$3 groups of size $C$4. For each group $C$5 and scale $C$6, the module computes

$C$7

followed by linear attention scores

$C$8

where $C$9 is shared across all groups and scales. The fused representation is

$G$0

A two-layer convolutional head then produces a refined depth map $G$1. The stated complexity is $G$2 rather than $G$3.

Gaussian initialization is then performed by lifting each sampled 2D point $G$4 with depth $G$5 and fused feature $G$6 into an initial Gaussian primitive $G$7. The primitive mean is

$G$8

while scale $G$9, rotation $I_{\mathrm{rgb}}$0, opacity $I_{\mathrm{rgb}}$1, and semantic logit $I_{\mathrm{rgb}}$2 are predicted by a small MLP that takes $I_{\mathrm{rgb}}$3 as input. The resulting set has $I_{\mathrm{rgb}}$4 primitives, each with full anisotropic covariance

$I_{\mathrm{rgb}}$5

The intended effect is a sparse but representative primitive set. In the reported formulation, this depth-feature guided initialization “dramatically reduces redundancy versus random scattering,” and the ablations indicate that the initialization strategy is one of the two dominant sources of performance gain.

3. Decoupled Gaussian Aggregator

After several refinement stages on the Gaussian set $I_{\mathrm{rgb}}$6, SplatSSC renders semantic voxels through the Decoupled Gaussian Aggregator (DGA), which factors geometry and semantics explicitly (Qian et al., 4 Aug 2025). This is the core mechanism for suppressing floaters.

For geometry, let

$I_{\mathrm{rgb}}$7

be the unnormalized Gaussian kernel, and let $I_{\mathrm{rgb}}$8 be the learned existence confidence. For each 3D point $I_{\mathrm{rgb}}$9, only primitives in its neighborhood $a_i$0 are considered. The occupancy probability is defined by a noisy-OR rule:

$a_i$1

This formulation gates each Gaussian by its opacity and suppresses low-confidence outliers.

For semantics, the model conditions on the event that $a_i$2 is occupied. With

$a_i$3

and with primitive semantic logits $a_i$4 normalized by softmax to obtain $a_i$5, the conditional class distribution is

$a_i$6

Final voxel probabilities are then

$a_i$7

The decoupling is operationally important. Because $a_i$8 for outlier Gaussians, any spurious $a_i$9 is gated out; the formulation is explicitly described as eliminating floaters. A common misconception would be that semantic robustness follows simply from better semantic logits. The DGA results suggest instead that robust semantics depend on geometric gating: semantics are only meaningful once occupancy has been established.

4. Probability Scale Loss and optimization objective

To stabilize geometry learning across the multi-stage encoder, SplatSSC extends the geometric scale loss from MonoScene into a Probability Scale Loss that supervises $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$0 at each encoder layer (Qian et al., 4 Aug 2025). If $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$1 denotes the Huber-based scale loss comparing $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$2 at layer $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$3 to ground-truth occupancy, then

$\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$4

This imposes weak supervision at early layers and strong supervision at the final layer.

The total SSC loss is

$\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$5

The reported weights are $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$6, $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$7, $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$8, $\{f_{\mathrm{rgb}}^l\}^{l=1\ldots L}$9, and $F_d$0, with $F_d$1–$F_d$2 used for depth-branch losses.

The training procedure is two-stage. In stage 1, the depth branch is pretrained on Occ-ScanNet for 10 epochs with a cosine learning-rate schedule, peak learning rate $F_d$3, 1k warmup, and batch size 4 on 2 GPUs. In stage 2, the full model is trained end-to-end with an initial Gaussian grid of $F_d$4 primitives, scale range $F_d$5, total batch size 8, peak learning rate $F_d$6, cosine schedule with 1k warmup, for 10 epochs on Occ-ScanNet and 20 epochs on Occ-ScanNet-mini, on NVIDIA RTX 4090. Optimization uses AdamW with weight decay $F_d$7 and a $F_d$8 learning-rate multiplier on pretrained backbones. Input resolution is $F_d$9 with standard color augmentations.

One empirical result bears directly on model interpretation: including explicit depth loss consistently degrades SSC performance. This indicates that the depth branch is most useful as an initialization and feature-refinement mechanism rather than as an auxiliary objective tightly coupled to SSC optimization.

5. Empirical performance and ablation structure

On Occ-ScanNet, SplatSSC reports IoU = 62.83% and mIoU = 51.83%, while the prior best result, RoboOcc, reports 56.48% IoU and 47.67% mIoU, yielding +6.35 percentage points IoU and +4.16 percentage points mIoU (Qian et al., 4 Aug 2025). The method also reports –9.32% inference latency and –9.64% GPU memory versus EmbodiedOcc, with only +0.19 M parameters.

Ablations on Gaussian parameterization indicate that 1200 primitives with scale range [0.01, 0.16] yield 61.47% IoU, 48.87% mIoU, 3.112 MiB, and 115.6 ms. Denser or larger-scale primitive sets bring marginal gains in IoU but hurt mIoU, speed, or memory. This suggests that the primitive budget is not the principal source of the final gain.

The component ablations isolate the contributions of depth-guided initialization and DGA:

Variant	IoU	mIoU
No depth guidance + GF.agg	11.64%	12.62%
No depth + GF2.agg	27.54%	17.27%
No depth + DGA	48.85%	36.91%
With GMF + GF2.agg	57.70%	45.13%
With GMF + DGA	60.61%	48.01%

These numbers are the basis for the paper’s conclusion that both depth-guided initialization and DGA are crucial. The loss-function study further reports that Focal+Lovasz+ProbScale+no depth loss yields the best geometry, with 60.61% IoU, and strong semantics, with 48.01% mIoU. The depth-branch study reports the following sequence: Baseline DAv2 with $I_d^0$0, RMSE $I_d^0$1, C-$I_d^0$2; +GMF on DAv2 with $I_d^0$3, RMSE $I_d^0$4, C-$I_d^0$5; FT-DAv2 with $I_d^0$6, RMSE $I_d^0$7, C-$I_d^0$8; and +GMF on FT-DAv2 with $I_d^0$9, RMSE $K$0, C-$K$1. The stated interpretation is that GMF is a powerful multi-modal feature refiner.

Hyperparameter sensitivity is also explicit: SplatSSC requires a total batch size $K$2 for stable training, with mIoU jumping from 36.1% at batch size 2 to 45.3% at batch size 4, whereas EmbodiedOcc shows only linear gains. This qualifies the reported accuracy improvements with a concrete optimization constraint.

6. Position within splatting research and stated limitations

SplatSSC belongs to a broader wave of work that uses splatting primitives as a computational substrate, but its target is semantic scene completion, not novel view synthesis. Within the same period, “3D Student Splatting and Scooping” proposes replacing 3D Gaussian primitives with an unnormalized mixture of 3D Student’s $K$3 distributions and allowing both positive (“splatting”) and negative (“scooping”) densities, motivated by the claim that, as an unnormalized mixture model, the representation needs to be neither Gaussians nor splatting (Zhu et al., 13 Mar 2025). SplatSSC does not pursue that direction: it retains Gaussian primitives and concentrates on depth-guided initialization, voxel-space aggregation, and semantic conditioning. This contrast clarifies that the “splatting” label now spans multiple problem classes and multiple primitive families.

The stated limitations of SplatSSC are twofold. First, the end-to-end primitive optimization is hyperparameter sensitive and requires a moderate batch size for stable convergence. Second, the current system has a local-view constraint: it processes each frame independently, and simply accumulating primitives across frames would cause unbounded growth. The paper therefore identifies online primitive management—specifically pruning and fusion—as a requirement for global mapping. Future directions are described as extending to large-scale and unbounded scenes through dynamic spatial data structures such as hash grids and octrees, and to Embodied AI settings with persistent interactive world models, continuous observation fusion, real-time refinement for manipulation tasks, and finer semantic granularity such as handles and hinges (Qian et al., 4 Aug 2025).

Taken together, the reported results support a narrow but important conclusion. SplatSSC’s gains are attributed not to indiscriminately increasing primitive count, but to enforcing a more structured pipeline: depth-guided Gaussian placement, geometry-first occupancy inference, and semantics conditioned on occupancy. Within monocular SSC, this makes Gaussian splatting a representation for volumetric reasoning rather than merely a rendering primitive.