GaussianSSC: Triplane-Guided Directional Gaussian Fields for 3D Semantic Completion

Abstract: We present \emph{GaussianSSC}, a two-stage, grid-native and triplane-guided approach to semantic scene completion (SSC) that injects the benefits of Gaussians without replacing the voxel grid or maintaining a separate Gaussian set. We introduce \emph{Gaussian Anchoring}, a sub-pixel, Gaussian-weighted image aggregation over fused FPN features that tightens voxel--image alignment and improves monocular occupancy estimation. We further convert point-like voxel features into a learned per-voxel Gaussian field and refine triplane features via a triplane-aligned \emph{Gaussian--Triplane Refinement} module that combines \emph{local gathering} (target-centric) and \emph{global aggregation} (source-centric). This directional, anisotropic support captures surface tangency, scale, and occlusion-aware asymmetry while preserving the efficiency of triplane representations. On SemanticKITTI~\cite{behley2019semantickitti}, GaussianSSC improves Stage~1 occupancy by +1.0\% Recall, +2.0\% Precision, and +1.8\% IoU over state-of-the-art baselines, and improves Stage~2 semantic prediction by +1.8\% IoU and +0.8\% mIoU.

• The paper introduces GaussianSSC, a method that combines Gaussian representations with a triplane-factorized pipeline, achieving significant improvements in semantic completion accuracy, reaching 61.2% IoU, and 53.2 IoU for occupancy, outperforming state-of-the-art methods and achieving 0.8% and 1.8% increases in mIoU.
• GaussianSSC refines voxel embeddings and aggregates image features with Gaussian kernels, proving more adaptable for outdoor scene structures and capable of capturing directional and occlusive information with the aid of a dual-stage process.
• Compared to existing methods, GaussianSSC is characteristically computational lighter, demonstrating fewer costs in time and memory, while significantly surpassing consummate benchmarks on crucial datasets like SemanticKITTI.

GaussianSSC addresses monocular semantic scene completion (SSC) by integrating Gaussian representations into a grid-native, triplane-factorized pipeline. Rather than maintaining a separate set of 3D Gaussians as in GaussianFormer-style approaches (Huang et al., 2024), the method learns a per-voxel Gaussian field that refines triplane features, preserving the efficiency and deployability of plane-based SSC while importing the adaptive, anisotropic spatial support characteristic of Gaussian primitives.

Motivation and design rationale

Two limitations of existing monocular SSC methods motivate the design. First, lifting image features into 3D from a single view is underconstrained: errors propagate along camera rays, and point-sampled projection of voxel centers onto image features is sensitive to sub-pixel misalignment, discretization, and monocular depth ambiguity. Second, uniform voxel or BEV lattices allocate computation uniformly even to empty space, whereas real outdoor scenes are strongly anisotropic—planar roads, façades, and elongated structures demand directional, scale-adaptive support. Full 3D Gaussian pipelines offer such adaptivity but are computationally expensive and awkward to integrate with grid-based decoders. GaussianSSC's compromise is to keep the voxel grid and triplane factorization as the backbone and use Gaussians only as learned aggregation kernels within it.

Method

The framework follows a two-stage factorization: Stage 1 estimates a binary occupancy prior $p_{\theta_1}(o \mid I)$; Stage 2 predicts semantics conditioned on both the image and this prior, $p_{\theta_2}(y \mid I, o)$.

Stage 1: query-conditioned triplane with Gaussian Anchoring. Multi-scale ResNet-50 + FPN features seed a sparse set of voxel queries from a coarse monocular depth prior (MobileStereoNet). Queries are rasterized via scatter-add onto three orthogonal planes (HW, HD, WD) with learned axis-wise positional embeddings, refined with lightweight 2D convolution/attention blocks, and gathered back into per-voxel descriptors. The key component, Gaussian Anchoring, treats each voxel as a latent 3D Gaussian whose image footprint is modeled directly on a fused FPN map as a learnable 2D Gaussian $(\boldsymbol{\mu}_v, \boldsymbol{\Sigma}_v)$ with offset $\boldsymbol{\delta}_v$ and opacity $\alpha_v$. Instead of sampling a single pixel at the projected location, the anchor feature aggregates a fixed $5\times5$ neighborhood under normalized Gaussian weights, providing sub-pixel, content-adaptive alignment. A gated residual fusion injects this evidence into the voxel descriptor, followed by a lightweight dilated 3D head for occupancy. Notably, the authors bypass explicit covariance prediction and Jacobian-based projection of a 3D Gaussian, learning the image-plane footprint instead—an approximation they acknowledge implicitly by describing it as modeling the projected footprint "up to the local projection Jacobian and depth scaling."

Stage 2: Gaussian–Triplane Refinement. Voxel embeddings are gated by the Stage-1 occupancy mask and conditioned on multi-scale image features via deformable cross-attention. After triplane construction and per-plane self/cross-attention refinement, each voxel decodes three per-plane 2D Gaussian parameter sets plus a shared opacity. Refinement combines two complementary operations per plane:

• Local gathering (target-centric): each target location aggregates neighboring features weighted by its own Gaussian window, improving localization and reducing quantization artifacts.
• Global aggregation (source-centric): each source distributes its feature according to its own Gaussian and opacity, so locations covered by multiple Gaussians receive opacity-normalized mixtures.

The two are blended with a scalar weight $\beta$, and the refined planes are lifted back to voxels for semantic decoding. This directional support is intended to capture surface tangency, scale, and occlusion-aware asymmetry while retaining triplane efficiency.

Training. Stage 1 uses class-balanced cross-entropy over positives plus sampled negatives, with weak $\ell_1/\ell_2$ regularizers on Gaussian offsets and scales to stabilize the learned kernels. Stage 2 uses class-weighted cross-entropy plus a structure-aware penalty that pushes per-class precision, recall, and specificity toward one via BCE terms.

Results

On SemanticKITTI (256×256×32 grid at 0.2 m over a 51.2×51.2×6.4 m ROI), evaluated with the standard protocol excluding unknown voxels:

Method	Recall	Precision	IoU
VoxFormer	61.5	71.1	52.0
ETFormer	71.5	77.8	59.4
GaussianSSC	72.5	79.8	61.2

Stage-1 occupancy improves over ETFormer by +1.0 Recall, +2.0 Precision, and +1.8 IoU. For full semantic completion, GaussianSSC reaches 53.2 IoU / 17.1 mIoU, exceeding the prior camera-only best (ETFormer, 51.4/16.3) by +1.8/+0.8, with top per-class results on building, car, truck, vegetation, terrain, and traffic-sign. Efficiency measured in FP32 on an RTX A6000 shows moderate overhead relative to ETFormer (409.8 ms vs. 356.6 ms latency; 2.8 GB vs. 2.3 GB peak memory), while remaining substantially cheaper than Symphonies (612.0 ms, 4.3 GB) and more memory-efficient than VoxFormer (3.8 GB).

Ablations attribute gains to each component. In Stage 1, Gaussian Anchoring alone contributes +1.4 IoU over the triplane baseline, with negative sampling adding +0.4. In Stage 2, global-only ($\beta{=}0$) and local-only ($\beta{=}1$) refinement each yield +0.8 IoU, while the mixture ($p_{\theta_2}(y \mid I, o)$0) yields +1.2, indicating the two aggregation modes are complementary; combining Stage-1 anchoring with Stage-2 refinement accounts for the full +1.8 IoU gain. Qualitatively, the method produces more complete reconstructions in occluded regions and fewer spurious predictions in empty space, though the authors note a trade-off: aggressive completion can produce denser outputs than ground truth in heavily occluded areas, which is consistent with the SSC objective but reflects a stronger structural prior rather than guaranteed accuracy.

Limitations and open questions

The paper concedes several constraints. The formulation operates on single-frame RGB input without temporal modeling, which limits applicability to dynamic scenes. Monocular ambiguity persists in severely occluded regions where multiple completions are plausible, and the observed tendency toward denser completions suggests the occupancy prior may bias toward over-completion—the paper does not quantify this failure mode. Additionally, the image-plane Gaussian parameterization avoids explicit 3D covariance projection, so the claimed occlusion-aware asymmetry is only indirectly realized; whether learning footprints directly in image space versus projecting 3D covariances matters for accuracy is left unexamined. Evaluation is confined to SemanticKITTI; generalization to other datasets or sensor configurations is not reported.

Conclusion

GaussianSSC demonstrates that Gaussian-conditioned aggregation can be embedded within a triplane-based monocular SSC pipeline without abandoning the voxel grid or incurring the cost of a separate Gaussian representation. The reported gains—+1.8 IoU in occupancy and +1.8/+0.8 IoU/mIoU in semantics over the strongest camera-only baseline—are achieved at modest latency and memory cost, supported by ablations showing complementary contributions from local gathering and global aggregation. The main open questions concern temporal extension, quantification of over-completion behavior, and validation beyond SemanticKITTI.