VisHall3D: Monocular Semantic Scene Completion

• VisHall3D is a monocular semantic scene completion framework that separates visible voxel reconstruction from occluded region hallucination.
• The approach employs VisFrontierNet to build a precise visible frontier and OcclusionMAE to denoise and predict unseen geometries for coherent 3D occupancy.
• Empirical results on SemanticKITTI and SSCBench-KITTI-360 show improved IoU and mIoU, demonstrating enhanced geometric consistency and semantic quality.

Searching arXiv for the primary VisHall3D paper and closely related monocular semantic scene completion works. arXiv search query: "VisHall3D monocular semantic scene completion" VisHall3D is a monocular semantic scene completion (SSC) framework that explicitly separates “what we can really see” from “what we must hallucinate,” decomposing the task into reconstructing visible regions and inferring invisible regions from a single RGB image in an autonomous driving setting (Lu et al., 25 Jul 2025). It is designed to address two failure modes identified in prior monocular SSC systems—feature entanglement and geometric inconsistency—by coupling a visibility-aware visible-region module, VisFrontierNet, with a hallucination module, OcclusionMAE. Within the standard SemanticKITTI and SSCBench-KITTI-360 protocol, the method predicts occupancy and semantics on a voxelized volume of spatial extent $$51.2 \text{ m} \times 51.2 \text{ m} \times 6.4 \text{ m}$$, voxel size $0.2 \text{ m}$, and resolution $256 \times 256 \times 32$.

1. Problem formulation and motivation

Monocular semantic scene completion takes as input a single RGB image from a calibrated camera and predicts a 3D voxel grid whose occupied cells carry semantic labels such as road, building, car, and vegetation. Supervision is derived from aggregated LiDAR scans and distinguishes visible voxels, occluded voxels, and out-of-view (OOV) voxels. Visible voxels are those whose surface points project into the current image; occluded voxels lie behind visible surfaces from the current view; OOV voxels are outside the camera field of view.

The task is ill-posed from a single image because multiple 3D scenes can explain the same image, large parts of the volume are hidden by occlusion, and only a sparse 2D projection is directly observed. VisHall3D frames this difficulty as a mismatch between two subproblems that earlier methods often force into a single representation: reconstructing visible surfaces that are strongly constrained by image evidence, and generating invisible structure that is not directly observed. The paper identifies this as feature entanglement, in which the same 3D features are used to explain both visible and invisible voxels, and geometric inconsistency, in which visible surfaces and hallucinated volumes can become structurally incompatible (Lu et al., 25 Jul 2025).

A central implication of this formulation is that visibility is not treated as a secondary mask but as the organizing principle of the architecture. The visible set is defined by projecting each voxel $v \in G$ to image coordinates and depth,

$(x_v, y_v, d_v) = \mathcal{T}(v),$

and then comparing $d_v$ with the estimated depth map $D(x_v, y_v)$: $$V := \{ v \mid v \in G,\ d_v < D(x_v, y_v) + \theta_a \}.$$ Voxels that project inside the image but fail this criterion are treated as occluded; those that project outside the image are OOV.

2. Two-stage decomposition and system-level design

VisHall3D decomposes monocular SSC into two stages. Stage 1, VisFrontierNet, reconstructs visible voxels, specifically the “visual frontier,” defined as the set of voxels corresponding to the first surface hit along each camera ray. Stage 2, OcclusionMAE, hallucinates plausible geometry and semantics for occluded and OOV voxels by treating the first-stage output as a noisy partial scene and denoising it with volumetric context (Lu et al., 25 Jul 2025).

The system-level data flow is explicit. A 2D encoder composed of ResNet-50 and a MaskDINO neck extracts image features $F_{2D}$. A pretrained MobileStereoNet predicts both a dense depth map $D$ and a depth probability map $0.2 \text{ m}$0. VisFrontierNet then projects voxels to the image plane, identifies visibility from $0.2 \text{ m}$1, computes a truncated unsigned distance volume $0.2 \text{ m}$2, fuses 3D geometric encoding with RoI-pooled 2D features, and refines the result using Lift-Splat with $0.2 \text{ m}$3 and deformable 3D attention to obtain $0.2 \text{ m}$4. OcclusionMAE receives the visible-region prediction $0.2 \text{ m}$5, injects structured spatial noise controlled by a level $0.2 \text{ m}$6, modulates the latent features through adaptive batch normalization conditioned on $0.2 \text{ m}$7, concatenates them with $0.2 \text{ m}$8, and predicts full-volume occupancy and semantics through a 3D U-Net.

The conceptual claim of the architecture is not merely modularity. The paper argues that fixing a coherent visible frontier before hallucinating the invisible region reduces feature entanglement and encourages geometric consistency, because the hallucination stage operates relative to a visible structure that has already been reconstructed under direct image and depth constraints. This suggests a reconstruction-first, completion-second factorization of SSC rather than a monolithic image-to-volume mapping.

3. VisFrontierNet and visible-region reconstruction

VisFrontierNet is the visibility-aware first stage. Its goal is to reconstruct occupancy and semantics only for voxels that should be visible given the camera pose and estimated depth. The module combines explicit projection geometry, a volumetric unsigned distance representation, 2D appearance injection, and feature refinement.

Its most distinctive geometric primitive is the truncated unsigned distance to the visual frontier. For each voxel,

$0.2 \text{ m}$9

and the distance encoding is defined as

$256 \times 256 \times 32$0

Here $256 \times 256 \times 32$1 is a sharpness factor set to 10, and $256 \times 256 \times 32$2 is a truncation radius set to 1 m. Rather than lifting each pixel to a single voxel by hard assignment, this representation creates a “thick” volumetric band around the estimated visual frontier. The stated purpose is to preserve fine-grained geometry, mitigate sparsity, and retain useful signals even for voxels slightly off the surface.

The distance volume is processed by a 3D convolutional encoder $256 \times 256 \times 32$3, producing geometric features that are fused with image evidence. For each voxel, the method computes a projected image-space bounding box $256 \times 256 \times 32$4 and extracts per-voxel image features by RoI pooling on $256 \times 256 \times 32$5. The fusion is

$256 \times 256 \times 32$6

This sum combines geometry from the unsigned distance field with appearance and semantics from the 2D encoder.

A second refinement step performs a soft lifting using the depth probability map $256 \times 256 \times 32$7. The method applies Lift-Splat to distribute 2D features along depth bins into a coarse frustum-aligned grid and then applies 3D deformable attention,

$256 \times 256 \times 32$8

The deformable attention module learns offsets and attention weights to sample from surrounding voxels adaptively. The resulting $256 \times 256 \times 32$9 is the paper’s high-quality 3D representation of the visible scene, and a masked classification head predicts occupancy and semantics for the visible set $v \in G$0 (Lu et al., 25 Jul 2025).

The ablation results attribute a large part of the geometric gain to this visible-frontier representation. Starting from a baseline with ResNet-50, a MaskDINO neck, direct hard assignment lifting, and a single 3D U-Net head, adding the unsigned distance function raises IoU from 42.11 to 46.44 and mIoU from 14.56 to 15.11. The paper interprets this as evidence that visibility-aware geometry is a strong prior.

4. OcclusionMAE and invisible-region hallucination

OcclusionMAE is the second stage, designed for occluded and OOV voxels. It treats the visible prediction from VisFrontierNet as a noisy partial observation and frames completion as denoising plus full-scene reconstruction. This is described as a denoising masked autoencoder-style network operating at scene level rather than pixel level (Lu et al., 25 Jul 2025).

Let $v \in G$1 denote the visible-region prediction from stage 1. OcclusionMAE perturbs it by structured spatial noise,

$v \in G$2

where $v \in G$3 is a noise level and noise is applied only in the horizontal and depth directions, not vertically. The perturbation is geometric rather than semantic: semantics are not changed, and no class swaps are introduced. The stated intuition is that the network should recover a consistent full scene from corrupted visible predictions, improving robustness and discouraging overfitting to exact first-stage outputs.

Noise awareness is injected through adaptive batch normalization. Latent 3D features extracted from $v \in G$4 are normalized and then modulated by learnable scale $v \in G$5 and shift $v \in G$6 conditioned on the noise level $v \in G$7. The noise-modulated features are concatenated with the visible-context representation $v \in G$8, and the fused tensor is processed by a volumetric 3D U-Net with a contracting path, an expanding path, skip connections, and a final $v \in G$9 convolution to produce per-voxel logits over semantic classes plus empty.

Conceptually, the module differs from a conventional masked autoencoder in two ways. It corrupts known visible regions with structured noise rather than masking random patches, and it reconstructs full 3D occupancy with semantics rather than pixels. In the ablation sequence, adding OcclusionMAE without denoising changes IoU from 46.44 to 46.38 while improving mIoU from 15.11 to 15.99, which the paper interprets as evidence that separating hallucination from visible reconstruction benefits semantic quality even before explicit denoising. Adding denoising with noise injection and AdaBN yields a final ablation result of IoU 46.14 and mIoU 17.06, with the gain concentrated in mIoU rather than occupancy IoU.

The visibility split is essential to this second stage. An ablation that treats only OOV voxels as invisible, without explicit occlusion reasoning, obtains IoU 43.98 and mIoU 15.89, substantially below the occlusion-aware variants. With the occlusion-aware threshold $(x_v, y_v, d_v) = \mathcal{T}(v),$0, the reported results are IoU 45.87 and mIoU 16.65 at $(x_v, y_v, d_v) = \mathcal{T}(v),$1, IoU 45.83 and mIoU 16.92 at $(x_v, y_v, d_v) = \mathcal{T}(v),$2, IoU 46.14 and mIoU 17.06 at $(x_v, y_v, d_v) = \mathcal{T}(v),$3, and IoU 45.94 and mIoU 16.95 at $(x_v, y_v, d_v) = \mathcal{T}(v),$4. A common misconception is therefore that invisible-region completion is primarily a field-of-view problem; the paper’s evidence indicates that explicit occlusion modeling is crucial.

5. Optimization, benchmarks, and empirical performance

Training is organized as a two-stage flow within a single overall framework. VisFrontierNet is supervised on visible voxels, while OcclusionMAE predicts the full volume and receives loss over the entire grid. The losses include class-frequency weighted cross-entropy, a scene-class affinity loss with separate geometric and semantic constraints inspired by MonoScene, and a depth consistency loss ensuring coincidence between the probabilistic depth distribution $(x_v, y_v, d_v) = \mathcal{T}(v),$5 and the depth map $(x_v, y_v, d_v) = \mathcal{T}(v),$6 (Lu et al., 25 Jul 2025).

The experiments use two standard autonomous-driving benchmarks. SemanticKITTI provides 10 training sequences with 3,834 samples, 1 validation sequence with 815 samples, and 11 test sequences with 3,992 samples, with 20 valid semantic classes plus 1 invalid. SSCBench-KITTI-360 provides 7 training sequences with 8,487 samples, 1 validation sequence with 1,812 samples, and 1 test sequence with 2,566 samples, with 19 valid classes. The evaluation metrics are 3D occupancy IoU, mean semantic IoU, and per-class IoU.

Benchmark	Best previous monocular	VisHall3D
SemanticKITTI test	CGFormer: IoU 44.41, mIoU 16.63	IoU 46.50, mIoU 17.46
SSCBench-KITTI-360 test	CGFormer: IoU 48.07, mIoU 20.05	IoU 49.12, mIoU 20.95

On SemanticKITTI, the reported margin over CGFormer is +2.09 IoU and +0.83 mIoU. The method also exceeds StereoScene, which reports IoU 43.34 and mIoU 15.36, and HTCL-T, which reports IoU 44.23 and mIoU 17.09, slightly surpassing HTCL-T in mIoU and more clearly in IoU. On SSCBench-KITTI-360, the gain over CGFormer is +1.05 IoU and +0.90 mIoU. The paper further states that the method surpasses early LiDAR-based SSCNet and LMSCNet in mIoU on that benchmark.

Efficiency is reported on SemanticKITTI test. MonoScene uses 149.5M parameters, 19,041 MB, and 0.49 s per frame; CGFormer uses 122.4M parameters, 19,330 MB, and 0.41 s; VisHall3D uses 127.8M parameters, 22,597 MB, and 0.34 s. Despite the two-stage design, the parameter count remains close to CGFormer, memory consumption is higher, and inference is faster in the reported environment.

The qualitative comparison in Figure 1 emphasizes roads and sidewalks extending behind cars, building facades and fences beyond visible occluders, smoother surface continuation behind foreground objects, and fewer holes or inconsistent surfaces than MonoScene. The paper attributes these effects to the crisp stage-1 frontier reconstruction and the context-guided hallucination of stage 2.

6. Interpretation, limitations, and significance

VisHall3D’s main contribution is a decomposition of monocular SSC into visible-region reconstruction and invisible-region hallucination. In the paper’s framing, this is both a conceptual reformulation and an architectural intervention. It reassigns visible voxels to a geometry-dominated module with strong image and depth constraints, while assigning occluded and OOV voxels to a denoising completion module conditioned on the reconstructed frontier. This suggests a broader methodological shift in SSC toward explicit occlusion reasoning rather than end-to-end latent fusion (Lu et al., 25 Jul 2025).

The ablation study shows that the gains are not uniform across all objectives. VisFrontierNet without refinement attains the highest IoU among the ablated variants, 46.44, whereas adding feature refinement lowers IoU to 45.88 while improving mIoU to 16.59. This indicates that occupancy and semantic quality need not move monotonically together. The paper’s interpretation is that unsigned-distance geometry is a particularly strong prior for visible-region occupancy, while deformable refinement and denoising contribute more to semantics and class-wise performance.

The reported limitations are tied to the architecture’s explicit dependencies. Visibility estimation and unsigned distance encoding rely on the quality of $(x_v, y_v, d_v) = \mathcal{T}(v),$7 and $(x_v, y_v, d_v) = \mathcal{T}(v),$8, so severe depth errors can misclassify visible versus occluded voxels and degrade the frontier representation. Performance is sensitive to the visibility threshold $(x_v, y_v, d_v) = \mathcal{T}(v),$9. The additional 3D volumes and the two-stage design increase memory usage relative to some baselines. The paper also points to plausible failure modes in textureless regions where depth is unreliable and in unusual scene structures that are underrepresented in training data.

Future directions mentioned in the paper include generalization to other sensors such as multi-camera systems and radar, tighter integration of depth estimation with SSC rather than using off-the-shelf depth prediction, and richer generative priors for hallucination, including diffusion generative models over voxels. In the context of autonomous-driving perception, the broader significance claimed for VisHall3D is that occlusion reasoning and hallucination should be treated as first-class components of 3D scene understanding, and that explicit geometric modeling can be combined productively with denoising-based generative completion.