CIGOcc: Multi-Level 3D Occupancy Fusion

• CIGOcc is a camera-based framework that combines high-level segmentation, mid-level graphics, and low-level depth features to reconstruct 3D scene geometry and semantics.
• It employs a two-stage pipeline where DMFNet fuses segmentation and depth for a coarse voxel scaffold, and CIGNet refines details using deformable attention on graphics features.
• Benchmark results on SemanticKITTI show enhanced mean IoU and improved scene completion quality, particularly in near- and mid-range predictions.

CIGOcc, short for Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusion, is a camera-based 3D occupancy prediction framework for autonomous driving that infers complete 3D scene geometry and semantics from 2D images by fusing three complementary image representations: high-level segmentation features, mid-level graphics features, and low-level depth features. The method is formulated as a two-stage pipeline, combining a coarse 2D-to-3D fusion network with a deformable attention-based voxel refinement network, and further incorporates segmentation knowledge distilled from SAM-derived supervision. In the reported SemanticKITTI evaluation, it achieves a test-set mean IoU of 14.90, exceeding several camera-based semantic scene completion baselines while maintaining similar training and inference cost profiles (Xu et al., 15 Oct 2025).

1. Problem formulation and representational premise

CIGOcc is situated in the literature on camera-only 3D occupancy prediction and semantic scene completion (SSC), where the central task is to reconstruct a semantic occupancy tensor from monocular image evidence despite occlusions, missing observations, and long-range ambiguity. In this setting, occupancy prediction is not treated as a purely geometric lifting problem. Instead, the framework assumes that image understanding for SSC benefits from combining feature modalities that are individually incomplete but jointly informative (Xu et al., 15 Oct 2025).

The motivating decomposition is explicit. Mid-level graphics features encode edges, textures, contours, and shapes, and correspond to the information typically captured by CNN backbones. Low-level depth features carry 3D structural cues but weak semantics. High-level segmentation features, extracted from strong segmentation priors such as Grounded-SAM, provide robust semantics and objectness but no metric depth. The framework’s central claim is that prior work underuses this complementarity by focusing predominantly on architectural modification rather than representation fusion.

Within this formulation, the target occupancy representation is a semantic tensor

$O \in \mathbb{R}^{X \times Y \times Z \times C},$

where $X \times Y \times Z$ denotes the voxel grid and $C$ the number of semantic classes. For the SemanticKITTI setup used in the paper, the spatial extent is $[0, 51.2]$ m forward, $[-25.6, 25.6]$ m lateral, $[-2, 4.4]$ m vertical, with voxel size $0.2$ m, yielding a $256 \times 256 \times 32$ grid and $C=20$ classes.

A plausible implication is that CIGOcc treats occupancy prediction as a representation-alignment problem as much as a volumetric inference problem: the quality of the 3D result depends on how effectively semantic, geometric, and graphics-dominant cues are reconciled before and during lifting to voxel space.

2. Two-stage architecture

CIGOcc consists of two sequential components: DMFNet in Stage 1 and CIGNet in Stage 2 (Xu et al., 15 Oct 2025). Stage 1 constructs a coarse voxel scaffold from segmentation and depth; Stage 2 injects graphics features and performs deformable voxel refinement.

Core components

Component	Function
DMFNet	Fuses segmentation and depth to produce coarse 3D voxel features
CIGNet	Lifts graphics features to sparse voxels and refines semantic occupancy
Grounded-SAM	Supplies segmentation features $S$ and masks $X \times Y \times Z$0
MobileStereoNet	Supplies depth features $X \times Y \times Z$1
ResNet-50	Supplies graphics features $X \times Y \times Z$2
Semantic decoder $X \times Y \times Z$3	Distills SAM mask supervision into the 2D student

In Stage 1, the input is a single RGB image $X \times Y \times Z$4, a stereo depth estimate $X \times Y \times Z$5 from MobileStereoNet, and segmentation features and masks from Grounded-SAM. The segmentation branch contributes dense semantic feature maps and per-instance or thing masks $X \times Y \times Z$6. The depth branch back-projects depth into a point cloud and voxelizes it. DMFNet then fuses segmentation and depth cues using a multi-level deformable fusion procedure and a lightweight U-Net/LMSCNet-style 2D-to-3D transfer, producing three outputs: $X \times Y \times Z$7, a coarse voxel feature volume; $X \times Y \times Z$8, a coarse semantic segmentation volume; and $X \times Y \times Z$9, binary occupancy queries distinguishing occupied from empty voxels.

In Stage 2, the RGB image is passed through a ResNet-50 to obtain graphics features $C$0. These features are lifted into sparse voxel queries through deformable cross-attention (DCA) guided by the Stage-1 occupancy queries $C$1. The lifted graphics-aligned voxel features are then fused with the coarse voxel representation $C$2 and optional mask tokens derived from $C$3. A subsequent deformable self-attention (DSA) module refines the voxel features, and an occupancy head outputs the final semantic occupancy grid

$C$4

This division of labor is structural. DMFNet establishes geometric plausibility and semantic scaffolding from depth and segmentation priors; CIGNet then injects graphics-dominant detail and performs sparse deformable refinement over candidate occupied voxels. The architecture therefore does not attempt dense 3D inference from scratch in a single stage.

3. Deformable multi-level fusion and 2D-to-3D lifting

The technical core of CIGOcc is its deformable multi-level fusion mechanism, which operates first in 2D across modalities and then in sparse 3D query space (Xu et al., 15 Oct 2025).

At each feature level $C$5, segmentation, graphics, and depth features are denoted

$C$6

For a target location $C$7, bilinear sampling is represented by $C$8.

Stage-1 deformable fusion

In DMFNet, fusion is applied over segmentation and depth features. For each location $C$9, modality $[0, 51.2]$0, and sampling index $[0, 51.2]$1, the model predicts offsets $[0, 51.2]$2 and weights $[0, 51.2]$3, and forms the fused feature

$[0, 51.2]$4

with

$[0, 51.2]$5

This defines an adaptive sampling-and-weighting operator across modalities and feature levels. The resulting multi-level fused 2D representation is aggregated across scales and transferred into 3D through depth-guided back-projection and voxelization:

$[0, 51.2]$6

Stage-2 deformable lifting

Stage 2 lifts 2D graphics features into voxel space using DCA. For a voxel query $[0, 51.2]$7 from $[0, 51.2]$8, its center is projected to the image plane to obtain a 2D reference point $[0, 51.2]$9. The lifted feature is then

$[-25.6, 25.6]$0

This produces sparse 3D query features aligned to image evidence near geometry-consistent projections.

The model then fuses these lifted features with the coarse Stage-1 voxel representation:

$[-25.6, 25.6]$1

Refinement proceeds through DSA over deformably sampled voxel neighbors:

$[-25.6, 25.6]$2

where $[-25.6, 25.6]$3 are softmax-normalized attention weights.

Projection geometry

The 2D-to-3D mapping is defined through camera intrinsics $[-25.6, 25.6]$4 and extrinsics $[-25.6, 25.6]$5. A pixel $[-25.6, 25.6]$6 with depth $[-25.6, 25.6]$7 is lifted by

$[-25.6, 25.6]$8

Conversely, a voxel center projects into image coordinates through

$[-25.6, 25.6]$9

The significance of this design is that fusion is not implemented as simple concatenation. The paper presents it as a modality-adaptive, level-adaptive, and locality-adaptive mechanism: $[-2, 4.4]$0 weights encode cross-modality and cross-level confidence, while $[-2, 4.4]$1 weights encode local 3D neighborhood relevance.

4. Supervision, distillation, and optimization

CIGOcc uses different supervision signals at the two stages and augments 3D occupancy learning with a 2D knowledge distillation objective from SAM-derived masks (Xu et al., 15 Oct 2025).

In Stage 1, DMFNet is trained with a weighted semantic cross-entropy over the coarse voxel prediction:

$[-2, 4.4]$2

where $[-2, 4.4]$3 indexes voxels, $[-2, 4.4]$4 indexes semantic classes, and $[-2, 4.4]$5 are inverse-frequency class weights.

In Stage 2, four losses are combined. The first is the distillation loss. A lightweight 2D semantic decoder $[-2, 4.4]$6 is attached to graphics features $[-2, 4.4]$7, producing logits $[-2, 4.4]$8. These logits are aligned to SAM masks $[-2, 4.4]$9 using per-pixel BCE:

$0.2$0

The remaining terms supervise 3D geometry and semantics across scales:

$0.2$1

$0.2$2

and a final weighted semantic SSC loss

$0.2$3

The Stage-2 objective is

$0.2$4

Several implementation details are fixed in the paper. Grounded-SAM uses ViT-H HQ-SAM. Depth is provided by MobileStereoNet (MSNet3D SFDS pretrained). Graphics features are extracted with ResNet-50. DMFNet is trained for 20 epochs on 4× RTX 3090, taking approximately 4.5 hours; CIGNet is trained for 20 epochs on 4× RTX 3090, also approximately 4.5 hours. The paper further reports an efficiency comparison against VoxFormer-T on a single RTX 3090 with batch size 1: latency 0.79 s vs 0.76 s, training memory 17.0 GB vs 16.6 GB, and total hours 17 h vs 16 h.

The paper states that the exact values of the architectural hyperparameters $0.2$5 and $0.2$6 are not fixed in the text; the released code contains those settings.

5. Benchmark results and ablation evidence

On the SemanticKITTI test set, CIGOcc reports 14.90 mIoU, outperforming several camera-based baselines including VoxFormer-T (13.41), MonoOcc-S (13.80), LowRankOcc (13.56), SparseOcc (13.12), SurroundOcc (11.86), and MonoScene (11.08) (Xu et al., 15 Oct 2025).

Method	Test mIoU
CIGOcc	14.90
MonoOcc-S	13.80
LowRankOcc	13.56
VoxFormer-T	13.41
SparseOcc	13.12
SurroundOcc	11.86
MonoScene	11.08

The method also improves performance at different spatial ranges. The reported IoU values at 12.8 / 25.6 / 51.2 m are 67.66 / 59.04 / 44.28 for CIGOcc, compared with 65.38 / 57.69 / 44.15 for VoxFormer-T. The corresponding mIoU values are 23.81 / 20.35 / 14.90 for CIGOcc and 21.55 / 18.42 / 13.35 for VoxFormer-T. The paper interprets this as especially strong near- and mid-range behavior, which is operationally important for driving.

Selected per-class comparisons against VoxFormer-T are also reported. For car, CIGOcc obtains 28.56 vs 21.70; for truck, 11.84 vs 3.60; for other-vehicle, 7.63 vs 4.10; for person, 2.53 vs 1.60; for traffic-sign, 7.86 vs 5.70; for vegetation, 26.96 vs 24.40; and for terrain, 34.28 vs 24.20. The paper summarizes this as improvement on both small or long-tailed objects and large stuff classes.

The ablation study attributes these gains to complementary fusion rather than a single component. Removing knowledge distillation reduces mIoU from 14.49 to 14.10. Using only depth in $0.2$7 yields 13.85, while using depth + SAM features yields 14.49. Removing SAM entirely—meaning no SAM features in Stage 1 and no KD—gives 13.63, whereas the full model gives 14.49. The most pronounced ablation contrast is between the coarse first stage and the complete system: DMFNet alone achieves 9.77 mIoU, while full CIGOcc achieves 14.90, a gain of 5.13.

The reported qualitative results are consistent with these quantitative findings. The paper describes cleaner scene completions, sharper object boundaries, fewer overlaps between classes, and better road layout and object delineation.

6. Interpretation, limitations, and future directions

CIGOcc’s significance lies in its explicit treatment of occupancy prediction as a complementary information fusion problem rather than a purely architectural scaling problem (Xu et al., 15 Oct 2025). Its two-stage design gives a concrete operational form to that position: segmentation priors and depth establish a coarse 3D scaffold, while graphics features recover detail through geometry-aware deformable lifting and sparse voxel refinement.

The framework’s strengths are tightly coupled to the strengths of its three modalities. Depth helps under occlusion and incomplete visibility; SAM-derived segmentation priors stabilize semantic prediction for long-tailed and small objects; graphics features sharpen contours and improve local detail. The deformable attention operators are intended to reduce mismatch among these signals by allowing adaptive cross-level and cross-modality sampling.

The limitations reported in the paper are equally specific. Performance is sensitive to depth quality, especially at long range and under low texture. Adverse weather, lighting variation, and reflective surfaces degrade both depth and graphics features. SAM masks may be imperfect for automotive classes or under occlusion, and overly confident mask supervision can misguide distillation. The method also inherits the ambiguities of a single-camera setting, which multi-view systems can often resolve more directly. In addition, DCA depends on accurate camera calibration; deformable offsets can absorb modest errors, but large calibration biases remain harmful.

The paper outlines several future directions. These include jointly learning depth with occupancy, uncertainty-aware fusion to modulate deformable attention weights by modality confidence, and extending knowledge distillation with boundary-aware or contrastive objectives and text-conditioned SAM priors. A plausible implication is that the method can be generalized beyond the monocular setting to multi-camera rigs or additional sensing modalities, because the fusion principle is not tied to a single source of 2D evidence.

In that sense, CIGOcc marks a specific turn within camera-based SSC: from predominantly backbone- and decoder-centric optimization toward explicit fusion of semantic, geometric, and graphics-dominant representations within a deformable 2D-to-3D inference pipeline.