---
title: 'CIGOcc: Multi-Level 3D Occupancy Fusion'
url: https://www.emergentmind.com/topics/cigocc
type: topic
---

# CIGOcc: Multi-Level 3D Occupancy Fusion

CIGOcc, short for **Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusion**, is a camera-based 3D occupancy prediction framework for autonomous driving that infers complete 3D scene geometry and semantics from 2D images by fusing three complementary image representations: high-level segmentation features, mid-level graphics features, and low-level depth features. The method is formulated as a two-stage pipeline, combining a coarse 2D-to-3D fusion network with a deformable attention-based voxel refinement network, and further incorporates segmentation knowledge distilled from SAM-derived supervision. In the reported SemanticKITTI evaluation, it achieves a test-set mean IoU of **14.90**, exceeding several camera-based semantic scene completion baselines while maintaining similar training and inference cost profiles [2510.13198].

## 1. Problem formulation and representational premise

CIGOcc is situated in the literature on camera-only 3D occupancy prediction and semantic scene completion (SSC), where the central task is to reconstruct a semantic occupancy tensor from monocular image evidence despite occlusions, missing observations, and long-range ambiguity. In this setting, occupancy prediction is not treated as a purely geometric lifting problem. Instead, the framework assumes that image understanding for SSC benefits from combining feature modalities that are individually incomplete but jointly informative [2510.13198].

The motivating decomposition is explicit. **Mid-level graphics features** encode edges, textures, contours, and shapes, and correspond to the information typically captured by CNN backbones. **Low-level depth features** carry 3D structural cues but weak semantics. **High-level segmentation features**, extracted from strong segmentation priors such as Grounded-SAM, provide robust semantics and objectness but no metric depth. The framework’s central claim is that prior work underuses this complementarity by focusing predominantly on architectural modification rather than representation fusion.

Within this formulation, the target occupancy representation is a semantic tensor
$$
O \in \mathbb{R}^{X \times Y \times Z \times C},
$$
where $X \times Y \times Z$ denotes the voxel grid and $C$ the number of semantic classes. For the SemanticKITTI setup used in the paper, the spatial extent is **$[0, 51.2]$ m forward, $[-25.6, 25.6]$ m lateral, $[-2, 4.4]$ m vertical**, with **voxel size $0.2$ m**, yielding a **$256 \times 256 \times 32$** grid and **$C=20$** classes.

A plausible implication is that CIGOcc treats occupancy prediction as a representation-alignment problem as much as a volumetric inference problem: the quality of the 3D result depends on how effectively semantic, geometric, and graphics-dominant cues are reconciled before and during lifting to voxel space.

## 2. Two-stage architecture

CIGOcc consists of two sequential components: **DMFNet** in Stage 1 and **CIGNet** in Stage 2 [2510.13198]. Stage 1 constructs a coarse voxel scaffold from segmentation and depth; Stage 2 injects graphics features and performs deformable voxel refinement.

### Core components

| Component | Function |
|---|---|
| DMFNet | Fuses segmentation and depth to produce coarse 3D voxel features |
| CIGNet | Lifts graphics features to sparse voxels and refines semantic occupancy |
| Grounded-SAM | Supplies segmentation features $S$ and masks $M_s$ |
| MobileStereoNet | Supplies depth features $D$ |
| ResNet-50 | Supplies graphics features $G$ |
| Semantic decoder $\theta_s$ | Distills SAM mask supervision into the 2D student |

In **Stage 1**, the input is a single RGB image $I$, a stereo depth estimate $D$ from MobileStereoNet, and segmentation features and masks from Grounded-SAM. The segmentation branch contributes dense semantic feature maps and per-instance or thing masks $M_s$. The depth branch back-projects depth into a point cloud and voxelizes it. DMFNet then fuses segmentation and depth cues using a multi-level deformable fusion procedure and a lightweight U-Net/LMSCNet-style 2D-to-3D transfer, producing three outputs: **$F_{\text{raw}}$**, a coarse voxel feature volume; **$F_{\text{seg}}$**, a coarse semantic segmentation volume; and **$Q_d$**, binary occupancy queries distinguishing occupied from empty voxels.

In **Stage 2**, the RGB image is passed through a **ResNet-50** to obtain graphics features $G$. These features are lifted into sparse voxel queries through **deformable cross-attention (DCA)** guided by the Stage-1 occupancy queries $Q_d$. The lifted graphics-aligned voxel features are then fused with the coarse voxel representation $F_{\text{raw}}$ and optional mask tokens derived from $Q_d$. A subsequent **deformable self-attention (DSA)** module refines the voxel features, and an occupancy head outputs the final semantic occupancy grid
$$
\hat{Y} \in \mathbb{R}^{X \times Y \times Z \times C}.
$$

This division of labor is structural. DMFNet establishes geometric plausibility and semantic scaffolding from depth and segmentation priors; CIGNet then injects graphics-dominant detail and performs sparse deformable refinement over candidate occupied voxels. The architecture therefore does not attempt dense 3D inference from scratch in a single stage.

## 3. Deformable multi-level fusion and 2D-to-3D lifting

The technical core of CIGOcc is its **deformable multi-level fusion mechanism**, which operates first in 2D across modalities and then in sparse 3D query space [2510.13198].

At each feature level $l$, segmentation, graphics, and depth features are denoted
$$
F_{l,m} \in \mathbb{R}^{C_l \times H_l \times W_l}, \quad m \in \{S,G,D\}.
$$
For a target location $p$, bilinear sampling is represented by $\mathcal{I}(F,p)$.

### Stage-1 deformable fusion

In DMFNet, fusion is applied over segmentation and depth features. For each location $p$, modality $m \in \{S,D\}$, and sampling index $k$, the model predicts offsets $\Delta p_{l,m,k}(p)$ and weights $\alpha_{l,m,k}(p)$, and forms the fused feature
$$
F_{l,\mathrm{fuse2D}}(p)=\sum_{m\in\{S,D\}}\sum_k \alpha_{l,m,k}(p)\cdot \mathcal{I}(F_{l,m},\, p+\Delta p_{l,m,k}(p)),
$$
with
$$
\sum_{m}\sum_k \alpha_{l,m,k}(p)=1, \qquad \alpha \ge 0.
$$
This defines an adaptive sampling-and-weighting operator across modalities and feature levels. The resulting multi-level fused 2D representation is aggregated across scales and transferred into 3D through depth-guided back-projection and voxelization:
$$
F_{\mathrm{raw}}=\mathrm{DMF}(F_{\mathrm{fuse2D}},D).
$$

### Stage-2 deformable lifting

Stage 2 lifts 2D graphics features into voxel space using DCA. For a voxel query $q_i$ from $Q_d$, its center is projected to the image plane to obtain a 2D reference point $p_i$. The lifted feature is then
$$
Q_{s}^{3d}(i)=\sum_l\sum_k \alpha_{l,k}(q_i)\cdot \mathcal{I}(G_l,\, p_i+\Delta p_{l,k}(q_i)).
$$
This produces sparse 3D query features aligned to image evidence near geometry-consistent projections.

The model then fuses these lifted features with the coarse Stage-1 voxel representation:
$$
\hat{Q}_{s}^{3d}=\mathrm{Fuse3D}(Q_{s}^{3d},F_{\mathrm{raw}},\mathrm{Mask}(Q_d)).
$$
Refinement proceeds through DSA over deformably sampled voxel neighbors:
$$
\hat{V}_{s}^{3d}(i)=\sum_{r\in \mathrm{Nbr}(i)} \beta_{i,r}\cdot \hat{V}_{\mathrm{pre}}(r),
$$
where $\beta_{i,r}$ are softmax-normalized attention weights.

### Projection geometry

The 2D-to-3D mapping is defined through camera intrinsics $K$ and extrinsics $[R|t]$. A pixel $(u,v)$ with depth $z$ is lifted by
$$
x_{\mathrm{cam}}=zK^{-1}[u,v,1]^\top,\qquad x_{\mathrm{world}}=Rx_{\mathrm{cam}}+t.
$$
Conversely, a voxel center projects into image coordinates through
$$
\tilde{p}=KR^{-1}(x_{\mathrm{world}}-t), \qquad p=\left(\frac{\tilde{p}_x}{\tilde{p}_z}, \frac{\tilde{p}_y}{\tilde{p}_z}\right).
$$

The significance of this design is that fusion is not implemented as simple concatenation. The paper presents it as a modality-adaptive, level-adaptive, and locality-adaptive mechanism: $\alpha$ weights encode cross-modality and cross-level confidence, while $\beta$ weights encode local 3D neighborhood relevance.

## 4. Supervision, distillation, and optimization

CIGOcc uses different supervision signals at the two stages and augments 3D occupancy learning with a 2D **knowledge distillation** objective from SAM-derived masks [2510.13198].

In **Stage 1**, DMFNet is trained with a weighted semantic cross-entropy over the coarse voxel prediction:
$$
L_{\mathrm{ssc}}^{(1)} = - \sum_{k=1}^{K} \sum_{c=1}^{C} w_c \cdot \hat{y}_{k,c}\cdot \log \mathrm{softmax}(y_k)_c,
$$
where $k$ indexes voxels, $c$ indexes semantic classes, and $w_c$ are inverse-frequency class weights.

In **Stage 2**, four losses are combined. The first is the **distillation loss**. A lightweight 2D semantic decoder $\theta_s$ is attached to graphics features $G$, producing logits $\hat{S}=\theta_s(G)$. These logits are aligned to SAM masks $M_s$ using per-pixel BCE:
$$
L_{\mathrm{kd}} = - \frac{1}{|\Omega|}\sum_{x\in \Omega}\left[M_s(x)\log \sigma(\hat{S}(x)) + (1-M_s(x))\log(1-\sigma(\hat{S}(x)))\right].
$$

The remaining terms supervise 3D geometry and semantics across scales:
$$
L_{\mathrm{scal}}^{\mathrm{geo}} = \sum_{s\in \mathrm{Scales}} \mathrm{BCE}(\hat{\mathrm{occ}}_s,\mathrm{occ}_s),
$$
$$
L_{\mathrm{scal}}^{\mathrm{sem}} = \sum_{s\in \mathrm{Scales}} \mathrm{CE}(\hat{Y}_s[\mathrm{occupied}],Y_s[\mathrm{occupied}]),
$$
and a final weighted semantic SSC loss
$$
L_{\mathrm{ssc}} = -\sum_{k,c} w_c \cdot \hat{y}_{k,c}\cdot \log \mathrm{softmax}(\hat{Y}_k)_c.
$$
The Stage-2 objective is
$$
L^{(2)}=\lambda_1L_{\mathrm{kd}}+\lambda_2L_{\mathrm{scal}}^{\mathrm{geo}}+\lambda_3L_{\mathrm{scal}}^{\mathrm{sem}}+\lambda_4L_{\mathrm{ssc}}.
$$

Several implementation details are fixed in the paper. Grounded-SAM uses **ViT-H HQ-SAM**. Depth is provided by **MobileStereoNet (MSNet3D SFDS pretrained)**. Graphics features are extracted with **ResNet-50**. DMFNet is trained for **20 epochs on 4× RTX 3090**, taking approximately **4.5 hours**; CIGNet is trained for **20 epochs on 4× RTX 3090**, also approximately **4.5 hours**. The paper further reports an efficiency comparison against VoxFormer-T on a single RTX 3090 with batch size 1: **latency 0.79 s vs 0.76 s**, **training memory 17.0 GB vs 16.6 GB**, and **total hours 17 h vs 16 h**.

The paper states that the exact values of the architectural hyperparameters $L$ and $K$ are not fixed in the text; the released code contains those settings.

## 5. Benchmark results and ablation evidence

On the SemanticKITTI test set, CIGOcc reports **14.90 mIoU**, outperforming several camera-based baselines including **VoxFormer-T (13.41)**, **MonoOcc-S (13.80)**, **LowRankOcc (13.56)**, **SparseOcc (13.12)**, **SurroundOcc (11.86)**, and **MonoScene (11.08)** [2510.13198].

| Method | Test mIoU |
|---|---:|
| CIGOcc | 14.90 |
| MonoOcc-S | 13.80 |
| LowRankOcc | 13.56 |
| VoxFormer-T | 13.41 |
| SparseOcc | 13.12 |
| SurroundOcc | 11.86 |
| MonoScene | 11.08 |

The method also improves performance at different spatial ranges. The reported IoU values at **12.8 / 25.6 / 51.2 m** are **67.66 / 59.04 / 44.28** for CIGOcc, compared with **65.38 / 57.69 / 44.15** for VoxFormer-T. The corresponding mIoU values are **23.81 / 20.35 / 14.90** for CIGOcc and **21.55 / 18.42 / 13.35** for VoxFormer-T. The paper interprets this as especially strong near- and mid-range behavior, which is operationally important for driving.

Selected per-class comparisons against VoxFormer-T are also reported. For **car**, CIGOcc obtains **28.56 vs 21.70**; for **truck**, **11.84 vs 3.60**; for **other-vehicle**, **7.63 vs 4.10**; for **person**, **2.53 vs 1.60**; for **traffic-sign**, **7.86 vs 5.70**; for **vegetation**, **26.96 vs 24.40**; and for **terrain**, **34.28 vs 24.20**. The paper summarizes this as improvement on both small or long-tailed objects and large stuff classes.

The ablation study attributes these gains to complementary fusion rather than a single component. Removing knowledge distillation reduces mIoU from **14.49** to **14.10**. Using only depth in $F_{\mathrm{raw}}$ yields **13.85**, while using **depth + SAM features** yields **14.49**. Removing SAM entirely—meaning no SAM features in Stage 1 and no KD—gives **13.63**, whereas the full model gives **14.49**. The most pronounced ablation contrast is between the coarse first stage and the complete system: **DMFNet alone achieves 9.77 mIoU**, while **full CIGOcc achieves 14.90**, a gain of **5.13**.

The reported qualitative results are consistent with these quantitative findings. The paper describes **cleaner scene completions, sharper object boundaries, fewer overlaps between classes, and better road layout and object delineation**.

## 6. Interpretation, limitations, and future directions

CIGOcc’s significance lies in its explicit treatment of occupancy prediction as a **complementary information fusion** problem rather than a purely architectural scaling problem [2510.13198]. Its two-stage design gives a concrete operational form to that position: segmentation priors and depth establish a coarse 3D scaffold, while graphics features recover detail through geometry-aware deformable lifting and sparse voxel refinement.

The framework’s strengths are tightly coupled to the strengths of its three modalities. Depth helps under occlusion and incomplete visibility; SAM-derived segmentation priors stabilize semantic prediction for long-tailed and small objects; graphics features sharpen contours and improve local detail. The deformable attention operators are intended to reduce mismatch among these signals by allowing adaptive cross-level and cross-modality sampling.

The limitations reported in the paper are equally specific. Performance is sensitive to **depth quality**, especially at long range and under low texture. **Adverse weather, lighting variation, and reflective surfaces** degrade both depth and graphics features. **SAM masks may be imperfect for automotive classes or under occlusion**, and overly confident mask supervision can misguide distillation. The method also inherits the **ambiguities of a single-camera setting**, which multi-view systems can often resolve more directly. In addition, DCA depends on accurate camera calibration; deformable offsets can absorb modest errors, but large calibration biases remain harmful.

The paper outlines several future directions. These include **jointly learning depth with occupancy**, **uncertainty-aware fusion** to modulate deformable attention weights by modality confidence, and extending knowledge distillation with **boundary-aware or contrastive objectives** and **text-conditioned SAM priors**. A plausible implication is that the method can be generalized beyond the monocular setting to multi-camera rigs or additional sensing modalities, because the fusion principle is not tied to a single source of 2D evidence.

In that sense, CIGOcc marks a specific turn within camera-based SSC: from predominantly backbone- and decoder-centric optimization toward explicit fusion of semantic, geometric, and graphics-dominant representations within a deformable 2D-to-3D inference pipeline.

Source: https://www.emergentmind.com/topics/cigocc