UnsOcc: 3D Semantic Occupancy Prediction in Unstructured Scene via Rendering Fusion

Abstract: Unstructured scenes present unique challenges for autonomous driving, as irregular obstacles and sparse scene layouts undermine the effectiveness of traditional perception methods such as 3D object detection. 3D semantic occupancy prediction has emerged as a prominent focus due to its ability to provide dense spatial representations by assigning semantic labels to individual voxels in 3D space. However, directly applying 3D semantic occupancy prediction to unstructured scenes remains challenging because scene sparsity hinders effective cross-modal fusion and the more severe long-tail distribution in these scenarios further degrades prediction performance. To validate the effectiveness of our approach, we construct a dedicated dataset of unstructured scenes collected from open-pit mines. Based on this, we propose UnsOcc, a multi-modal 3D semantic occupancy prediction framework that improves robustness in unstructured environments. At its core, we introduce a rendering-based fusion module, RenderFusion, which enhances cross-modal feature alignment through bidirectional rendering supervision. Furthermore, we propose GSRefinement, a detail-aware auxiliary supervision method based on Gaussian Splatting that projects sparse 3D occupancy predictions into dense 2D semantic segmentation maps, enabling effective supervision for long-tail categories. Extensive experiments on both the open-pit mine dataset and the nuScenes dataset demonstrate that our method significantly outperforms existing state-of-the-art approaches.

• The paper introduces RenderFusion for bidirectional image–LiDAR alignment and GSRefinement for dense 2D supervision, achieving 20.57 mIoU and 17.74 long-tail mIoU on its open-pit mine benchmark.
• The method improves rare-category recognition, raising widebody truck IoU from 0.16 to 15.55 and muddy-terrain IoU from 8.34 to 14.70 by compensating for sparse LiDAR coverage with image-guided rendering.
• The paper releases a 135-sequence, 11-class open-pit mine dataset and shows 28.1 mIoU on nuScenes-Occ, while noting that extremely rare classes such as pedestrians remain challenging.

UnsOcc addresses 3D semantic occupancy prediction in unstructured environments, where irregular obstacles, sparse layouts, and severe class imbalance degrade conventional perception. The paper makes three contributions: a rendering-based cross-modal alignment module (RenderFusion), a Gaussian Splatting–based auxiliary supervision scheme (GSRefinement) targeting long-tail classes, and a new open-pit mine occupancy dataset collected from real mining sites.

Motivation and problem setting

The authors argue that unstructured scenes—exemplified by open-pit mines—break two assumptions underlying existing occupancy pipelines. First, cross-modal fusion is hampered because LiDAR features carry no semantics and image features carry no metric depth; in sparse scenes with weak geometric cues this misalignment becomes acute. Second, the long-tail problem is more extreme than on urban benchmarks: in their dataset, over 98% of voxels are empty, and rare categories such as widebody trucks (0.007% of voxels) and pedestrians (0.007%) receive almost no direct 3D supervision. Existing fusion schemes—channel concatenation (OccFusion), KNN-assisted coupling (Co-Occ), or weighted summation (CONet)—operate purely at the feature level and do not impose cross-modal consistency during training.

Method

The framework uses a dual-branch backbone: a ResNet + SECONDFPN image encoder producing 128-dimensional features, and a voxelized sparse-convolution LiDAR encoder. Two mechanisms act on these branches.

RenderFusion imposes bidirectional supervision before fusion. In the camera branch, a depth net produces a discrete depth distribution $p_d$, lifted into image-depth space via LSS-style outer product; projected LiDAR points supply one-hot depth ground truth $Z_p$ supervised by binary cross-entropy, so the lifted image features are geometrically consistent with the LiDAR frame. In the LiDAR branch, a lightweight segmentation head converts sparse voxel features into volumetric semantic logits, which are rendered to the image plane via alpha-blending (3D Gaussian projection with covariance $\Sigma' = JW\Sigma W^TJ^T$) and supervised by 2D segmentation maps derived from the images. This injects semantics into the geometry-only LiDAR features prior to voxel-level fusion.

GSRefinement targets label sparsity at training time. The fused decoder outputs 3D occupancy logits $\mathcal{O}$, which are splatted to the camera plane to yield 2D semantic logits and per-Gaussian center depths. These rendered predictions are supervised against dense 2D segmentation and depth ground truth using cross-entropy and SILog losses respectively, alongside the standard 3D occupancy loss. Because empty-class density is far lower in image space than in voxel space, this effectively multiplies the supervision signal for non-empty—and particularly long-tail—categories.

Dataset

The open-pit mine dataset comprises 135 sequences (~55 frames each) from multiple Chinese mines, captured with a front-mounted LiDAR, monocular camera, and INS under day/night, rain, snow, and dust conditions. Ground truth is built by multi-sweep static map reconstruction after dynamic-object removal, voxelized at 0.3 m resolution into a [256, 256, 32] grid with 11 semantic classes plus empty and unknown. The split is 100/10/25 for train/validation/test. The class distribution is heavily skewed: driveable surface (73.99%) and terrain (23.70%) dominate, while nine classes fall below 1.1%.

Results

On the open-pit mine test set, UnsOcc reaches 20.57 mIoU and 17.74 mIoU(long-tail), versus 18.50 mIoU for the Co-Occ fusion baseline—a relative gain of roughly 11%. Against recent camera-only methods the gains are larger: +30.6% over CGFormer (15.72 mIoU) and +28.6% over L2COcc (15.99 mIoU). On rare classes the improvements are pronounced: widebody improves from 0.16 to 15.55 IoU, car from 19.21 to 23.10, and muddy terrain from 8.34 to 14.70, the latter being notable because muddy regions produce few LiDAR returns, causing even fusion baselines like Co-Occ to fail; the paper attributes this gain specifically to image-guided rendering supervision compensating for the LiDAR deficiency.

On nuScenes-Occ validation, UnsOcc achieves 28.1 mIoU, surpassing Co-Occ (27.1) and M-CONet (24.7), with the largest per-class gains on traffic cones (22.7 vs. 15.9) and bicycles (18.2 vs. 16.1). Ablations attribute +1.13 mIoU (+1.45 long-tail) to GSRefinement alone and +0.55 mIoU (+0.75 long-tail) to RenderFusion alone, with complementary gains when combined. The method was also deployed on mining vehicles with both monocular and multi-view cameras, where the reported qualitative behavior was consistent with offline results.

Limitations and open questions

Several caveats deserve note. Pedestrian IoU remains exactly 0.00 on the mine dataset despite the framework's stated long-tail focus, indicating that sub-0.01%-frequency classes are still effectively unsolved. The nuScenes evaluation uses a monocular-camera configuration while most compared baselines use surround-view rigs, so cross-dataset comparisons should be read cautiously. The 2D supervision depends on the availability of image-space segmentation ground truth matching the input views, an assumption that may not hold for all deployment settings. Finally, the ablation isolates components only on the mine dataset; whether RenderFusion's bidirectional alignment generalizes to structured urban scenes as a standalone contribution is not separately quantified.

Conclusion

UnsOcc demonstrates that rendering-based bidirectional supervision can substitute for explicit feature-alignment machinery in multimodal occupancy prediction, and that projecting sparse 3D predictions into dense 2D space via Gaussian Splatting is an effective remedy for extreme label sparsity. The strong gains on muddy terrain and rare vehicle classes in open-pit mines, together with competitive nuScenes results, support the claim that these mechanisms improve robustness in unstructured scenes, though performance on the rarest classes remains an open problem.