Occ3D-nuScenes Benchmark

• Occ3D-nuScenes is a large-scale benchmark for 3D semantic occupancy prediction, specifically designed for urban autonomous driving scenes.
• It integrates synchronized multi-sensor data (six cameras, LiDAR, and radars) with a dense voxel annotation pipeline and robust occlusion reasoning.
• Evaluation protocols use metrics like per-class IoU, mIoU, and RayIoU, with diverse baselines highlighting advances in sensor fusion and robustness under occlusion.

The Occ3D-nuScenes benchmark is a large-scale, surround-view 3D semantic occupancy prediction benchmark designed for evaluation of fine-grained geometry and semantic reconstruction in urban autonomous driving scenes. Built atop nuScenes, Occ3D-nuScenes addresses the challenges of high-volume annotation, occlusion reasoning, sensor fusion, and scalable evaluation for vision-centric and multi-modal perception systems.

1. Dataset Composition and Annotation Pipeline

Occ3D-nuScenes comprises 1,000 driving scenes, each roughly 20 seconds, sampled at 2 Hz (about 40,000 multi-sensor frames total). The sensor suite includes six synchronized 360° RGB cameras (1600×900 px), a 32-beam spinning LiDAR, and five FMCW radars (Caesar et al., 2019). Each frame is annotated with a dense voxel grid covering spatial coordinates $[x, y] \in [–40, 40]$ m, $z \in [–1, 5.4]$ m, with $0.4\,\text{m}$ cubic resolution (i.e., $200 \times 200 \times 16$ grid, yielding 640,000 voxels per frame).

Three-stage label generation pipeline (Tian et al., 2023):

1. Voxel Densification:
   • Dynamic/static split: Lidar points are split into static backgrounds and dynamic objects using tracking annotations.
   • Aggregation: Static points are fused into global coordinates, dynamics into track-aligned systems and then re-projected.
   • Mesh reconstruction and KNN labeling: Holes in object surfaces are filled and sampled, labels are assigned via KNN on semantic neighbors.
2. Occlusion Reasoning:
   • Lidar/camera visibility masks: Ray-casting from sensor origins determines observed, free, and unobserved voxels. Only voxels visible in both modalities are used for training/evaluation.
3. Image-guided Voxel Refinement:
   • Camera rays are recast from camera centers to voxel centers, enforcing semantic alignment with 2D segmentation to sharpen 3D boundaries.

Annotation supports 17–18 semantic classes, matching nuScenes panoptic taxonomy (cars, pedestrians, bicyclists, drivable surface, vegetation, etc.) (Tian et al., 2023).

2. Evaluation Protocols and Metrics

The core evaluation metrics are per-class Intersection-over-Union (IoU), mean IoU (mIoU), and optionally geometric metrics such as RayIoU (Chen et al., 3 Jul 2025):

• Semantic IoU:

$$\text{IoU}_c = \frac{\text{TP}_c}{\text{TP}_c + \text{FP}_c + \text{FN}_c}$$

• TP: correctly predicted occupied voxels of class $c$;
• FP: predicted $c$, GT differs;
• FN: GT $c$, predicted otherwise.
  • Mean IoU:

$\text{mIoU} = \frac{1}{C}\sum_{c=1}^C \text{IoU}_c$

where $C$ is the number of semantic classes (typically 17).

• RayIoU (Chen et al., 3 Jul 2025):

$$\text{RayIoU} = \frac{1}{|R|} \sum_{r \in R} \frac{ |P_r \cap G_r | }{ |P_r \cup G_r | }$$

where $R$ is the set of camera rays, $P_r$/$G_r$ are sets of predicted/GT occupied voxels along $r$.

Metrics are calculated only on camera-visible voxels (using ray-casting masks). For downstream detection tasks, mean Average Precision (mAP) and class-wise metrics may also be used (Kumar et al., 21 Oct 2025).

3. Baseline Algorithms and State-of-the-Art Results

Occ3D-nuScenes establishes and evaluates on a variety of baselines spanning camera-only, LiDAR-only, and multi-modal fusion paradigms (Tian et al., 2023, Wang et al., 2023, Shi et al., 8 Jun 2025, Wang et al., 2023, Chen et al., 3 Jul 2025, Hong et al., 13 Jan 2024, Shi et al., 4 Jul 2024, Wu et al., 12 Sep 2024):

• Camera-only baselines: MonoScene, BEVDet, TPVFormer, BEVFormer, OccFormer, CTF-Occ
  • mIoU ranges from 6.06 % (MonoScene) to 28.53 % (CTF-Occ) for early baselines (Tian et al., 2023).
• Advanced vision algorithms:
  • BePo (Shi et al., 8 Jun 2025): BEV + sparse point query, 32.77 % mIoU (single-frame), best on 10/17 classes
  • PanoOcc (Wang et al., 2023): coarse-to-fine voxel queries, up to 42.13 % mIoU with temporal fusion
  • OSP: point-of-interest representation, 39.41 % mIoU (Shi et al., 4 Jul 2024)
  • UniVision (Hong et al., 13 Jan 2024): explicit-implicit view transformation, up to 39.8 % mIoU (high-res)
  • DHD (Wu et al., 12 Sep 2024): deep height decoupling, 41.49 % (DHD-M, with one history frame)
• Fusion and multi-modal approaches:
  • OccFusion (Zhang et al., 8 Mar 2024): LiDAR-camera fusion, active training/refinement, 22.0 % mIoU (A-M-CONet, OpenOccupancy)
  • HyDRa (Wolters et al., 12 Mar 2024): camera-radar fusion, 44.4 % mIoU, surpasses camera-only baselines
  • FMOcc (Chen et al., 3 Jul 2025): tri-perspective flow matching, 39.8 % mIoU, 43.1 % RayIoU (two-frame input)
  • ODG (Shi et al., 11 Jun 2025): dual sparse Gaussians, 38.18 % mIoU (8-frame), 42.3 % RayIoU
• Projective supervision:
  • GaussRender (Chambon et al., 7 Feb 2025): differentiable 2D Gaussian rendering for projective consistency, up to 30.48 % mIoU on TPVFormer backbone.
• Zero-shot/generalization:
  • GS-Occ3D (Ye et al., 25 Jul 2025): vision-only octree Gaussians, trained on Waymo, 33.4 % zero-shot IoU, 50.1 % F1 on nuScenes.

Recent methods integrating temporal fusion (GTAD (Li et al., 28 Jul 2025), DHD (Wu et al., 12 Sep 2024)), projective 2D rendering losses (GaussRender (Chambon et al., 7 Feb 2025)), and adaptive sampling or sparse queries (BePo (Shi et al., 8 Jun 2025), OSP (Shi et al., 4 Jul 2024)) have demonstrably advanced both geometric fidelity and semantic accuracy under challenging urban scenarios.

4. Architectural Innovations and Ablative Findings

Architectural strategies are diverse and extensively evaluated:

• Voxel query resolution & spatial encoding: Height encoding is crucial—omitting Z reduces mIoU by 5–6 points (e.g., PanoOcc: 66.1 % vs. 60.8 % for 16 vs. 4 height bins) (Wang et al., 2023).
• Coarse-to-fine refinement: Reduces memory and compute by focusing high-resolution refinement on uncertain foreground voxels (Tian et al., 2023, Zhang et al., 8 Mar 2024).
• Cross-attention bridges: Injecting sparse-point features into BEV (BePo) brings a +0.49 mIoU gain (Shi et al., 8 Jun 2025).
• Flow matching vs. diffusion: FMOcc’s flow matching SSM dominates diffusion and vanilla Transformer, improving RayIoU by +4.5–10.5 points (Chen et al., 3 Jul 2025).
• Explicit height decoupling: DHD’s Mask Guided Height Sampling reduces feature confusion, yielding up to +4.78 % mIoU improvement over non-decoupled baselines (Wu et al., 12 Sep 2024).
• Projective supervision and rendering: Gaussian splatting-based 2D losses (GaussRender) induce spatial coherence, boosting mIoU by up to +2.65 points (Chambon et al., 7 Feb 2025).
• Temporal fusion and denoising: GTAD’s global temporal aggregation uses in-model latent denoising for improved holistic scene understanding (+4.1 pts over PanoOcc for 12-epoch train) (Li et al., 28 Jul 2025).

5. Robustness to Sensor Occlusion and Adverse Conditions

The Occ3D-nuScenes benchmark is extended for rigorous robustness testing under controlled sensor occlusions via the Occluded nuScenes dataset (Kumar et al., 21 Oct 2025). It provides parameterizable scripts for:

• Camera: Dirt simulation, water-blur, scratch overlay, WoodScape-style soiling. Opacity $\alpha$ and Gaussian smoothing kernel size $\sigma$ are tunable, yielding degradation in IoU of 18–33% depending on occlusion type.
• Radar, LiDAR: Sensor dropout, uniform point dropout (0–99%), region/angle-based occlusion, Gaussian noise ($\sigma \in [0.1, 2]$ m).
• Evaluation protocol: Model pipelines are evaluated with identical metrics (mIoU, IoU, mAP) under controlled severity levels. Baseline performance drops up to 33% (vehicle segmentation) for maximal occlusion.

Scripts, reproducibility tools, and documentation are provided for integration with existing Occ3D-nuScenes pipelines (Kumar et al., 21 Oct 2025). This enables benchmarking against partial sensor failures, environmental artifacts, and resilient fusion architectures.

6. Analysis of Strengths, Limitations, and Generalization

Analysis across benchmarks yields several findings:

• Vision-only methods: Struggle with thin objects and far occlusions, require explicit spatial priors, and benefit from octree or Gaussian surfel decomposition (GS-Occ3D (Ye et al., 25 Jul 2025)). High precision on static large structures, but recall is sensitive to training domain.
• Multi-modal fusion: LiDAR/radar features provide superior long-range and depth consistency but are costlier. HyDRa (Wolters et al., 12 Mar 2024) demonstrates camera-radar synergy, yielding highest mIoU reported.
• Coarse-to-fine and point-adaptive methods: Achieve competitive throughput, memory efficiency, and accuracy—critical for real-time applications on embedded hardware (BePo (Shi et al., 8 Jun 2025), OSP (Shi et al., 4 Jul 2024)).
• Projective consistency: Methods relying on 2D rendering losses generalize better for geometric surface fidelity, particularly under sparse voxel evaluations (GaussRender (Chambon et al., 7 Feb 2025)).
• Robustness: Mask training (FMOcc (Chen et al., 3 Jul 2025)) and selective fusion (Occluded nuScenes (Kumar et al., 21 Oct 2025)) are decisive for resilience under input-dropout and occlusions.

Zero-shot generalization results (GS-Occ3D (Ye et al., 25 Jul 2025)) indicate that explicit modeling of ground/dynamic/static splits and multi-scale geometry can mitigate domain shift between datasets (Waymo → nuScenes).

7. Impact, Tooling, and Reproducibility

Occ3D-nuScenes has catalyzed methodological diversity in semantic occupancy prediction and robust spatial reasoning. Reference codebases are available for core models, annotation pipelines, and controlled occlusion generators (Tian et al., 2023, Kumar et al., 21 Oct 2025). Comprehensive evaluation protocols, standardized metrics, and large annotated data volume position Occ3D-nuScenes as the central benchmark for scalable, deployable 3D perception research in autonomous driving.

Recommended practices include:

• Use camera visibility masks for fair semantic evaluation.
• Integrate cross-modal sensor fusion for challenging urban coverage and occlusion resolution.
• Employ coarse-to-fine or adaptive query strategies for compute-constrained scenarios.
• Apply projective-consistency losses and explicit height priors to achieve superior geometric fidelity.
• Benchmark under controlled occlusion for robust deployment in adverse and failure-prone scenarios.

Occ3D-nuScenes, through its rich annotation pipeline, exhaustive evaluation metrics, methodological benchmarks, and reproducibility standards, anchors research in high-fidelity, resilience-tested, real-world 3D occupancy prediction and fusion (Tian et al., 2023, Wang et al., 2023, Wu et al., 12 Sep 2024, Shi et al., 8 Jun 2025, Kumar et al., 21 Oct 2025, Zhang et al., 8 Mar 2024, Wolters et al., 12 Mar 2024, Chambon et al., 7 Feb 2025, Ye et al., 25 Jul 2025, Chen et al., 3 Jul 2025, Shi et al., 4 Jul 2024).