LightOcc: Efficient 3D Occupancy Prediction

• LightOcc is a vision-based 3D occupancy prediction framework that employs spatial embedding to recover fine-grained height cues from multi-view images.
• It uses a spatial-to-channel mechanism and tri-perspective (TPV) 2D convolutions to efficiently fuse multi-view data, significantly boosting mIoU with minimal latency.
• Extensive evaluations on the Occ3D-nuScenes benchmark demonstrate LightOcc’s ability to enhance 3D scene understanding while remaining computationally viable for real-time applications.

LightOcc is a vision-based 3D occupancy prediction framework designed to deliver fine-grained, voxel-level scene understanding from multi-view images with computational efficiency suitable for real-time autonomous driving systems. LightOcc introduces a set of spatial embedding strategies that reconstruct essential height information typically lost in efficient 2D BEV pipelines, while entirely sidestepping the memory and compute challenges posed by dense 3D voxel representations and 3D convolution.

1. Motivation and Context

Traditional 3D occupancy prediction models rely on high-dimensional voxel features and expensive 3D convolutional operations, imposing large overheads in both memory and computation. While efficient methods approximate the 3D occupancy with 2D BEV features, such compression discards critical scene height cues and suffers in accuracy. Approaches such as Channel-to-Height attempt to repurpose BEV channels to encode height but are fundamentally limited by the channel budget and lack explicit spatial grounding. LightOcc addresses this by supplementing BEV pipelines with a lightweight spatial embedding that injects explicit and implicit height information directly into the representation, realized through a series of innovations in global sampling, spatial-channel permutation, and efficient multi-view fusion (Zhang et al., 2024).

2. Global Spatial Sampling

LightOcc begins with global spatial sampling to aggregate multi-view depth information into a compact single-channel 3D "occupancy" tensor. For each camera view $i$ with depth distribution

$\mathbf{D}_i \in \mathbb{R}^{D \times H \times W},$

where $D$ is the number of depth bins, the framework defines a 3D voxel grid in world coordinates $(x, y, z) \in [1..X] \times [1..Y] \times [1..Z]$. These coordinates are projected into each image using intrinsic and extrinsic camera parameters: $$(d, h, w) = K^{-1}\left[R\,(x,y,z)^\top + t\right].$$ For each voxel, samples from all views are summed to yield the aggregated single-channel occupancy: $$\mathbf{O}_{SC}(x, y, z) = \sum_{i=1}^N \text{Sampling}\left(\mathbf{D}_i,\, (d_i, h_i, w_i)\right).$$ This tensor $$\mathbf{O}_{SC} \in \mathbb{R}^{1 \times X \times Y \times Z}$$ provides efficient global volumetric evidence at only one occupancy channel per voxel, resulting in computational complexity $O(XYZ)$, substantially below full $C$-channel 3D volumes.

3. Spatial-to-Channel Mechanism and TPV Embeddings

To recover 3D structure without high-dimensional volumes, LightOcc reinterprets spatial axes as convolutional channels, enabling the extraction of three tri-perspective view (TPV) embeddings via 2D convolutions:

• BEV Embedding: Permute $\mathbf{O}_{SC}$ to $[Z; X; Y]$, casting height $Z$ as channel, and apply a $3 \times 3$ 2D convolution to yield

$$\mathbf{E}_{\mathrm{BEV}} = \mathrm{Conv}_{2d}\left(\mathbf{O}_{SC}\right) \in \mathbb{R}^{C \times X \times Y}.$$

This representation contains implicit height cues within its channels.

• Front-View (FV) Embedding: Permute $\mathbf{O}_{SC}$ to $[X; Y; Z]$, taking $X$ as channel and applying a 2D convolution:

$$\mathbf{E}_{\mathrm{FV}} = \mathrm{Conv}_{2d}\left(\mathbf{O}_{SC}\right) \in \mathbb{R}^{C \times Y \times Z}.$$

Height $Z$ is preserved as a spatial dimension.

• Side-View (SV) Embedding: Permute $\mathbf{O}_{SC}$ to $[Y; X; Z]$, $Y$ as channel, 2D convolution:

$$\mathbf{E}_{\mathrm{SV}} = \mathrm{Conv}_{2d}\left(\mathbf{O}_{SC}\right) \in \mathbb{R}^{C \times X \times Z}.$$

All convolutions are a single 2D layer with $3 \times 3$ kernel and padding 1, with channel width $C=64$ (small) or $C=128$ (large). This approach enables the framework to exploit both implicit and explicit height cues from the data.

4. Lightweight TPV Interaction and BEV Fusion

The tri-perspective embeddings are fused via matrix multiplications and 2D convolutions without expensive attention mechanisms:

• Interact to BEV:

$$\mathbf{E}^{M}_{\mathrm{BEV}} = \left(\mathbf{E}_{\mathrm{SV}} \rightarrow C \times X \times Z\right) \otimes \left(\mathbf{E}_{\mathrm{FV}} \rightarrow C \times Z \times Y\right) \in \mathbb{R}^{C \times X \times Y},$$

where $\otimes$ denotes batched matrix multiplication. This is combined with the BEV embedding and convolved:

$$\mathbf{E}^{I}_{\mathrm{BEV}} = \mathrm{Conv}_{2d}\left(\mathbf{E}_{\mathrm{BEV}} + \mathbf{E}^{M}_{\mathrm{BEV}}\right).$$

• FV and SV can be mixed similarly for completeness.
• The final spatial embedding is produced as:

$$\mathbf{E}^{M}_{S} = \left(\mathbf{E}^{I}_{\mathrm{SV}} \rightarrow C \times X \times Z\right) \otimes \left(\mathbf{E}^{I}_{\mathrm{FV}} \rightarrow C \times Z \times Y\right),$$

$$\mathbf{E}_{S} = \mathrm{Conv}_{2d}\left(\mathbf{E}^{I}_{\mathrm{BEV}} + \mathbf{E}^{M}_{S}\right) \in \mathbb{R}^{C \times X \times Y}.$$

• $\mathbf{E}_{S}$ encodes both implicit (channel-based) and explicit (spatial $Z$-dimension) height cues.
• BEV fusion is obtained by direct addition of the spatial embedding to the original BEV feature:

$$\mathbf{F}'_{\mathrm{BEV}} = \mathbf{F}_{\mathrm{BEV}} + \mathbf{E}_{S}.$$

This is followed by the standard BEV backbone for per-voxel semantic prediction.

5. Training, Augmentation, and Implementation

Optimization is based on a weighted sum of per-voxel cross-entropy, Lovász-Softmax, and Scene-Class Affinity losses: $$\mathcal{L} = \lambda_{\mathrm{CE}}\,\mathcal{L}_{\mathrm{CE}} + \lambda_{\mathrm{Lovasz}}\,\mathcal{L}_{\mathrm{Lovasz}} + \lambda_{\rm SC}\,\mathcal{L}_{\rm SC}.$$ AdamW optimizer is used with learning rate $2\times10^{-4}$ and a 48-epoch schedule. Regularization and data augmentation strategies include BEV-CutMix, which divides the visibility-masked BEV into four quadrants for inter-scene mixing, preventing occlusion artifacts. Image flips and scale jittering are applied to the RGB streams, with BEV flip augmentation. A sigmoid activation on depth bins is employed, as it better matches occupancy supervision compared to softmax settings. Visibility masks are used during BEV-CutMix to avoid invalid occlusion permutations.

6. Quantitative Evaluation and Ablation

On the Occ3D-nuScenes benchmark, LightOcc demonstrates significant improvements in mIoU over strong BEV-only baselines, with minimal computational overhead:

Method	History	mIoU ↑	Latency ms ↓
FlashOcc (baseline)	55	32.08	35.08
+ LightOcc-S	55	37.93	35.23

Larger models (Swin-B, $512 \times 1408$ input) further increase accuracy (LightOcc-L, mIoU 46.00 with 1 historical frame, 47.24 with 8), outperforming previous SOTA occupied voxel predictors such as TPVFormer and COTR, with a marginal inference cost increase of approximately 0.15 ms per frame.

Ablation studies show the stepwise effect of each module:

Variant	mIoU ↑	Latency ms
Baseline (FlashOcc-opt)	35.00	33.51
+ Spatial-to-Channel (only BEV Embedding)	36.30	34.81
+ LTI (full Spatial Embedding)	36.86	35.23
+ longer training (48 epochs)	37.27	35.23
+ BEV-CutMix	37.93	35.23

Notable module-wise improvements are +1.30% mIoU (Spatial-to-Channel), +0.56% (Lightweight TPV Interaction), and +0.66% (BEV-CutMix).

7. Significance and Implications

LightOcc achieves a balance between voxel-level accuracy and deployment efficiency for vision-based 3D occupancy, equaling or surpassing prior state-of-the-art methods on established benchmarks. By leveraging single-channel 3D occupancy sampling, axial permutations, and lightweight fusion, LightOcc recovers accurate 3D structure without the cost of high-dimensional voxel features or 3D convolutions. Its minimal computational overhead is suitable for real-time systems, and the core ideas extend to any framework adopting a BEV-centric pipeline, such as FlashOcc. This suggests LightOcc provides a deployable pathway for embedding fine-grained 3D spatial context into efficient real-time scene perception architectures (Zhang et al., 2024).