Fusion-to-Occupancy Active Distillation

• The paper introduces a teacher–student distillation mechanism that leverages fused LiDAR and camera BEV features to supervise image-only representations.
• It incorporates occupancy-aware region selection to focus on active and inactive regions, ensuring precise transfer of geometric and semantic information.
• Empirical results on nuScenes datasets show significant IoU and mIoU improvements, achieving state-of-the-art real-time performance in autonomous driving.

Searching arXiv for the cited occupancy and distillation papers to ground the article and citations. arXiv search: (Duan et al., 22 Jul 2025) SDGOCC Fusion-to-occupancy-driven active distillation denotes a teacher–student distillation mechanism for 3D multimodal occupancy prediction in which a fused bird’s-eye-view (BEV) representation, derived from LiDAR and camera inputs, supervises an image-only BEV representation through occupancy-aware region selection. In SDG-OCC, the method is paired with a joint semantic and depth-guided view transformation and is designed to extract rich semantic information from multimodal data and selectively transfer knowledge to image features based on LiDAR-identified regions. The resulting framework is presented for autonomous-driving occupancy prediction, with state-of-the-art performance with real-time processing on the Occ3D-nuScenes dataset and comparable performance on the more challenging SurroundOcc-nuScenes dataset (Duan et al., 22 Jul 2025).

1. Problem setting within 3D occupancy prediction

3D occupancy prediction has emerged as a scene-understanding task that estimates occupancy states and semantics of 3D scenes from sensor observations. Within this setting, multimodal methods are motivated by a persistent asymmetry between sensing modalities: camera-based methods lack depth information, while LiDAR-based methods struggle with occlusions. The SDG-OCC formulation situates fusion-to-occupancy-driven active distillation precisely at this interface, using multimodal supervision during training to improve an image-only pathway at inference time (Duan et al., 22 Jul 2025).

The method also addresses a broader efficiency problem in occupancy learning. EFFOcc characterizes existing 3D occupancy networks as both computationally heavy and label-hungry, commonly relying on heavy Conv3D modules or transformers at the voxel level and expensive large-scale dense voxel labels. EFFOcc therefore proposes an efficient fusion-based OccNet and a multi-stage occupancy-oriented distillation to efficiently transfer knowledge to vision-only OccNet, indicating that distillation from stronger multimodal occupancy models to lighter visual models had already become a central design pattern in the field (Shi et al., 2024).

A related concern appears in RadOcc, which observes that image-based scene perception encounters significant challenges in achieving accurate prediction due to the absence of geometric priors, and further states that directly applying features or logits alignment, proposed and widely used in bird's-eye-view perception, does not yield satisfactory results for 3D occupancy prediction. This positions fusion-to-occupancy-driven active distillation as part of a broader search for occupancy-specific distillation criteria rather than a direct reuse of generic BEV distillation recipes (Zhang et al., 2023).

2. Placement inside SDG-OCC and teacher–student structure

In SDG-OCC, fusion-to-occupancy-driven active distillation is inserted between the fused BEV features and the image BEV features, acting as a teacher–student scheme. The teacher network is the fused BEV pipeline, which processes LiDAR and camera inputs through neighborhood fusion to obtain $F^{fuse}_{bev}$. The student network is the image-only BEV pipeline, which maps camera inputs through the SDG view transformation to obtain $F^C_{bev}$. During training, the module unidirectionally transfers knowledge from the teacher’s fused BEV features into the student’s image-only BEV features, after which both branches feed into the same occupancy head (Duan et al., 22 Jul 2025).

The full SDG-OCC architecture is described as four major blocks: Image Feature Encoder, SDG View Transformation, LiDAR Branch plus Dynamic Neighborhood Fusion, and Occupancy Prediction Head. The student input is multi-view RGB frames,

$X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$

and the teacher input adds a LiDAR point cloud,

$X_L\in\mathbb R^{N_L\times (3+d)}.$

On the student side, an image encoder such as ResNet-50 + FPN produces multi-scale feature maps

$F_C\in\mathbb R^{N_C\times C\times H\times W},$

which are transformed into image BEV features

$F^C_{bev}\in\mathbb R^{H\times W\times C'}.$

On the teacher side, the same image encoder provides $F^C_{bev}$, the LiDAR branch uses an SPVCNN voxel$\to$point encoder to produce

$F^L_{bev}\in\mathbb R^{H\times W\times C'},$

and dynamic neighborhood fusion produces

$F^{fuse}_{bev}\in\mathbb R^{H\times W\times C'}.$

The representations entering distillation are therefore explicitly

$F^C_{bev}$0

This organization matters because the distillation target is not a latent feature from an unrelated modality encoder, nor a final voxel logit tensor alone. It is a fused BEV representation already shaped by multimodal geometric and semantic evidence, and therefore closer to the occupancy prediction interface.

3. Occupancy-aware activation: AR and IR region selection

The defining step in fusion-to-occupancy-driven active distillation is region selection via LiDAR-identified masks. Binary occupancy masks are generated on the BEV grid from teacher and student representations:

$F^C_{bev}$1 if the teacher BEV feature at $F^C_{bev}$2 lies in an occupied voxel, and $F^C_{bev}$3 similarly for the student. Two regions are then defined: $F^C_{bev}$4

$F^C_{bev}$5

The paper names these the Active Region (AR) and Inactive Region (IR) (Duan et al., 22 Jul 2025).

These masks focus distillation on BEV cells where LiDAR sees occupancy and either the student already sees it or misses it. This suggests that the method does not treat all BEV locations as equally informative for transfer. Instead, it allocates supervision only where the fused teacher indicates occupancy, and differentiates between agreement and miss cases. A plausible implication is that this makes the distillation objective explicitly occupancy-driven rather than purely feature-driven.

This occupancy-aware partition also clarifies the meaning of “active” in the method name. The activation criterion is not active learning in the annotation-selection sense; it is selective distillation over teacher-identified occupied regions, with distinct handling for already-recognized and missed image-only cells.

4. Feature alignment, fusion, and optimization objective

The method assumes that teacher features $F^C_{bev}$6 and student features $F^C_{bev}$7 are misaligned by small extrinsic or calibration offsets. To compensate, the teacher obtains a local neighborhood feature around each pixel via Neighborhood Attention, using a $F^C_{bev}$8 patch; in ablation, $F^C_{bev}$9 was optimal. The fused BEV feature is then computed by a gated fusion in which a sigmoid gate and a $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$0 projection modulate the neighborhood feature (Duan et al., 22 Jul 2025).

The SDG view transformation, which supplies the student representation, is also part of the optimization context. It uses a LiDAR-sparse depth map $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$1 and an image semantic mask $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$2 to diffuse depth within each semantic region, and applies bidirectional linear-increasing and decreasing discretization around each diffused depth to place far fewer but more accurate virtual rays into BEV. This reduces overhead relative to vanilla Lift-Splat-Shoot while tightening the coupling between semantic grouping and depth placement.

The distillation loss is governed by a per-pixel weighting map: $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$3 where

$X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$4

balances the typically smaller IR region, and $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$5 are hyper-parameters. The fusion-to-occupancy distillation loss is an $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$6 penalty between teacher and student BEV features: $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$7 The complete objective adds this term to the standard occupancy classification loss $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$8, described as voxel-wise cross-entropy over semantics plus empty or unknown: $X_C\in\mathbb R^{N_C\times H_C\times W_C\times 3},$9 with $X_L\in\mathbb R^{N_L\times (3+d)}.$0 set via validation (Duan et al., 22 Jul 2025).

The reported weighting design is important because, without masking or with uniform weights, distillation over-focuses on already-correct areas in AR. Adaptive $X_L\in\mathbb R^{N_L\times (3+d)}.$1 scaling ensures that the missing-in-image region IR receives stronger gradient. This makes the transfer objective selective twice over: first by restricting supervision to teacher-occupied cells, and second by amplifying cells that remain unrecognized by the student.

5. Empirical effect and reported system variants

The ablation study on Occ3D-nuScenes reports a clear contribution from fusion-to-occupancy-driven active distillation. The baseline without SDG and without FOAD obtains IoU $X_L\in\mathbb R^{N_L\times (3+d)}.$2 and mIoU $X_L\in\mathbb R^{N_L\times (3+d)}.$3. Adding FOAD only yields IoU $X_L\in\mathbb R^{N_L\times (3+d)}.$4 $X_L\in\mathbb R^{N_L\times (3+d)}.$5 and mIoU $X_L\in\mathbb R^{N_L\times (3+d)}.$6 $X_L\in\mathbb R^{N_L\times (3+d)}.$7. Adding SDG only yields IoU $X_L\in\mathbb R^{N_L\times (3+d)}.$8 $X_L\in\mathbb R^{N_L\times (3+d)}.$9 and mIoU $F_C\in\mathbb R^{N_C\times C\times H\times W},$0 $F_C\in\mathbb R^{N_C\times C\times H\times W},$1. Combining SDG and FOAD yields IoU $F_C\in\mathbb R^{N_C\times C\times H\times W},$2 $F_C\in\mathbb R^{N_C\times C\times H\times W},$3 and mIoU $F_C\in\mathbb R^{N_C\times C\times H\times W},$4 $F_C\in\mathbb R^{N_C\times C\times H\times W},$5 (Duan et al., 22 Jul 2025).

These numbers indicate that FOAD alone already brings a large jump in both IoU and mIoU, while the joint use of the SDG view transform and FOAD yields the reported final performance. The paper further states that, for optimal performance, it introduces SDG-Fusion, which uses fusion alone, and SDG-KL, which integrates both fusion and distillation for faster inference. The abstract also states that the method achieves state-of-the-art performance with real-time processing on Occ3D-nuScenes and shows comparable performance on SurroundOcc-nuScenes.

Within the narrower question of distillation efficacy, the ablation suggests that FOAD is not merely auxiliary regularization. Its performance increment is substantial even when introduced without SDG, which suggests that selective transfer from fused BEV features captures supervision not readily available through image-only occupancy classification loss alone.

6. Relation to adjacent occupancy distillation paradigms

Fusion-to-occupancy-driven active distillation belongs to a family of cross-modal occupancy distillation methods, but its mechanism is distinct from both rendering-based transfer and label-efficiency-driven occupancy distillation. RadOcc introduces a rendering assisted distillation paradigm for 3D occupancy prediction, using differentiable volume rendering to generate depth and semantic maps in perspective views and defining a depth consistency loss that aligns the termination distributions of rendered rays together with a semantic consistency loss that mimics intra-segment similarity guided by vision foundation models. Experimental results on nuScenes are reported to enhance the baseline by $F_C\in\mathbb R^{N_C\times C\times H\times W},$6 in the metric of mIoU and achieve $F_C\in\mathbb R^{N_C\times C\times H\times W},$7 in the Occ3D benchmark (Zhang et al., 2023).

The contrast is methodologically instructive. RadOcc explicitly reports that directly applying features or logits alignment, proposed and widely used in BEV perception, does not yield satisfactory results. FOAD responds to a closely related issue from a different angle: instead of moving the supervision target into perspective rendering space, it keeps the transfer in BEV space but restricts it to LiDAR-identified occupancy regions and reweights missed image-only regions. This suggests that occupancy distillation can be specialized either by changing the representational domain of supervision, as in rendering-assisted distillation, or by changing the spatial support and weighting of BEV-space supervision, as in FOAD.

EFFOcc provides a second adjacent reference point. It proposes an efficient fusion-based OccNet that only uses simple 2D operators and improves accuracy to the state-of-the-art on Occ3D-nuScenes, Occ3D-Waymo, and OpenOccupancy-nuScenes. On Occ3D-nuScenes, the fusion-based model with ResNet-18 as the image backbone has $F_C\in\mathbb R^{N_C\times C\times H\times W},$8M parameters and achieves $F_C\in\mathbb R^{N_C\times C\times H\times W},$9 in terms of mean Intersection over Union. EFFOcc also proposes a multi-stage occupancy-oriented distillation to efficiently transfer knowledge to vision-only OccNet, and reports that $F^C_{bev}\in\mathbb R^{H\times W\times C'}.$0 of the performance of a $F^C_{bev}\in\mathbb R^{H\times W\times C'}.$1 labeled vision OccNet can be achieved using the same OccNet trained with only $F^C_{bev}\in\mathbb R^{H\times W\times C'}.$2 labeled sequences and distillation from the fusion-based OccNet, corresponding to mIoU $F^C_{bev}\in\mathbb R^{H\times W\times C'}.$3 versus $F^C_{bev}\in\mathbb R^{H\times W\times C'}.$4 (Shi et al., 2024).

Taken together, these works show that fusion-based teachers are used for at least three distinct purposes in occupancy prediction: to provide stronger BEV features for active region distillation, to provide renderable occupancy fields for perspective consistency objectives, and to provide supervision under limited-label regimes. A common misconception is therefore to treat “distillation for occupancy” as a single standardized recipe. The reported literature instead indicates several non-equivalent design choices, each tied to different failure modes of image-only 3D occupancy prediction.