GaussianFusionOcc: 3D Sensor Fusion

Updated 7 July 2026

The paper presents GaussianFusionOcc, a novel 3D semantic occupancy prediction framework that uses sparse, learnable 3D Gaussians to represent driving scenes.
It fuses camera, LiDAR, and radar features through a modality-agnostic deformable attention mechanism, converting sparse Gaussian insights into a dense occupancy grid.
Experiments on the nuScenes dataset demonstrate improved IoU and mIoU metrics with lower computational cost, validating its efficiency and robustness in adverse conditions.

GaussianFusionOcc is a 3D semantic occupancy prediction framework for autonomous driving that represents a scene with a sparse set of learnable 3D Gaussians and fuses camera, LiDAR, and radar features through a modality-agnostic deformable attention mechanism. Introduced in “GaussianFusionOcc: A Seamless Sensor Fusion Approach for 3D Occupancy Prediction Using 3D Gaussians” (Pavković et al., 24 Jul 2025), it departs from dense voxel-centric fusion by using semantic Gaussians as the intermediate carriers of geometry and semantics, while still producing a dense voxelized semantic occupancy grid through Gaussian splatting. The framework is formulated for the nuScenes occupancy setting and is designed to support camera-only, LiDAR-only, radar-only, and fused sensor configurations within a single architecture.

1. Problem formulation and design rationale

GaussianFusionOcc addresses 3D semantic occupancy prediction, in which a model estimates, for each voxel in a 3D volume around the ego vehicle, whether the voxel is occupied and by which semantic class. The paper adopts the nuScenes setup with a prediction range of $[-50, 50]$ m in $x$ , $[-50, 50]$ m in $y$ , and $[-5, 3]$ m in $z$ , discretized at $0.5$ m (Pavković et al., 24 Jul 2025). The target output is therefore a dense semantic 3D map suitable for situation awareness, free-space reasoning, and navigation.

The motivation is explicitly framed against dense grid-based occupancy methods. Conventional approaches commonly use dense 3D voxel grids, for which memory and computation scale as $O(N_x N_y N_z)$ even though most voxels are empty. The paper further notes that recent BEV and TPV methods alleviate some of the 3D cost by compressing into 2D planes, but lose vertical structure that is critical for full 3D occupancy prediction (Pavković et al., 24 Jul 2025). GaussianFusionOcc instead exploits the observation that driving scenes are sparse and structured, with most relevant geometry concentrated around surfaces and object boundaries.

Within this formulation, 3D Gaussians are used as adaptive volumetric primitives. The representation is intended to be object- or region-centric rather than cell-centric: more Gaussians can be allocated where scene complexity is high, and fewer are needed elsewhere. This makes the Gaussian set independent of the voxel resolution used for the final occupancy output. The multi-modal emphasis is also central. The framework is built to combine dense RGB appearance, LiDAR geometry, and radar robustness, with the paper specifically arguing that LiDAR contributes range geometry and radar contributes robustness and velocity-related cues under adverse conditions (Pavković et al., 24 Jul 2025).

2. Gaussian representation and iterative architecture

The scene is represented by a set of semantic Gaussians

$\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$

where $m_i \in \mathbb{R}^3$ is the Gaussian mean, $x$ 0 the scale, $x$ 1 the rotation, $x$ 2 the opacity, and $x$ 3 the semantic logits vector (Pavković et al., 24 Jul 2025). Conceptually, each Gaussian is an anisotropic volumetric primitive whose covariance is formed from the scale and rotation. The details summarize the underlying Gaussian as

$x$ 4

with covariance $x$ 5, which makes the representation continuous in 3D space rather than tied to a fixed voxel lattice (Pavković et al., 24 Jul 2025).

Initialization is flexible. The paper studies random positions, fully learnable Gaussian parameters, and variants in which Gaussian means are derived from LiDAR points. In the reported ablations, typical Gaussian counts are $x$ 6 and $x$ 7 (Pavković et al., 24 Jul 2025). This fixed set of Gaussians is then refined over a stack of four GaussianFusion blocks. Each block receives the current Gaussian parameters and their associated queries, extracts per-Gaussian features from available sensor streams, fuses those features, and predicts refined Gaussian properties for the next stage.

After each block, and again at the final stage, the refined Gaussians are splatted to a dense voxel grid using the GaussianFormer-2 superposition module: $x$ 8 where $x$ 9 denotes a voxel center and $[-50, 50]$ 0 denotes voxel semantic logits or occupancy (Pavković et al., 24 Jul 2025). This stagewise splatting enables deep supervision. The architecture therefore alternates between sparse Gaussian-space reasoning and dense voxel-space supervision, rather than carrying dense 3D tensors throughout the entire network.

3. Modality-agnostic sensor fusion mechanism

GaussianFusionOcc uses separate sensor backbones but a shared encoder design for extracting per-Gaussian features. The camera pathway uses a ResNet101-DCN backbone initialized from FCOS3D, followed by an FPN that outputs multi-scale image features. The LiDAR pathway uses voxelization and a VoxelNet encoder with FPN, initialized from FUTR3D. The radar pathway uses pillarization, followed by PointPillars voxel and middle encoders, also initialized from FUTR3D (Pavković et al., 24 Jul 2025). The architectural distinction is therefore in the feature maps, not in the Gaussian encoder itself.

The core mechanism is the modality-agnostic Gaussian encoder with deformable attention. For a Gaussian with mean $[-50, 50]$ 1, scale matrix $[-50, 50]$ 2, rotation matrix $[-50, 50]$ 3, and query embedding $[-50, 50]$ 4, the model predicts offsets in the Gaussian’s local coordinates: $[-50, 50]$ 5 These 3D reference points are projected into the appropriate sensor feature coordinates through calibration-aware projection $[-50, 50]$ 6. The resulting per-Gaussian feature for one modality is defined as

$[-50, 50]$ 7

where $[-50, 50]$ 8 is deformable attention, $[-50, 50]$ 9 is the Gaussian query, $y$ 0 the projected reference positions, and $y$ 1 the corresponding feature maps (Pavković et al., 24 Jul 2025). For cameras, the projection is perspective projection into image and FPN feature space; for LiDAR and radar, it is into BEV feature maps.

The phrase “modality-agnostic” is used in a strict architectural sense: the same Gaussian encoder design is applied to camera, LiDAR, and radar features. Sensor combinations are therefore handled by choosing which modality-specific encoder outputs are present. The paper reports experiments for $y$ 2, $y$ 3, $y$ 4, $y$ 5, $y$ 6, and $y$ 7 settings (Pavković et al., 24 Jul 2025).

Once per-modality Gaussian features are obtained, the fusion module concatenates them, applies an MLP, and combines the result with sparse 3D convolutional context derived from voxelized Gaussian means: $y$ 8 where $y$ 9 denotes element-wise addition (Pavković et al., 24 Jul 2025). This unified per-Gaussian feature then drives parameter refinement: $[-5, 3]$ 0 The Gaussian set is updated block by block, so positions can move toward stronger geometric evidence, scales and orientations can adapt to local structure, opacities can be reweighted, and semantic logits can become more discriminative.

4. Supervision, optimization, and operating regime

Training is supervised by dense semantic occupancy labels from SurroundOcc for nuScenes. At each refinement stage $[-5, 3]$ 1, the current Gaussian set is splatted to the voxel grid, yielding stagewise occupancy predictions $[-5, 3]$ 2. The total loss is the sum over stages of Lovász-Softmax and binary cross-entropy terms: $[-5, 3]$ 3 with $[-5, 3]$ 4 GaussianFusion blocks in the reported configuration (Pavković et al., 24 Jul 2025). Lovász-Softmax is used as a semantic occupancy loss, while BCE is used for occupied-versus-empty supervision.

The reported evaluation metrics are occupancy IoU and semantic mIoU: $[-5, 3]$ 5

$[-5, 3]$ 6

where $[-5, 3]$ 7 denotes the empty class and $[-5, 3]$ 8 the set of non-empty semantic classes (Pavković et al., 24 Jul 2025). This separates class-agnostic occupancy quality from semantic labeling quality.

Implementation details are conventional for large-scale occupancy learning. Training uses batch size $[-5, 3]$ 9, $z$ 0 epochs, AdamW with weight decay $z$ 1, and a learning rate warmed up to $z$ 2 over $z$ 3 iterations followed by cosine decay (Pavković et al., 24 Jul 2025). The default Gaussian encoder and fusion feature dimension is $z$ 4 channels, and the experiments are conducted on an NVIDIA RTX A6000. The paper further states that all Gaussians and splatting operate in the global or ego-centric 3D coordinate frame used by nuScenes, with sensor projection defined by the dataset’s camera, LiDAR, and radar intrinsics and extrinsics (Pavković et al., 24 Jul 2025).

A notable operational property is the separation between sparse and dense computation. Gaussian-level reasoning, deformable attention, and sparse convolution all occur over the Gaussian set, whereas dense computation is deferred to the final voxel output required for loss and evaluation. This division is the basis for the method’s efficiency claims.

5. Quantitative behavior, robustness, and efficiency

The main nuScenes validation results reported for GaussianFusionOcc are summarized below (Pavković et al., 24 Jul 2025).

Configuration	IoU	mIoU
C	37.05	22.43
C+R	37.37	22.80
L	45.32	29.75
C+L	45.16	30.21
C+L+R	45.20	30.37

The best reported GaussianFusionOcc variant is the $z$ 5 model with learnable initialization, which achieves IoU $z$ 6 and mIoU $z$ 7 on nuScenes validation (Pavković et al., 24 Jul 2025). In the same comparison, OccFusion with $z$ 8 is reported at IoU $z$ 9 and mIoU $0.5$0. The LiDAR-only configuration is especially notable: GaussianFusionOcc with random initialization reports IoU $0.5$1 and mIoU $0.5$2, whereas L-CONet is reported at IoU $0.5$3 and mIoU $0.5$4 (Pavković et al., 24 Jul 2025).

The adverse-condition subsets follow the same pattern. On the rainy subset, GaussianFusionOcc with $0.5$5 reports IoU $0.5$6 and mIoU $0.5$7, compared with OccFusion $0.5$8 at IoU $0.5$9 and mIoU $O(N_x N_y N_z)$ 0. On the night subset, GaussianFusionOcc $O(N_x N_y N_z)$ 1 reports IoU $O(N_x N_y N_z)$ 2 and mIoU $O(N_x N_y N_z)$ 3, compared with OccFusion $O(N_x N_y N_z)$ 4 at IoU $O(N_x N_y N_z)$ 5 and mIoU $O(N_x N_y N_z)$ 6 (Pavković et al., 24 Jul 2025). The paper interprets these results as evidence that LiDAR provides the dominant gain over camera-only prediction, while radar contributes smaller but consistent improvements when added to camera and LiDAR.

The efficiency analysis is equally central to the method’s positioning. For the $O(N_x N_y N_z)$ 7 configuration, GaussianFusionOcc is reported with $O(N_x N_y N_z)$ 8M parameters, $O(N_x N_y N_z)$ 9 GB memory, and $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 0 ms latency. OccFusion $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 1 is reported with $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 2M parameters, $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 3 GB memory, and approximately $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 4 ms latency. The LiDAR-only GaussianFusionOcc model uses $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 5M parameters, $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 6 GB memory, and $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 7 ms latency (Pavković et al., 24 Jul 2025). The ablation on Gaussian count further shows a tradeoff rather than a regime change: increasing from $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 8 to $\mathcal{G} = \{m_i, s_i, r_i, a_i, c_i\}_{i=1}^P,$ 9 Gaussians for $m_i \in \mathbb{R}^3$ 0 improves IoU from $m_i \in \mathbb{R}^3$ 1 to $m_i \in \mathbb{R}^3$ 2 and mIoU from $m_i \in \mathbb{R}^3$ 3 to $m_i \in \mathbb{R}^3$ 4, with negligible memory increase but latency growth from $m_i \in \mathbb{R}^3$ 5 to $m_i \in \mathbb{R}^3$ 6 ms (Pavković et al., 24 Jul 2025).

Ablations also indicate that increasing channel width from $m_i \in \mathbb{R}^3$ 7 to $m_i \in \mathbb{R}^3$ 8 produces only modest accuracy gains, from IoU $m_i \in \mathbb{R}^3$ 9 and mIoU $x$ 00 to IoU $x$ 01 and mIoU $x$ 02, while increasing parameters from $x$ 03M to $x$ 04M and memory from $x$ 05 GB to $x$ 06 GB (Pavković et al., 24 Jul 2025). The initialization study shows random and learnable Gaussian initialization are both competitive, whereas the probabilistic initialization inherited from GaussianFormer-2 is reported as significantly slower (Pavković et al., 24 Jul 2025).

The name GaussianFusionOcc refers directly to the 2025 framework that fuses camera, LiDAR, and radar into a sparse semantic Gaussian representation for 3D occupancy prediction (Pavković et al., 24 Jul 2025). A related terminological complication arises because the later paper “GaussianFusion: Unified 3D Gaussian Representation for Multi-Modal Fusion Perception” describes “GaussianFusionOcc” as the semantic occupancy instantiation of its task-agnostic GaussianFusion backbone, namely GaussianFusion with a 3D semantic occupancy head (Zhao et al., 1 Jul 2026). In that usage, the term denotes an occupancy variant of a broader Gaussian fusion framework rather than the specific 2025 method name.

This ambiguity reflects a broader convergence in Gaussian-based occupancy research. GaussianFusion (Zhao et al., 1 Jul 2026) replaces the discrete BEV grid with a continuous set of 3D Gaussians shared across cameras and LiDAR, uses forward-projection-based camera Gaussian initialization, a shared cross-modal Gaussian encoder, and Gaussian-to-voxel pooling, and describes the occupancy branch as one task-specific head on top of a unified Gaussian backbone. Gau-Occ, by contrast, models the scene as a compact collection of semantic 3D Gaussians, initializes anchors from a LiDAR Completion Diffuser, and uses Gaussian Anchor Fusion with geometry-aligned 2D sampling and cross-modal alignment for semantic occupancy prediction (Lv et al., 24 Mar 2026). GaussianFusionOcc differs from both in its explicit inclusion of radar and in its use of modality-agnostic deformable attention across camera, LiDAR, and radar streams (Pavković et al., 24 Jul 2025).

The limitations reported for GaussianFusionOcc are structural rather than incidental. The method uses a fixed number $x$ 07 of Gaussians, which means sparse scenes may underuse the Gaussian budget while complex scenes may exceed it. Radar features remain sparse, and the paper characterizes radar’s contribution to dense occupancy as modest. The formulation is also per-frame and does not explicitly model motion, so dynamic temporal reasoning is not part of the reported system (Pavković et al., 24 Jul 2025). Proposed extensions in the paper include dynamic Gaussian allocation, improved exploitation of radar information, temporal modeling through sequential or 4D Gaussians, and alternative Gaussian parameterizations (Pavković et al., 24 Jul 2025).

Taken together, these works suggest a broader shift from dense voxel or BEV tensors toward Gaussian carriers as the latent space for multi-modal occupancy reasoning. Within that landscape, GaussianFusionOcc occupies a specific position: it is a sensor-fusion occupancy framework in which semantic 3D Gaussians serve simultaneously as the representation of scene structure, the locus of cross-modal attention, and the precursor to dense voxelized occupancy output (Pavković et al., 24 Jul 2025).