- The paper presents a unified continuous 3D Gaussian representation to overcome the quantization errors and inefficiencies of discrete BEV-based fusion.
- It integrates a shared Gaussian encoder with deformable attention to refine feature alignment and optimize parameter updates across modalities.
- Empirical results on nuScenes and Waymo datasets demonstrate superior 3D object detection, efficiency, and reduced memory footprint compared to state-of-the-art methods.
GaussianFusion: Unified 3D Gaussian Representation for Multi-Modal Fusion Perception
Motivation and Context
The standard BEV (bird's-eye view) framework for multi-modal 3D perception leverages a grid-based, discrete spatial representation to enable the fusion of heterogeneous sensor data, most noticeably camera and LiDAR. While effective for sensor alignment and supporting multiple downstream tasks, the discrete nature of BEV induces quantization errors, loss of geometric and textural detail, and suboptimal cross-modal feature integration. Increasing BEV grid resolution to mitigate these limitations incurs significant computational overhead and memory inefficiency, further limiting the practicality of this approach in real-time or resource-constrained 3D perception systems.
GaussianFusion introduces an alternative paradigm, constructing a unified, continuous 3D space by representing sensory data as parameterized 3D Gaussian distributions. This avoids the information loss associated with early discretization and provides an intrinsically geometric, uncertainty-aware mechanism for fusing multi-modal inputs at higher fidelity.
Methodology
GaussianFusion's architecture comprises three principal components: (a) modality-specific 3D Gaussian initialization leveraging sensor geometry, (b) a shared cross-modal Gaussian encoder with iterative refinement via deformable attention, and (c) a Gaussian-to-voxel fusion mechanism for broad, task-agnostic applicability. The pipeline is fundamentally differentiable and supports both camera-LiDAR and camera-only configurations.
3D Gaussian Initialization
Distinct parameter initialization is used for each sensor:
- Camera branch: Employs forward projection to initialize 3D Gaussian means using statistically grounded depth estimates (LSS) derived from calibrated camera rays, with scales and rotations initialized randomly. Unlike prior methods (e.g., GaussianFormer (Huang et al., 2024)) which randomly initialize Gaussians, this model ensures geometric consistency and accelerates convergence.
- LiDAR branch: Utilizes the structured BEV grid of LiDAR point clouds to place Gaussian means, while scales and orientations remain learnable.
Both branches associate learned feature queries with each initialized Gaussian, effectively localizing semantic and geometric information.
The core innovation lies in a shared encoder architecture where camera and LiDAR Gaussians are stacked and processed together. This encoder:
- Augments feature queries with embeddings of the Gaussian parameters (mean, scale, rotation) via an MLP.
- Applies a custom deformable attention mechanism where initial sampling points are generated from the Gaussian's covariance rather than a regular grid, imposing an object-shape-adaptive prior.
- Refines Gaussian parameters incrementally by predicting and applying property offsets rather than full reinitialization, increasing efficiency, robustness to misalignment, and preserving continuous geometric structure throughout the fusion process.
Gaussian-to-Voxel Fusion
Fused Gaussian representations are pooled into a voxel grid via Gaussian-based aggregation (mean VFE [VoxelNet]), effectively supporting task-agnostic 3D perception. The final 3D scene features are produced as Gaussian-weighted mixtures, exploiting each Gaussian's uncertainty encoding to facilitate robust, detail-preserving feature aggregation for both object-centric and dense semantic tasks.
Empirical Results
GaussianFusion demonstrates consistent SOTA results across primary 3D perception benchmarks:
- 3D object detection (nuScenes, val): Achieves 74.0 NDS, 71.7 mAP, outperforming BEVFusion by +2.6 NDS and +3.2 mAP, and surpassing transformers such as UniTR and FusionFormer-S.
- 3D semantic occupancy (nuScenes, val): Multi-modal variant attains 28.65 mIoU, exceeding OccFusion and other SOTA fusion frameworks. The camera-only GaussianFusion-C delivers a +1.55 mIoU gain and 4.5x speedup relative to GaussianFormer, with only 30% the number of Gaussians (43k vs. 140k).
- Efficiency: Lower inference latency (132 ms) and memory footprint (4271 MB) relative to BEVFusion.
- Generality: SOTA performance generalizes to Waymo Open Dataset object detection, further confirming the robustness of the unified Gaussian representation.
These empirical results definitively support the core claim: continuous 3D Gaussian representations offer superior cross-modal alignment, structural fidelity, and computational efficiency compared to prior discrete, grid-based fusion systems.
Ablative Analysis
Ablation studies quantify the contribution of each architectural feature:
- Gaussian initialization: Forward projection initialization provides +2.8 NDS over random; lidar projection achieves near-parity, backward projection is suboptimal.
- Gaussian encoder: Shared cross-modal encoding improves mAP by +0.7 over separate branch-specific encoding. Deformable attention with Gaussian priors further yields +0.4 NDS and improved convergence.
- Parameter updates: Offset-based parameter refinement outperforms direct parameter prediction (+0.9 mAP).
- Downstream utility: The unified architecture is readily adaptable for temporal fusion (GaussianFusion-T), yielding NDS and mAP competitive with more complex temporal fusion baselines.
Theoretical and Practical Implications
The adoption of continuous 3D Gaussian mixture models in place of discrete BEV or voxel grids establishes a new foundational representation for multi-modal fusion in 3D perception. This paradigm:
- Enhances the retention of scene granularity and geometric details.
- Provides native support for uncertainty, indispensable for real-world autonomous driving scenarios with ambiguous sensor returns.
- Unifies cross-view and cross-modal fusion in a manner agnostic to task, sensor, or backbone, enabling flexible model extension and facilitating future application to fully end-to-end perception stacks.
- Reduces memory and compute requirements for high-resolution perception, thus improving deployability in embedded platforms.
On the theoretical front, this work bridges implicit radiance-field modeling (e.g., NeRFs, 3DGS) and explicit perception, demonstrating the viability of deep Gaussian mixtures as both an intermediate and an end-task representation for downstream tasks requiring semantic reasoning and object-centric inference.
Limitations and Future Directions
While GaussianFusion achieves strong single-frame performance and modest gains in simple temporal extensions, the absence of sophisticated motion-aware Gaussian updating limits performance in challenging dynamic, multi-frame regimes. Integrating learned velocity priors or motion-adaptive parameterization into the iterative refinement mechanism constitutes a promising direction for simultaneously modeling geometric-temporal coherence and cross-modal alignment in 4D semantic scene understanding.
Conclusion
GaussianFusion establishes a technically rigorous, effective, and extensible framework for multi-modal 3D perception by unifying sensor features within a continuous Gaussian representation space. The approach demonstrates quantifiable improvements in both accuracy and efficiency across multiple 3D vision tasks. Importantly, it points toward a new standard for sensor fusion—grounded in continuous, uncertainty-aware, and detail-preserving spatial representations—opening avenues for further research on generalizable perception architectures and domain-agnostic multi-sensor fusion.