Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion

Abstract: In recent years, 4D imaging radar has gained wide attention in autonomous driving for its robustness against harsh weather and ability to output target velocity. Nevertheless, mainstream 4D radar-camera fusion methods only support front-view perception, lacking mature solutions for surround-view sensing. Directly expanding these pipelines to full 360° coverage introduces excessive computation cost and limits real-world deployment. To tackle these limitations, this work proposes Sparse4D-Radar, an efficient robust surround-view multi-modal fusion framework. We first design a Deformable Fusion module to embed radar-camera features into sparse queries, constructing the lightweight base version Sparse4D-Radar-Base. Two dedicated modules are further introduced to boost localization accuracy and modality stability: Velocity-Consistency Sampling (VCS) refines features via radar velocity cues for motion awareness, and Adaptive Modality Gating (AMG) dynamically adjusts cross-modal fusion weights according to feature confidence. Combining all components, we build Sparse4D-Radar-Acc for high-precision detection demands. Comprehensive experiments on OmniHD-Scenes verify that our approach achieves state-of-the-art surround-view 3D detection performance. Compared with prior arts, our method obtains over 7% mAP and 10% ODS improvements under complex driving scenes while running at nearly 10 FPS, striking a favorable trade-off among detection accuracy, environmental robustness and inference efficiency. Our open-source code is available at https://github.com/Aiuan/Sparse4D-Radar.

• The paper introduces a sparse query-based fusion paradigm that leverages 4D radar and cameras for surround-view 3D object detection.
• It employs Velocity-Consistency Sampling and Adaptive Modality Gating to refine features and ensure robust detection under adverse conditions.
• The framework achieves state-of-the-art metrics with reduced computational overhead, enabling real-time performance in autonomous driving applications.

Sparse4D-Radar: Efficient, Robust Surround-View 3D Object Detection via 4D Radar-Camera Fusion

Motivation and Context

Sparse4D-Radar addresses core limitations in current surround-view perception systems for autonomous driving. Existing radar-camera fusion schemes are typically constrained to front-view sensing and exhibit poor scalability when expanded to full 360° coverage due to dense BEV grid-based strategies, leading to prohibitive computational overhead. While 4D imaging radar possesses advantages in robustness, velocity measurement, and spatial fidelity, its stochastic sparsity remains challenging for object detection. Sparse4D-Radar introduces a sparse query-based paradigm, specifically tailored for efficient multi-modal interaction in surround-view scenarios, integrating both 4D radar and camera information.

Architectural Overview

The overall framework processes multi-view images and 4D radar point clouds using dedicated encoders, extracting multi-scale features which are further refined using cascaded decoder blocks. At the core lies the Deformable Fusion module, engineered for high-quality feature aggregation across modalities. The system further introduces two plug-and-play components:

• Velocity-Consistency Sampling (VCS): Enforces motion-aware feature refinement by leveraging explicit velocity measurements from 4D radar, enhancing spatial alignment and suppressing ghost points using dynamic velocity similarity scoring.
• Adaptive Modality Gating (AMG): Implements a gating mechanism to dynamically weight the contributions of camera and radar features based on real-time reliability, improving robustness in adverse conditions.

  Figure 1: The Sparse4D-Radar surround-view fusion architecture, highlighting the sparse query-based interaction, deformable fusion module, and the VCS/AMG components.

Deformable Fusion: Core Sparse Multimodal Aggregation

The Deformable Fusion module executes feature integration in four stages:

1. Keypoint Generation: For each anchor, a set of fixed and learned 3D keypoints is produced to guide spatial sampling.
2. Feature Sampling: Keypoints are projected onto PV and BEV spaces; multi-view/multi-scale feature maps are sampled via bilinear interpolation.
3. Hierarchy Fusion: Features across views, scales, and keypoints are adaptively weighted using learnable parameters dependent on instance features, anchor embedding, and modality-specific sensor encoding.
4. Multimodal Fusion: Aggregated camera and radar features are concatenated and linearly transformed, updating instance representations for downstream detection.

   Figure 2: The Deformable Fusion module, detailing keypoint generation, feature sampling, hierarchy fusion, and multimodal integration logic.

Specialized Modules: Velocity-Consistency and Adaptive Gating

VCS utilizes ball query-based neighborhood retrieval to compare anchor and radar point velocities, encoding similarity via a shared-weight velocity encoder. The resulting similarity scores recalibrate feature aggregation, favoring motion-consistent data and suppressing non-physical artifacts.

Figure 3: Velocity-Consistency Sampling module workflow, enhancing spatial and motion fidelity.

AMG applies a dynamically computed gating vector to weighted fusion of modalities. It relies on the reliability of modality-specific features, enabling adaptive reliance on robust geometric radar information during image degradation (e.g., rain, nighttime).

Figure 4: Adaptive Modality Gating module, facilitating runtime reliability-based fusion.

Experimental Benchmarking and Results

Evaluation was conducted on the OmniHD-Scenes dataset—a comprehensive multi-sensor benchmark for surround-view perception. Sparse4D-Radar achieves substantial improvements in all detection metrics over state-of-the-art fusion architectures. Notably, the Sparse4D-Radar-Base model delivers 47.01% mAP and 57.25% ODS at 11.5 FPS; Sparse4D-Radar-Acc achieves 47.57% mAP, 58.35% ODS, and 8.7 FPS. Both configurations surpass highly competitive radar-camera and LiDAR-based baselines, especially under adverse weather and nighttime conditions (ODS: 58.58% for Base, 58.61% for Acc).

Figure 5: Performance–efficiency trade-off across methods; Sparse4D-Radar demonstrates leading accuracy at real-time speeds.

Sparse4D-Radar demonstrates particularly strong velocity estimation, achieving significantly lower mAVE compared to all prior work, attributed to optimal handling of dynamic cues in the sparse fusion architecture.

Computational efficiency is a highlight, with FLOPs and parameter counts substantially reduced compared to dense BEV methods (Sparse4D-Radar-Base: 472.06G FLOPs, 53.47M params), rendering the system highly suitable for on-board real-time deployment.

Ablation Studies

Systematic ablation reveals synergistic gains from VCS and AMG modules. VCS improves ODS and reduces orientation/velocity errors; AMG further contributes to robustness. PointPillars is selected as the radar backbone, offering optimal accuracy-speed trade-offs. Feature sampling strategies and fusion mechanisms are validated, with AMG and VCS outperforming attention-based baselines.

Qualitative Evaluation

Visualization across environmental scenarios confirms superior localization and fewer false positives/missed detections, notably under challenging nighttime and rain conditions. Sparse4D-Radar leverages multimodal synergy to maintain high-fidelity perception.

Figure 6: 3D bounding box projections and BEV analysis across multiple conditions, showing enhanced robustness and accuracy.

Implications and Future Directions

Sparse4D-Radar demonstrates that sparse query-based fusion, augmented by motion-consistency and adaptive gating modules, can significantly elevate the accuracy and robustness of surround-view 3D object detection using radar-camera configurations. The reduced reliance on dense BEV grids, coupled with hardware-efficient design, enables practical, real-time deployment in resource-constrained autonomous platforms.

Theoretically, this architecture advances the state-of-the-art in multi-modal interaction—optimally leveraging spatial and physical cues (velocity, robustness) unique to 4D radar. Practically, Sparse4D-Radar bridges performance gaps with LiDAR-based systems, offering scalable alternatives with comparable accuracy. Limitations in radar feature extraction and computational trade-offs suggest future work in more specialized radar encoding architectures and further reduction of module overhead.

Conclusion

Sparse4D-Radar establishes a new benchmark for surround-view radar-camera fusion, achieving state-of-the-art detection accuracy, environmental robustness, and real-time efficiency. Its architectural developments—including deformable fusion, velocity-consistency refinement, and adaptive modality gating—redefine the landscape of efficient 3D perception in autonomous driving, with implications for broader multimodal sensing applications and theoretical exploration of sparse fusion paradigms.