CLLAP: Contrastive Learning-based LiDAR-Augmented Pretraining for Enhanced Radar-Camera Fusion

Abstract: Accurate 3D object detection is critical for autonomous driving, necessitating reliable, cost-effective sensors capable of operating in adverse weather conditions. Camera and millimeter-wave radar fusion has emerged as a promising solution; however, these methods often rely on finely annotated radar data, which is scarce and labor-intensive to produce. To address this challenge, we present CLLAP, a Contrastive Learning-based LiDAR-Augmented Pretraining framework that enhances the performance of existing radar-camera fusion methods for 3D object detection. CLLAP leverages abundant LiDAR data to generate pseudo-radar data using the proposed L2R (LiDAR-to-Radar) Sampling method. Then, it incorporates this data into a novel dual-stage, dual-modality contrastive learning strategy, enabling effective self-supervised learning from paired pseudo-radar and image data. This approach facilitates effective pretraining of existing radar-camera fusion models in a plug-and-play manner, enhancing their feature extraction capabilities and improving detection accuracy and robustness. Experimental results using NuScenes and Lyft Level 5 datasets demonstrate significant performance improvements across three baseline models, highlighting CLLAP's effectiveness in advancing radar-camera fusion for autonomous driving applications.

• The paper presents CLLAP, a novel method that generates pseudo-radar measurements from unlabeled LiDAR data to enable self-supervised pretraining for 3D object detection.
• It employs a dual-stage, dual-modality contrastive learning approach combining local and global losses to robustly align features between radar and camera modalities.
• Experimental evaluations on NuScenes and Lyft Level 5 datasets show significant improvements in mAP and NDS, confirming enhanced robustness even under adverse weather conditions.

Contrastive Learning-based LiDAR-Augmented Pretraining (CLLAP) for Enhanced Radar-Camera Fusion

Motivation and Background

Reliable 3D object detection is fundamental to autonomous driving systems, requiring robust sensor fusion strategies that function effectively under adverse and variable environmental conditions. Camera-radar fusion has gained traction due to radar's resilience and affordability compared to LiDAR. However, the efficacy of radar-camera fusion models is impeded by the scarcity of finely annotated radar datasets, which significantly limits their performance, scalability, and adaptability. Existing knowledge distillation techniques rely on paired labeled radar-LiDAR data, but such corpora are rare and laborious to construct.

Addressing these constraints, the CLLAP framework introduces a novel approach leveraging abundant unlabeled LiDAR corpora. By converting LiDAR data into pseudo-radar measurements and employing a dual-stage, dual-modality contrastive learning regimen, CLLAP provides a self-supervised, plug-and-play pretraining paradigm that enhances downstream radar-camera fusion for 3D object detection.

Figure 1: Existing fusion models rely solely on scarce radar-image data (a); CLLAP generates pseudo-radar data from LiDAR and uses dual-stage, dual-modality contrastive learning for robust pretraining (b).

Methodology

LiDAR-to-Radar (L2R) Sampling Module

CLLAP's core is the L2R Sampling module, which generates pseudo-radar point clouds from unlabeled LiDAR datasets without neural translation layers. The process incorporates: Gaussian Mixture Model density fitting, KD-tree redundancy reduction, weighted two-stage spatial sampling (center vs. periphery, with weights for intensity, sparsity, and distance), FlowNet3D-based velocity augmentation, and radar-plane mapping for altitude insensitivity. This procedure replicates radar's spatial and motion distribution patterns, overcoming LiDAR-radar modality gaps and enhancing training diversity.

Figure 2: Pipeline of L2R Sampling for generating pseudo-radar point clouds from LiDAR data.

Figure 3: Visualization comparing original LiDAR, ground-truth radar, L2R-generated pseudo-radar, and distance-weighted sampling.

Dual-Stage Dual-Modality Contrastive Learning

The pretraining procedure consists of two main stages:

• Stage 1: Feature encoding pretraining using paired pseudo-radar and camera images.
• Stage 2: Secondary pretraining with real radar data, adapting to practical radar distribution.

CLLAP employs both local and global contrastive loss components:

Local Contrastive Loss

Fine-grained feature alignment is enforced between radar and camera modalities in BEV format. Column-wise feature correspondence is established via sliding window mechanisms and bidirectional channel-spatial attention (BCSA), promoting robust local feature discrimination.

Figure 4: The architecture of the BCSA module for channel-spatial decoupling and gating.

Global Contrastive Loss

Multi-view and multi-modality features are aggregated using shared attention, enabling cross-modality and cross-view global invariance. Six sets of contrastive pairs are formed covering BEV and front views for both radar and image modalities. The overall contrastive objective combines global and column-wise terms with empirically optimal weighting.

Figure 5: Contrastive learning framework for integrating features from multiple modalities and views.

Figure 6: Visualizations of point cloud and image features from various modalities and viewpoints highlight semantic alignment.

Experimental Results

Performance Evaluation

CLLAP was benchmarked on NuScenes and Lyft Level 5 datasets, using CRN, RCBEVDet, and BEVFusion backbones. Standard metrics include mean Average Precision (mAP) and NuScenes Detection Score (NDS). CLLAP consistently improved all backbone models, e.g., +3.23 mAP / +2.28 NDS for CRN on NuScenes. The method also demonstrated superior robustness under adverse weather (rain, snow), showing clear gains over baseline radar-camera fusion models.

Figure 7: Visualization illustrating improved localization accuracy using CLLAP compared with CRN baseline.

Figure 8: Example scenario under adverse weather conditions.

Ablations and Component Analysis

• Both local and global contrastive loss components are indispensible, each contributing complementary improvements; their combination further increases model performance.
• L2R sampling outperforms other learned pseudo-radar generation baselines (SampleNet, APES) in terms of spatial fidelity and downstream mAP/NDS gains despite not achieving minimal Chamfer Distance.
• Pretraining with LiDAR data from multiple datasets (Lyft and NuScenes) yields additive benefits, demonstrating generalization and volume scalability.
• BCSA module outperforms generic cross-attention strategies; structured channel and spatial attention, along with gated bi-directional fusion, are key for effective cross-modal feature alignment.
• Optimal global-to-local contrastive loss weighting parameter ($\lambda = 1/6$) maximizes detection performance.

  Figure 9: Heatmaps show feature consistency before and after application of Global Contrastive Loss; object boundaries become more distinct.

Practical and Theoretical Implications

CLLAP establishes a formal methodology for exploiting unlabeled LiDAR corpora to breach the radar data scarcity bottleneck, enabling effective self-supervised pretraining for radar-camera fusion. The plug-and-play nature of the framework allows seamless integration with heterogeneous backbone architectures. Its superior robustness under adverse weather and dataset diversity implies high potential for transfer learning, domain adaptation, and deployment in real-world autonomous driving and intelligent transportation systems.

Future directions include further exploration of pseudo-radar generation utilizing richer LiDAR semantic cues, extension to 4D radar and additional modalities, and refinement of cross-modal contrastive losses to improve scene-level understanding and contextual reasoning in multi-agent settings.

Figure 10: Visualization comparison of pseudo-radar spatial distribution fidelity across various sampling strategies.

Conclusion

CLLAP pioneers a contrastive learning-based pretraining paradigm utilizing LiDAR-generated pseudo-radar data, advancing radar-camera fusion for 3D object detection. Its dual-stage, dual-modality loss structure, combined with principled pseudo-radar synthesis, consistently enhances baseline detection models' accuracy and robustness, while circumventing the need for large-scale annotated radar corpora. This formal strategy opens avenues for resource-efficient, strongly robust multi-modal perception pipelines in autonomous systems.

(2604.24044)