Leveraging Previous-Traversal Point Cloud Map Priors for Camera-Based 3D Object Detection and Tracking

Abstract: Camera-based 3D object detection and tracking are central to autonomous driving, yet precise 3D object localization remains fundamentally constrained by depth ambiguity when no expensive, depth-rich online LiDAR is available at inference. In many deployments, however, vehicles repeatedly traverse the same environments, making static point cloud maps from prior traversals a practical source of geometric priors. We propose DualViewMapDet, a camera-only inference framework that retrieves such map priors online and leverages them to mitigate the absence of a LiDAR sensor during deployment. The key idea is a dual-space camera-map fusion strategy that avoids one-sided view conversion. Specifically, we (i) project the map into perspective view (PV) and encode multi-channel geometric cues to enrich image features and support BEV lifting, and (ii) encode the map directly in bird's-eye view (BEV) with a sparse voxel backbone and fuse it with lifted camera features in a shared metric space. Extensive evaluations on nuScenes and Argoverse 2 demonstrate consistent improvements over strong camera-only baselines, with particularly strong gains in object localization. Ablations further validate the contributions of PV/BEV fusion and prior-map coverage. We make the code and pre-trained models available at https://dualviewmapdet.cs.uni-freiburg.de .

• The paper introduces DualViewMapDet, which fuses offline LiDAR map priors with multi-view camera features to mitigate depth ambiguity.
• It employs dual-space fusion using perspective view and bird’s-eye view encodings combined with deformable attention for robust 3D localization.
• Experimental results on nuScenes and Argoverse 2 demonstrate significant improvements in detection accuracy, localization precision, and tracking performance.

Dual-Space Fusion of Prior Point Cloud Maps for Camera-Only 3D Object Detection and Tracking

Introduction

The paper "Leveraging Previous-Traversal Point Cloud Map Priors for Camera-Based 3D Object Detection and Tracking" (2604.25405) introduces DualViewMapDet, a framework designed to mitigate depth ambiguity in camera-only 3D object detection and multi-object tracking. In the context of autonomous driving, typical approaches using multi-view cameras lack precise metric depth, which fundamentally constrains accurate object localization and tracking compared to methods that rely on online LiDAR. However, repeated traversals in real driving scenarios allow the offline construction of high-resolution static LiDAR point cloud maps, creating an opportunity to inject strong geometric priors into image-based perception models—substantially narrowing the performance gap without LiDAR at test time.

Methodology

Camera-Map Dual-Space Fusion

The primary technical advancement is a dual-space fusion mechanism (Figure 1) combining two complementary perspectives:

• Perspective View (PV) Fusion: The prior static map, constructed offline, is projected into each calibrated camera to produce sparse geometric PV cues. This includes normalized depth, ego-frame coordinates, depth positional embeddings, validity masks, spread-nearest valid depth, and distance-to-validity cues. These are concatenated with per-camera image features, producing depth-aware PV features before lifting to BEV.
• Bird’s-Eye View (BEV) Fusion: The static map is also voxelized and encoded directly in BEV. After image-map PV fusion and BEV lifting, the resulting camera-derived BEV features are concatenated with BEV map features. Additional refinement is applied through a lightweight BEV encoder to enable robust, spatially-aligned fusion in the metric space.

  Figure 1: Overview of the camera-map fusion framework combining PV and BEV encodings of multi-view camera images and static map priors for dual-space feature fusion.

Detection Head and Data Flow

A transformer-based detection head operates with sparse, object-centric queries. Deformable attention aggregates context first from the fused BEV then from the fused PV features, maximizing utilization of metric geometry for localization and PV semantics for recognition.

Crucially, map features do not contain current dynamic object information; they encode only the static background. The architectural choice to fuse in both PV and BEV, followed by sequential attention, avoids the representational bottlenecks of PV-only (limited geometric aggregation) and BEV-only (semantic attenuation) strategies. It also resolves view conversion issues, leveraging complementary strengths of each space.

Figure 3: DualViewMapDet employs multi-channel PV and sparse voxel BEV encoders with sequential fusion and deformable aggregation for robust 3D detection.

Map Construction and Robustness

The static map is constructed from aggregated LiDAR readings with dynamic points explicitly filtered out. During inference, the map is discretized into spatial tiles, patched around the current ego pose, and filtered to remove artifacts or possible 'leakage' from adjacent traversals.

To counteract dependency on perfect map priors, the authors apply grid masking augmentation to both image and map channels during training. This increases robustness to missing or incomplete priors, a common real-world failure mode.

Training and Inference Regimes

DualViewMapDet is trained end-to-end with detection and image-based depth supervision, but at inference, it requires only multi-view RGB, localization, calibration, and the retrieved local map. No online LiDAR is utilized.

Experimental Results

The framework is evaluated on nuScenes and Argoverse 2 and benchmarked against top camera-only models, including Sparse4Dv3, StreamPETR, Far3D, and BEVFormer. Strong quantitative gains are demonstrated across multiple metrics and both short-range (50 m) and long-range (150 m) evaluation regimes.

On nuScenes, DualViewMapDet achieves higher NDS (0.588 vs. 0.561 / 0.553 for variants of Sparse4Dv3) and mAP (0.481 vs. 0.469 / 0.447), with notable improvements in localization (mATE drops from 0.553 to 0.484). Tracking performance also improves, raising AMOTA to 0.513.

On Argoverse 2, improvements are similarly pronounced, especially in subsets with dense map prior coverage. For example, in the 50 m regime, DualViewMapDet reaches a CDS of 0.311 (vs. 0.294), mAP of 0.397 (vs. 0.381), and mATE of 0.622 (vs. 0.706). In both full and high-overlap subsets, the margin increases with better prior coverage, underpinning the central claim that geometric anchors from the static map reduce depth mislocalization of dynamic objects.

Figure 2: Map priors provide stable geometric anchors that significantly reduce depth distortion of car detections relative to image-only baselines.

Qualitative examples (Figure 4) illustrate improved 3D box tightness, especially under severe occlusion or at long distances.

Figure 4: DualViewMapDet yields accurate 3D box alignments and tracking, with orange boxes from DualViewMapDet closely following ground truth compared to competitive baselines.

Ablation Study and Analysis

The ablation analysis confirms that:

• PV fusion alone provides limited gains unless the PV map encoding is explicitly enhanced with local coordinate, spread, and embedding features.
• BEV-only fusion substantially aids metric localization, but the joint dual-space approach yields the greatest improvements.
• The dual fusion design is necessary not only for strong performance but also for architectural robustness.
• Performance scales with the density and accuracy of the prior map. Regularization with grid masking is essential to prevent overfitting to priors, maintaining competitive results if the map is unavailable at deployment.

Implications and Future Work

Practically, DualViewMapDet enables deployment of camera-only perception stacks with detection and tracking accuracy previously achievable only with online LiDAR. This substantially reduces hardware cost and energy requirements while facilitating operation in mapped environments with frequent revisits—an assumption valid for most urban autonomous driving applications.

From a research perspective, the work signals a pivot from multi-modal sensor fusion to sensor and prior fusion, raising important questions about optimal architectures for leveraging static geometric priors, coverage completeness, and pose alignment. The explicit dual fusion strategy highlights that naïve concatenation of representations is suboptimal; architectural subtleties governing metric and semantic information flow will be central for future progress in map-aware perception.

Possible directions include real-time updating of map priors, online map validation, and broader exploitation of prior representations for other peri-perceptual tasks such as occupancy, segmentation, and non-autonomous robotics.

Conclusion

DualViewMapDet presents a robust, dual-space fusion framework for integrating prior static LiDAR maps into camera-only 3D object detection and tracking pipelines. By developing novel PV and BEV map encodings and a sequential fusion architecture, the method consistently improves localization precision and robustness across several large-scale benchmarks. Theoretical and experimental analysis confirms the necessity of dual-space representation and the practical value of prior geometric anchors, establishing a new architectural baseline for future research in map-based camera-only 3D perception.