CAM3DNet: Comprehensively mining the multi-scale features for 3D Object Detection with Multi-View Cameras

Abstract: Query-based 3D object detection methods using multi-view images often struggle to efficiently leverage dynamic multi-scale information, e.g., the relationship between the object features and the geometric of the queries are not sufficiently learned, directly exploring the multi-scale spatiotemporal features will pay too many costs. To address these challenges, we propose CAM3DNet, a novel sparse query-based framework which combines three new modules, composite query (CQ), adaptive self-attention (ASA), and multi-scale hybrid sampling (MSHS). First, the core idea in the CQ module is a multi-scale projection strategy to transform 2D queries into 3D space. Second, the ASA module learns the interactions between the spatiotemporal multi-scale queries. Third, the MSHS module uses the deformable attention mechanism to sample multi-scale object information by considering multi-scales queries, pyramid feature maps, and 2D-camera prior knowledge. The entire model employs a backbone network and a feature pyramid network (FPN) as the encoder, then introduces a YOLOX and a DepthNet as a ROI_Head to produce CQ, and repeatedly utilizes ASA and MSHS as the decoder to gain detection features. Extensive experiments on the nuScenes, Waymo, and Argoverse benchmark datasets demonstrate the effectiveness of our CAM3DNet, and most existing camera-based 3D object detection methods are outperformed. Besides, we make comprehensive ablation studies to check the individual effect of CQ, ASA, and MSHS, as well as their cost of space and computation complexity.

• The paper introduces composite query generation to integrate adaptive, temporal, and global queries, effectively mitigating redundancy in multi-scale feature extraction.
• The methodology employs adaptive self-attention with distance-aware Gaussian kernels and multi-scale hybrid sampling to enhance precision, especially for small and rare object classes.
• Experimental results on nuScenes, Waymo, and Argoverse benchmarks show significant improvements over prior camera-only detectors in mAP and NDS.

CAM3DNet: Multi-Scale Feature Mining for 3D Object Detection with Multi-View Cameras

Introduction

CAM3DNet addresses a fundamental bottleneck in camera-only 3D object detection: efficiently leveraging dynamic multi-scale spatiotemporal information in multi-view scenes. While convolution-based approaches suffer from dense BEV feature redundancy and struggle with distant/small objects, pure query-based alternatives (e.g., PETR, DETR3D, Sparse4D) exhibit limitations in multi-scale perception and overall precision. CAM3DNet introduces a comprehensive framework built upon three key innovations: Composite Query (CQ) generation, an Adaptive Self-Attention (ASA) mechanism, and Multi-Scale Hybrid Sampling (MSHS), each designed for optimal multi-scale, temporal, and spatial feature exploitation. The architecture achieves marked advances in detection accuracy, with strong outperformances on nuScenes, Waymo, and Argoverse benchmarks.

Figure 1: Schematic overview of CAM3DNet, detailing the flow from multi-view image ingestion to backbone/FPN feature extraction, query generation (adaptive, temporal, global), and the composite-query-guided decoder (ASA and MSHS) for 3D bounding box prediction.

Methodology

CAM3DNet is architectured with clear modular separation between the encoder (feature extraction and query generation) and decoder (feature interaction and prediction), with a strong emphasis on adaptive multi-scale and multi-domain query mining.

1. Composite Query (CQ) Generation

CAM3DNet unifies three query sources:

• Adaptive Query (AQ): Projects multi-scale 2D detections (via YOLOX and DepthNet) into 3D, embedding both semantic and geometric information at multiple image scales. Depth estimation for each 2D detection is fused, and these 3D anchor points are constructed with explicit camera intrinsics/extrinsics.
• Temporal Query (TQ): Derives from a memory queue storing past adaptive queries. A transformer-based positional predictor updates geometric information over time, with full propagation across contiguous frames. This enables robust exploitation of historical spatial-temporal cues (as ablated in Figure 2).
• Global Query (GQ): Randomly-initialized BEV grid queries serve as a general global anchor, complemented with positional encoding.

These components are concatenated to form high-quality, redundancy-reduced composite queries that inherently encode spatiotemporal and scale diversity throughout the batch.

2. Adaptive Self-Attention (ASA)

Standard self-attention lacks scale adaptivity and locality-awareness, treating all queries globally. ASA incorporates learned distance-dependent Gaussian kernels: attention between queries is modulated by their spatial 3D distance (with head-specific, trainable variance $\epsilon_i$), enabling each attention head to focus on features at an adaptive receptive field. This design is rigorously justified and numerically shown superior to variations in both the functional form and the parameterization (Table—see main paper, Section 4.4).

3. Multi-Scale Hybrid Sampling (MSHS)

“Hybrid” reference points for cross-attention arise from a mixture of:

• Fixed points: Normatively placed in the bounding box geometry (e.g., centers, corners, edge points).
• Learned offsets: Derived from multi-scale feature maps in conjunction with the CQ.

A learnable weight $\alpha$ interpolates these two, and final 2D projection is obtained with explicit camera intrinsics/extrinsics. Deformable attention aggregates features over these sampled points at each decoder layer, achieving highly context-sensitive, scale-aware feature gathering.

Experimental Results

Results on nuScenes, Waymo, and Argoverse

CAM3DNet consistently surpasses prior camera-only SOTA detectors:

• nuScenes (ResNet-50 backbone, $256 \times 704$ input): CAM3DNet achieves 0.4598 mAP and 0.5511 NDS, +1.57% mAP and +1.04% NDS over Far3D, and 0.98% mAP over StreamPETR, while maintaining competitive efficiency (Section 5.2).
• nuScenes (V2-99, $320 \times 800$): 0.5168 mAP, 0.6021 NDS, surpassing StreamPETR by 3.48% mAP and 3.11% NDS and competitive with fully dense methods (Table, Section 5.2).
• Waymo (ResNet-101): On mLETAP, mLETAPH, mLETAPL, CAM3DNet achieves 0.576, 0.413, 0.535, outperforming query-based and convolutional competitors with the exception of methods exploiting denser grids.
• Argoverse 2: 0.266 mAP, 0.193 CDS—+2.2% mAP and +1.2% CDS over Far3D.

These improvements are particularly pronounced on rare and small classes, such as trailers, traffic cones, and buses (see Figure 3 and Figure 4).

Figure 3: Category-wise Average Precision (AP) comparison between CAM3DNet and SOTA on nuScenes; colors denote evaluation settings and clear improvements appear for small/rare classes such as bus, trailer, and traffic cone.

Figure 4: AP scores under 1.0m distance threshold across object categories on nuScenes, visualizing maintained superiority in stringent localization regimes.

Visualization and Quantitative Analysis

Qualitative outputs illustrate accurate scene parsing with only occasional failures on extremely distant/crowded targets (see Figure 5). Ablation studies confirm nontrivial, largely orthogonal performance gains from each module in CAM3DNet, especially from ASA and the full CQ design. MSHS further benefits from a principled blend of learned and geometric reference sampling, with edge points and moderate learnable point counts yielding maximal NDS (Table in main text).

Figure 5: Visualizations contrasting CAM3DNet detections (bottom) against ground truth (top) in both the image and BEV plane; red circles indicate rare failure cases, generally in remote and crowded regions.

Figure 2: Impact of temporal queue length on detection; an optimal queue length (4) is observed where NDS peaks for the incorporation of temporal queries.

Efficiency and Resource Analysis

The bulk of computational and parameter cost arises from the RoI_Head (YOLOX + DepthNet), accounting for ca. 37.73% of total FLOPs—as expected, given the joint 2D detection and depth estimation. The overall efficiency is competitive with alternative SOTA, and much of the trade-off over StreamPETR is offset by the significant accuracy boosts derived from multi-scale and adaptive querying. Opportunities for future efficiency improvement are identified in possible RoI_Head streamlining.

Implications and Future Directions

CAM3DNet demonstrates that principled exploitation of multi-view, multi-scale, and temporal cues—integrated through composite queries and adaptive attention—yields substantial accuracy gains for camera-only 3D object detection. Its robustness to small and rare object categories enhances practical utility in autonomous driving and mobile robotics, where sensor cost, coverage, and energy constraints often preclude LiDAR.

The main limitation remains depth ambiguity inherent to images, especially under adverse conditions. Prospective work includes augmenting the framework for improved depth robustness, potentially via cross-sensor knowledge distillation, enhanced synthetic augmentation (e.g., low-light, occlusion scenarios), and further lightweight designs in query and feature extraction heads.

Conclusion

CAM3DNet establishes a state-of-the-art benchmark for multi-view camera-based 3D object detection by unifying adaptive, multi-scale, temporal, and geometric feature mining within an efficient, sparse query framework. The design provides a clear blueprint for extensible, high-accuracy camera-only 3D perception applicable to real-world autonomous systems and offers a strong foundation for future advances in robust, efficient scene understanding.

---

Reference:

CAM3DNet: Comprehensively mining the multi-scale features for 3D Object Detection with Multi-View Cameras (2604.17024)