CAPE: Camera View Position Embedding for Multi-View 3D Object Detection (2303.10209v1)

Abstract: In this paper, we address the problem of detecting 3D objects from multi-view images. Current query-based methods rely on global 3D position embeddings (PE) to learn the geometric correspondence between images and 3D space. We claim that directly interacting 2D image features with global 3D PE could increase the difficulty of learning view transformation due to the variation of camera extrinsics. Thus we propose a novel method based on CAmera view Position Embedding, called CAPE. We form the 3D position embeddings under the local camera-view coordinate system instead of the global coordinate system, such that 3D position embedding is free of encoding camera extrinsic parameters. Furthermore, we extend our CAPE to temporal modeling by exploiting the object queries of previous frames and encoding the ego-motion for boosting 3D object detection. CAPE achieves state-of-the-art performance (61.0% NDS and 52.5% mAP) among all LiDAR-free methods on nuScenes dataset. Codes and models are available on \href{https://github.com/PaddlePaddle/Paddle3D}{Paddle3D} and \href{https://github.com/kaixinbear/CAPE}{PyTorch Implementation}.

• The paper introduces CAPE, a method that uses camera view position embeddings to simplify multi-view 3D object detection.
• It employs a bilateral attention mechanism and a local coordinate embedding strategy to bypass complex camera extrinsic challenges.
• Empirical results on nuScenes show CAPE-T achieves state-of-the-art performance with an NDS of 61.0 and mAP of 52.5.

Camera View Position Embedding for Multi-View 3D Object Detection

The paper "CAPE: Camera View Position Embedding for Multi-View 3D Object Detection" presents an innovative approach to tackling the complexities of 3D object detection using multi-view images, specifically addressing the challenges posed by varied camera extrinsics in autonomous driving scenarios. The proposed method, CAPE, introduces a novel embedding technique that leverages camera-view position embeddings, marking a departure from traditional global space interaction methodologies.

Key Contributions

The paper makes several significant contributions:

1. Camera View Position Embedding (CAPE): CAPE constructs 3D position embeddings under a local camera-view coordinate system rather than a global one. This strategy eliminates the encumbrance of encoding camera extrinsic parameters, thereby simplifying the learning of view transformations.
2. Temporal Modeling Extension (CAPE-T): By exploiting object queries of previous frames and incorporating ego-motion into the position embedding process, CAPE is extended to efficiently handle temporal modeling. This enhances 3D object detection performance further.
3. State-of-the-Art Performance: Empirical evaluations on the nuScenes dataset indicate that CAPE achieves superior performance among LiDAR-free methods, reporting an NDS of 61.0 and mAP of 52.5, indicating its robust capabilities and practical applicability in real-world scenarios.

Methodological Insights

Camera View Position Embedding

CAPE posits that direct interaction of 2D image features with global 3D position embeddings complicates the learning process due to variations in camera extrinsics. To address this, CAPE introduces:

• Key Position Embedding Construction: This involves transforming 3D coordinates within the camera system, free from extrinsic parameters, using a multiple-layer perceptron (MLP).
• Query Position Embedding Construction: CAPE maps global reference points to the local camera system, using camera extrinsics for transformation, and leverages MLP to obtain embeddings aligned with the camera's perspective.

Bilateral Attention Mechanism

The adoption of a bilateral attention mechanism is crucial to CAPE's success, as it decouples embeddings across different systems while simultaneously facilitating local and global interactions independently. This is evidenced by CAPE's improved NDS and mAP upon employing this mechanism.

Temporal Modeling

CAPE-T introduces an innovative temporal modeling approach leveraging ego-motion embeddings to fuse queries across frames. Consideration is given to spatial alignment and frame-specific query sets, fostering consistency and enhancing velocity estimation.

Experimental Outcomes

The results from the nuScenes test set underscore CAPE's efficacy, with CAPE-T exhibiting notable robustness against existing methods in handling temporal information. Additionally, ablation studies furnish insights into the effectiveness of each component, particularly the feature-guided position embeddings in queries and keys, which notably enhance detection accuracy and precision.

Theoretical and Practical Implications

CAPE presents a compelling framework for multi-view 3D object detection devoid of the reliance on LiDAR data, emphasizing the potential to scale easily with reduced computational overhead. The concept of local camera-view embeddings could be applied to other domains of computer vision, aiding in scenarios where transformations between multiple perspectives are required.

Future Directions

While CAPE presents a substantial advancement, its current formulation may adapt to longer temporal sequences, which remains a challenge due to resource constraints. Future work may delve into more sophisticated temporal fusion techniques that retain efficiency while extending the temporal modeling capabilities. Moreover, further exploration into integration with other sensory modalities could amplify robustness and accuracy in dynamic environments.

In essence, CAPE marks a pivotal development in multi-view 3D object detection, offering a scalable and efficient solution to the complex challenges of autonomous perception systems.