Sparse4D v3: Advancing End-to-End 3D Detection and Tracking (2311.11722v1)

Abstract: In autonomous driving perception systems, 3D detection and tracking are the two fundamental tasks. This paper delves deeper into this field, building upon the Sparse4D framework. We introduce two auxiliary training tasks (Temporal Instance Denoising and Quality Estimation) and propose decoupled attention to make structural improvements, leading to significant enhancements in detection performance. Additionally, we extend the detector into a tracker using a straightforward approach that assigns instance ID during inference, further highlighting the advantages of query-based algorithms. Extensive experiments conducted on the nuScenes benchmark validate the effectiveness of the proposed improvements. With ResNet50 as the backbone, we witnessed enhancements of 3.0\%, 2.2\%, and 7.6\% in mAP, NDS, and AMOTA, achieving 46.9\%, 56.1\%, and 49.0\%, respectively. Our best model achieved 71.9\% NDS and 67.7\% AMOTA on the nuScenes test set. Code will be released at \url{https://github.com/linxuewu/Sparse4D}.

• The paper introduces two auxiliary training tasks—Temporal Instance Denoising and Quality Estimation—to enhance 3D detection quality.
• It employs a decoupled attention mechanism that concatenates features for clearer self-attention, reducing query interference.
• An end-to-end tracking model with a simple ID assignment strategy boosts performance by 3.0% mAP, 2.2% NDS, and 7.6% AMOTA on the nuScenes benchmark.

Sparse4D v3: Enhancing 3D Detection and Tracking in Autonomous Driving

The paper "Sparse4D v3: Advancing End-to-End 3D Detection and Tracking" presents a comprehensive approach to improving 3D object detection and tracking systems used in autonomous driving. Building upon the Sparse4D framework, the authors propose several key advancements aimed at optimizing detection performance and extending these improvements to tracking tasks. The modifications fundamentally enhance both the practical applications and theoretical understandings of perception in autonomous systems.

Key Contributions and Methodologies

1. Auxiliary Training Tasks: The paper introduces two auxiliary training tasks aimed at strengthening the Sparse4D framework—Temporal Instance Denoising and Quality Estimation.

• Temporal Instance Denoising: This task extends the conventional 2D single-frame denoising into a more complex 3D temporal domain. By adding noise to ground truth targets, positive and negative samples are generated to stabilize the decoder's training. This approach effectively increases the number of positive samples available, improving overall performance metrics.
• Quality Estimation: By introducing centerness and yawness as evaluation metrics, the output confidence scores become more representative of detection quality, improving the reliability and ranking of detections. This addresses the discrepancies in detecting smooth and accurate bounding boxes.

1. Decoupled Attention Mechanism: This structural improvement replaces the addition operation with concatenation in attention calculations, addressing the issue of attention weight interference. Although similar to Conditional DETR, this method emphasizes self-attention among queries.
2. End-to-End Tracking Model: Sparse4D is transformed into a 3D multi-object tracking model by using a simple ID assignment strategy during inference. This approach eliminates the need for additional data association or ground truth ID reliance, creating a seamless transition from detection to tracking.

Experimental Validation

The proposed Sparse4D v3 underwent rigorous testing on the nuScenes benchmark. Using ResNet50 as a backbone, the variant demonstrated improvements of 3.0% in mAP, 2.2% in NDS, and 7.6% in AMOTA. The highest-performing model reached 71.9% NDS and 67.7% AMOTA on the test set. These results underscore the robustness of the Sparse4D improvements in real-world autonomous driving conditions.

Implications and Future Directions

The improvements proposed carry significant implications:

• Practical Enhancements: The approach significantly reduces computational load without compromising performance, making it suitable for real-time autonomous driving applications.
• End-to-End Integration: The integration of detection and tracking tasks into a single framework reduces system complexity and streamlines the pipeline.
• Potential for Expansion: The framework can be extended to lidar-only or multi-modal models, opening avenues for more comprehensive perception systems. Furthermore, incorporating prediction and planning tasks presents opportunities for a more integrated autonomous driving system.

Future developments could explore richer integration of downstream tasks like prediction and planning, enhancing the holistic capability of perception and action systems in autonomous vehicles. There is also room for enhancing tracking performance by refining the starting framework and implementing additional perception tasks such as online mapping.

In conclusion, Sparse4D v3 delivers substantial improvements to 3D detection and tracking in autonomous systems, showcasing both theoretical advancements and practical implementations that could inform future research and application developments in this domain.