Papers
Topics
Authors
Recent
Search
2000 character limit reached

Towards Consistent Object Detection via LiDAR-Camera Synergy

Published 2 May 2024 in cs.CV, cs.RO, and eess.IV | (2405.01258v2)

Abstract: As human-machine interaction continues to evolve, the capacity for environmental perception is becoming increasingly crucial. Integrating the two most common types of sensory data, images, and point clouds, can enhance detection accuracy. Currently, there is no existing model capable of detecting an object's position in both point clouds and images while also determining their corresponding relationship. This information is invaluable for human-machine interactions, offering new possibilities for their enhancement. In light of this, this paper introduces an end-to-end Consistency Object Detection (COD) algorithm framework that requires only a single forward inference to simultaneously obtain an object's position in both point clouds and images and establish their correlation. Furthermore, to assess the accuracy of the object correlation between point clouds and images, this paper proposes a new evaluation metric, Consistency Precision (CP). To verify the effectiveness of the proposed framework, an extensive set of experiments has been conducted on the KITTI and DAIR-V2X datasets. The study also explored how the proposed consistency detection method performs on images when the calibration parameters between images and point clouds are disturbed, compared to existing post-processing methods. The experimental results demonstrate that the proposed method exhibits excellent detection performance and robustness, achieving end-to-end consistency detection. The source code will be made publicly available at https://github.com/xifen523/COD.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “YOLOX: Exceeding YOLO series in 2021,” arXiv preprint arXiv:2107.08430, 2021.
  2. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proc. CVPR, 2016, pp. 779–788.
  3. Y. Zhao et al., “DETRs beat YOLOs on real-time object detection,” arXiv preprint arXiv:2304.08069, 2023.
  4. C. He, H. Zeng, J. Huang, X.-S. Hua, and L. Zhang, “Structure aware single-stage 3D object detection from point cloud,” in Proc. CVPR, 2020, pp. 11 870–11 879.
  5. Y. Chen et al., “FocalFormer3D: Focusing on hard instance for 3D object detection,” in Proc. ICCV, 2023, pp. 8360–8371.
  6. H. Hu, F. Wang, J. Su, Y. Wang, L. Hu, W. Fang, J. Xu, and Z. Zhang, “Ea-lss: Edge-aware lift-splat-shot framework for 3d bev object detection,” arXiv preprint arXiv:2303.17895, 2023.
  7. Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. Rus, and S. Han, “Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,” in IEEE International Conference on Robotics and Automation (ICRA), 2023.
  8. O. D. Team, “Openpcdet: An open-source toolbox for 3d object detection from point clouds,” https://github.com/open-mmlab/OpenPCDet, 2020.
  9. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in Proc. ECCV, vol. 12346, 2020, pp. 213–229.
  10. A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in Proc. CVPR, 2012, pp. 3354–3361.
  11. H. Yu et al., “DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3D object detection,” in Proc. CVPR, 2022, pp. 21 329–21 338.
  12. R. Girshick, “Fast R-CNN,” in Proc. ICCV, 2015, pp. 1440–1448.
  13. S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137–1149, 2017.
  14. T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proc. CVPR, 2017, pp. 936–944.
  15. K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proc. ICCV, 2017, pp. 2980–2988.
  16. W. Liu et al., “SSD: Single shot multibox detector,” in Proc. ECCV, vol. 9905, 2015, pp. 21–37.
  17. J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018.
  18. C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” Proc. CVPR, pp. 7464–7475, 2022.
  19. T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang, and Y. Shan, “YOLO-World: Real-time open-vocabulary object detection,” in Proc. CVPR, 2024.
  20. Z. Yao, J. Ai, B. Li, and C. Zhang, “Efficient DETR: Improving end-to-end object detector with dense prior,” arXiv preprint arXiv:2104.01318, 2021.
  21. F. Li, H. Zhang, S. guang Liu, J. Guo, L. M. shuan Ni, and L. Zhang, “DN-DETR: Accelerate DETR training by introducing query denoising,” Proc. CVPR, pp. 13 609–13 617, 2022.
  22. H. Zhang et al., “DINO: DETR with improved denoising anchor boxes for end-to-end object detection,” in Proc. ICLR, 2023.
  23. Y. Zhou and O. Tuzel, “VoxelNet: End-to-end learning for point cloud based 3D object detection,” Proc. CVPR, pp. 4490–4499, 2017.
  24. Y. Yan, Y. Mao, and B. Li, “SECOND: Sparsely embedded convolutional detection,” Sensors, vol. 18, no. 10, p. 3337, 2018.
  25. A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “PointPillars: Fast encoders for object detection from point clouds,” in Proc. CVPR, 2019, pp. 12 697–12 705.
  26. M. Ye, S. Xu, and T. Cao, “HVNet: Hybrid voxel network for LiDAR based 3D object detection,” Proc. CVPR, pp. 1628–1637, 2020.
  27. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” in Proc. NeurIPS, vol. 30, 2017, pp. 5099–5108.
  28. C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep learning on point sets for 3D classification and segmentation,” Proc. CVPR, pp. 77–85, 2016.
  29. A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “PointPillars: Fast encoders for object detection from point clouds,” in Proc. CVPR, 2018, pp. 12 697–12 705.
  30. Z. Yang, Y. Sun, S. Liu, and J. Jia, “3DSSD: Point-based 3D single stage object detector,” Proc. CVPR, pp. 11 037–11 045, 2020.
  31. S. Vora, A. H. Lang, B. Helou, and O. Beijbom, “PointPainting: Sequential fusion for 3D object detection,” Proc. CVPR, pp. 4603–4611, 2019.
  32. L. Pan et al., “Variational relational point completion network,” in Proc. CVPR, 2021, pp. 8524–8533.
  33. X. Bai et al., “TransFusion: Robust LiDAR-camera fusion for 3D object detection with transformers,” Proc. CVPR, pp. 1080–1089, 2022.
  34. M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun, “Multi-task multi-sensor fusion for 3D object detection,” Proc. CVPR, pp. 7345–7353, 2019.
  35. Y. Li et al., “DeepFusion: Lidar-camera deep fusion for multi-modal 3D object detection,” Proc. CVPR, pp. 17 161–17 170, 2022.
  36. X. Wu et al., “Sparse fuse dense: Towards high quality 3D detection with depth completion,” Proc. CVPR, pp. 5408–5417, 2022.
  37. S. Pang, D. Morris, and H. Radha, “CLOCs: Camera-LiDAR object candidates fusion for 3D object detection,” Proc. IROS, pp. 10 386–10 393, 2020.
  38. T. Yin, X. Zhou, and P. Krähenbühl, “Center-based 3D object detection and tracking,” in Proc. CVPR, 2021, pp. 11 784–11 793.
  39. C. M. Guangsheng Shi, Ruifeng Li, “PillarNet: Real-time and high-performance pillar-based 3D object detection,” in Proc. ECCV, vol. 13670, 2022, pp. 35–52.
  40. G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLO,” Jan. 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up