Multi-Object Tracking with Camera-LiDAR Fusion for Autonomous Driving
Abstract: This paper presents a novel multi-modal Multi-Object Tracking (MOT) algorithm for self-driving cars that combines camera and LiDAR data. Camera frames are processed with a state-of-the-art 3D object detector, whereas classical clustering techniques are used to process LiDAR observations. The proposed MOT algorithm comprises a three-step association process, an Extended Kalman filter for estimating the motion of each detected dynamic obstacle, and a track management phase. The EKF motion model requires the current measured relative position and orientation of the observed object and the longitudinal and angular velocities of the ego vehicle as inputs. Unlike most state-of-the-art multi-modal MOT approaches, the proposed algorithm does not rely on maps or knowledge of the ego global pose. Moreover, it uses a 3D detector exclusively for cameras and is agnostic to the type of LiDAR sensor used. The algorithm is validated both in simulation and with real-world data, with satisfactory results.
- A. Reich and H. -J. Wuensche, “Monocular 3D Multi-Object Tracking with an EKF Approach for Long-Term Stable Tracks”, 2021 IEEE 24th International Conference on Information Fusion (FUSION), Sun City, South Africa, 2021, pp. 1-7, doi: 10.23919/FUSION49465.2021.9626850.
- Zhou, X., Koltun, V. & Krähenbühl, P., “Tracking Objects as Points.”, CoRR. abs/2004.01177, 2020, https://arxiv.org/abs/2004.01177
- H. -K. Chiu, J. Li, R. Ambruş and J. Bohg, “Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving”, 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, 2021, pp. 14227-14233, doi: 10.1109/ICRA48506.2021.9561754.
- J. Choi, S. Ulbrich, B. Lichte and M. Maurer, “Multi-Target Tracking using a 3D-Lidar sensor for autonomous vehicles”, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), The Hague, Netherlands, 2013, pp. 881-886, doi: 10.1109/ITSC.2013.6728343.
- Luo, C., Yang, X. & Yuille, A., “Exploring Simple 3D Multi-Object Tracking for Autonomous Driving.”, CoRR. abs/2108.10312, 2021, https://arxiv.org/abs/2108.10312
- T. Yin, X. Zhou and P. Krähenbühl, “Center-based 3D Object Detection and Tracking”, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 11779-11788, doi: 10.1109/CVPR46437.2021.01161.
- X. Wang, C. Fu, Z. Li, Y. Lai and J. He, “DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion With Deep Association”, in IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 8260-8267, July 2022, doi: 10.1109/LRA.2022.3187264.
- X. Wang, C. Fu, J. He, S. Wang and J. Wang, “StrongFusionMOT: A Multi-Object Tracking Method Based on LiDAR-Camera Fusion”, in IEEE Sensors Journal, vol. 23, no. 11, pp. 11241-11252, 1 June1, 2023, doi: 10.1109/JSEN.2022.3226490.
- Zhu, Z., Nie, J., Wu, H., He, Z. & Gao, M., “MSA-MOT: Multi-Stage Association for 3D Multimodality Multi-Object Tracking.”, Sensors. 22, 2022, https://www.mdpi.com/1424-8220/22/22/8650
- Parravicini, F., Corno, M. & Savaresi, S., “Extended target tracking for autonomous street crossing.”, IFAC-PapersOnLine. 53, 15440-15445 (2020), https://www.sciencedirect.com/science/article/pii/S2405896320330366, 21st IFAC World Congress
- 1000MAD project, https://1000mad.deib.polimi.it/
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.