Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud (2309.09737v7)

Published 18 Sep 2023 in cs.CV, cs.AI, cs.LG, and cs.RO

Abstract: Mobile autonomy relies on the precise perception of dynamic environments. Robustly tracking moving objects in 3D world thus plays a pivotal role for applications like trajectory prediction, obstacle avoidance, and path planning. While most current methods utilize LiDARs or cameras for Multiple Object Tracking (MOT), the capabilities of 4D imaging radars remain largely unexplored. Recognizing the challenges posed by radar noise and point sparsity in 4D radar data, we introduce RaTrack, an innovative solution tailored for radar-based tracking. Bypassing the typical reliance on specific object types and 3D bounding boxes, our method focuses on motion segmentation and clustering, enriched by a motion estimation module. Evaluated on the View-of-Delft dataset, RaTrack showcases superior tracking precision of moving objects, largely surpassing the performance of the state of the art. We release our code and model at https://github.com/LJacksonPan/RaTrack.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, et al., “Tnt: Target-driven trajectory prediction,” in Proceedings of the Conference on Robot Learning.   PMLR, 2021, pp. 895–904.
  2. N. Deo, E. Wolff, and O. Beijbom, “Multimodal trajectory prediction conditioned on lane-graph traversals,” in Proceedings of the Conference on Robot Learning.   PMLR, 2022, pp. 203–212.
  3. M. Wei and V. Isler, “Energy-efficient path planning for ground robots by and combining air and ground measurements,” in Proceedings of the Conference on Robot Learning.   PMLR, 2020, pp. 766–775.
  4. A. Li, L. Sun, W. Zhan, M. Tomizuka, and M. Chen, “Prediction-based reachability for collision avoidance in autonomous driving,” in Proceedings of the IEEE International Conference on Robotics and Automation.   IEEE, 2021, pp. 7908–7914.
  5. J. Lin, H. Zhu, and J. Alonso-Mora, “Robust vision-based obstacle avoidance for micro aerial vehicles in dynamic environments,” in Proceedings of the IEEE International Conference on Robotics and Automation.   IEEE, 2020, pp. 2682–2688.
  6. H. Zhang, H. Jin, Z. Liu, Y. Liu, Y. Zhu, and J. Zhao, “Real-time kinematic control for redundant manipulators in a time-varying environment: Multiple-dynamic obstacle avoidance and fast tracking of a moving object,” IEEE Transactions on Industrial Informatics, vol. 16, no. 1, pp. 28–41, 2019.
  7. C. Petres, Y. Pailhas, P. Patron, Y. Petillot, J. Evans, and D. Lane, “Path planning for autonomous underwater vehicles,” IEEE Transactions on Robotics, vol. 23, no. 2, pp. 331–341, 2007.
  8. R. Yonetani, T. Taniai, M. Barekatain, M. Nishimura, and A. Kanezaki, “Path planning using neural a* search,” in Proceedings of the International Conference on Machine Learning.   PMLR, 2021, pp. 12 029–12 039.
  9. H. Inotsume, T. Kubota, and D. Wettergreen, “Robust path planning for slope traversing under uncertainty in slip prediction,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3390–3397, 2020.
  10. X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2020, pp. 10 359–10 366.
  11. T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11 784–11 793.
  12. H.-k. Chiu, J. Li, R. Ambruş, and J. Bohg, “Probabilistic 3d multi-modal, multi-object tracking for autonomous driving,” in Proceedings of the IEEE International Conference on Robotics and Automation.   IEEE, 2021, pp. 14 227–14 233.
  13. T. Fischer, Y.-H. Yang, S. Kumar, M. Sun, and F. Yu, “Cc-3dt: Panoramic 3d object tracking via cross-camera fusion,” in Proceedings of the Conference on Robot Learning, 2022.
  14. H.-N. Hu, Y.-H. Yang, T. Fischer, T. Darrell, F. Yu, and M. Sun, “Monocular quasi-dense 3d object tracking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 1992–2008, 2022.
  15. F. Ding, C. Fu, Y. Li, J. Jin, and C. Feng, “Automatic failure recovery and re-initialization for online uav tracking with joint scale and aspect ratio optimization,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2020, pp. 5970–5977.
  16. A. Shenoi, M. Patel, J. Gwak, P. Goebel, A. Sadeghian, H. Rezatofighi, R. Martin-Martin, and S. Savarese, “Jrmot: A real-time 3d multi-object tracker and a new large-scale dataset,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2020, pp. 10 335–10 342.
  17. A. Kim, A. Ošep, and L. Leal-Taixé, “Eagermot: 3d multi-object tracking via sensor fusion,” in Proceedings of the IEEE International Conference on Robotics and Automation.   IEEE, 2021, pp. 11 315–11 321.
  18. Y. Zeng, C. Ma, M. Zhu, Z. Fan, and X. Yang, “Cross-modal 3d object detection and tracking for auto-driving,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2021, pp. 3850–3857.
  19. M. Liang, B. Yang, W. Zeng, Y. Chen, R. Hu, S. Casas, and R. Urtasun, “Pnpnet: End-to-end perception and prediction with tracking in the loop,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 553–11 562.
  20. A. Palffy, E. Pool, S. Baratam, J. F. Kooij, and D. M. Gavrila, “Multi-class road user detection with 3+ 1d radar in the view-of-delft dataset,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4961–4968, 2022.
  21. A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1231–1237, 2013.
  22. A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler, “Mot16: A benchmark for multi-object tracking,” arXiv preprint arXiv:1603.00831, 2016.
  23. G. Ciaparrone, F. L. Sánchez, S. Tabik, L. Troiano, R. Tagliaferri, and F. Herrera, “Deep learning in video multi-object tracking: A survey,” Neurocomputing, vol. 381, pp. 61–88, 2020.
  24. Z. Wang, L. Zheng, Y. Liu, Y. Li, and S. Wang, “Towards real-time multi-object tracking,” in Proceedings of the European Conference on Computer Vision.   Springer, 2020, pp. 107–122.
  25. P. Voigtlaender, M. Krause, A. Osep, J. Luiten, B. B. G. Sekar, A. Geiger, and B. Leibe, “Mots: Multi-object tracking and segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 7942–7951.
  26. T. Meinhardt, A. Kirillov, L. Leal-Taixe, and C. Feichtenhofer, “Trackformer: Multi-object tracking with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8844–8854.
  27. J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. L. Waslander, “Joint 3d proposal generation and object detection from view aggregation,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2018, pp. 1–8.
  28. B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu, “Class-balanced grouping and sampling for point cloud 3d object detection,” arXiv preprint arXiv:1908.09492, 2019.
  29. S. Shi, X. Wang, and H. Li, “Pointrcnn: 3d object proposal generation and detection from point cloud,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 770–779.
  30. B. Yang, W. Luo, and R. Urtasun, “Pixor: Real-time 3d object detection from point clouds,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2018.
  31. W. Shi and R. Rajkumar, “Point-gnn: Graph neural network for 3d object detection in a point cloud,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1711–1719.
  32. E. Baser, V. Balasubramanian, P. Bhattacharyya, and K. Czarnecki, “Fantrack: 3d multi-object tracking with feature association network,” in IEEE Intelligent Vehicles Symposium.   IEEE, 2019, pp. 1426–1433.
  33. J. Pöschmann, T. Pfeifer, and P. Protzel, “Factor graph based 3d multi-object tracking in point clouds,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2020, pp. 10 343–10 350.
  34. X. Weng, Y. Wang, Y. Man, and K. M. Kitani, “Gnn3dmot: Graph neural network for 3d multi-object tracking with 2d-3d multi-feature learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6499–6508.
  35. N. Benbarka, J. Schröder, and A. Zell, “Score refinement for confidence-based 3d multi-object tracking,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2021, pp. 8083–8090.
  36. Z. Pang, Z. Li, and N. Wang, “Simpletrack: Understanding and rethinking 3d multi-object tracking,” arXiv preprint arXiv:2111.09621, 2021.
  37. J.-N. Zaech, A. Liniger, D. Dai, M. Danelljan, and L. Van Gool, “Learnable online graph representations for 3d multi-object tracking,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 5103–5110, 2022.
  38. T. Wen, Y. Zhang, and N. M. Freris, “Pf-mot: Probability fusion based 3d multi-object tracking for autonomous vehicles,” in Proceedings of the International Conference on Robotics and Automation.   IEEE, 2022, pp. 700–706.
  39. A. Kim, G. Brasó, A. Ošep, and L. Leal-Taixé, “Polarmot: How far can geometric relations take us in 3d multi-object tracking?” in Proceedings of the European Conference on Computer Vision.   Springer, 2022, pp. 41–58.
  40. T. Sadjadpour, J. Li, R. Ambrus, and J. Bohg, “Shasta: Modeling shape and spatio-temporal affinities for 3d multi-object tracking,” arXiv preprint arXiv:2211.03919, 2022.
  41. M. Chaabane, P. Zhang, J. R. Beveridge, and S. O’Hara, “Deft: Detection embeddings for tracking,” arXiv preprint arXiv:2102.02267, 2021.
  42. X. Zhou, V. Koltun, and P. Krähenbühl, “Tracking objects as points,” in Proceedings of the European Conference on Computer Vision.   Springer, 2020, pp. 474–490.
  43. H. W. Kuhn, “The hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, no. 1-2, pp. 83–97, 1955.
  44. G. Brasó and L. Leal-Taixé, “Learning a neural solver for multiple object tracking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6247–6257.
  45. R. Sinkhorn and P. Knopp, “Concerning nonnegative matrices and doubly stochastic matrices,” Pacific Journal of Mathematics, vol. 21, no. 2, pp. 343–348, 1967.
  46. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” Proceedings of the Advances in Neural Information Processing Systems, vol. 30, 2017.
  47. W. Wu, Z. Y. Wang, Z. Li, W. Liu, and L. Fuxin, “Pointpwc-net: Cost volume on point clouds for (self-) supervised scene flow estimation,” in Proceedings of the European Conference on Computer Vision.   Springer, 2020, pp. 88–107.
  48. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, “On the Properties of Neural Machine Translation: Encoder–Decoder Approaches,” in Proceedings of the Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014, pp. 103–111.
  49. M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al., “A density-based algorithm for discovering clusters in large spatial databases with noise.” in Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, vol. 96, no. 34, 1996, pp. 226–231.
  50. L. Leal-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler, “Motchallenge 2015: Towards a benchmark for multi-target tracking,” arXiv preprint arXiv:1504.01942, 2015.
  51. K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: the clear mot metrics,” EURASIP Journal on Image and Video Processing, vol. 2008, pp. 1–10, 2008.
  52. A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12 697–12 705.
  53. S. A. Baur, D. J. Emmerichs, F. Moosmann, P. Pinggera, B. Ommer, and A. Geiger, “SLIM: Self-Supervised LiDAR Scene Flow and Motion Segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13 126–13 136.
  54. F. Ding, Z. Pan, Y. Deng, J. Deng, and C. X. Lu, “Self-supervised scene flow estimation with 4-d automotive radar,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 8233–8240, 2022.
  55. F. Ding, A. Palffy, D. M. Gavrila, and C. X. Lu, “Hidden gems: 4d radar scene flow learning using cross-modal supervision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1–10.
  56. P. Jund, C. Sweeney, N. Abdo, Z. Chen, and J. Shlens, “Scalable scene flow from point clouds in the real world,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 1589–1596, 2021.
  57. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
Citations (10)
View on Semantic Scholar

Summary

  • The paper introduces RaTrack, an end-to-end trainable system using 4D radar data for robust moving object detection and tracking.
  • It employs class-agnostic detection via motion segmentation and point-wise scene flow estimation, simplifying traditional bounding box methods.
  • Evaluated on the View-of-Delft dataset, RaTrack achieves superior performance in MOTA, MODA, and sAMOTA metrics compared to LiDAR-based approaches.

RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud

The paper "RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud" proposes an innovative approach to track moving objects using 4D radar data, a capability that has been largely underutilized in comparison to LiDARs and cameras. This research is significant in the field of mobile autonomy and has profound implications for autonomous vehicles that require accurate perception and understanding of dynamic environments to achieve tasks such as trajectory prediction and obstacle avoidance.

Key Contributions

  1. Utilization of 4D Radar Data: Unlike traditional methods that primarily rely on LiDAR or camera data for Multiple Object Tracking (MOT), this paper introduces RaTrack, an approach specialized for radar data. 4D radar is increasingly viewed as a robust and cost-effective alternative to LiDAR due to its resilience in adverse weather conditions and enhanced imaging capabilities.
  2. Class-agnostic Detection: A central innovation of RaTrack is its method of detecting moving objects without relying on 3D bounding box estimations or specific object type recognitions. Through motion segmentation and clustering, this method effectively addresses the challenge posed by the noisy and sparse nature of radar point clouds. This departure from conventional bounding box regression simplifies the detection process and improves accuracy in object tracking.
  3. Scene Flow Estimation: The inclusion of a point-wise motion estimation module enriches the radar data with additional motion cues that are traditionally absent due to the nature of radar returns. By backward-estimating point velocities, RaTrack offers enhanced tracking precision by incorporating both motion segmentation and scene flow in its detection framework.
  4. End-to-End Trainable Network: The end-to-end architecture of RaTrack integrates motion segmentation, scene flow estimation, and data association within a single framework. This integration allows for simultaneous optimization across tasks, thereby improving overall tracking performance.

Experimental Evaluation

The authors evaluated RaTrack using the View-of-Delft (VoD) dataset, demonstrating substantial improvements over existing LiDAR-based methods. Specifically, RaTrack achieved superior performance on metrics such as MOTA, MODA, and sAMOTA. Notably, it surpassed other techniques in detecting and maintaining trajectories of moving objects, underscoring the efficacy of using radar data for these tasks.

Implications and Future Directions

The implications of successfully integrating 4D radar data into MOT systems are considerable. Not only does this offer a more cost-effective and reliable solution under varying environmental conditions, but it also alleviates computational burdens associated with processing dense point clouds from LiDAR. This could pave the way for more scalable and accessible autonomous systems.

Future work could explore the fusion of radar with other sensors to enhance data richness and improve object recognition capabilities. Combining radar with optical data holds potential for even more robust multi-modal perception systems. Additionally, investigating the scalability of RaTrack in large-scale scenarios, as well as its adaptability to different radar configurations, could further solidify its application in autonomous vehicles.

In conclusion, RaTrack's approach to leveraging 4D radar for object detection and tracking marks a significant advancement in the field of autonomous navigation. Its novel methodology and proven performance through extensive experimentation indicate promising pathways for the application of radar in autonomous systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - LJacksonPan/RaTrack: [ICRA2024] RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud (127 stars)