Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing mmWave Radar Point Cloud via Visual-inertial Supervision (2404.17229v1)

Published 26 Apr 2024 in cs.RO

Abstract: Complementary to prevalent LiDAR and camera systems, millimeter-wave (mmWave) radar is robust to adverse weather conditions like fog, rainstorms, and blizzards but offers sparse point clouds. Current techniques enhance the point cloud by the supervision of LiDAR's data. However, high-performance LiDAR is notably expensive and is not commonly available on vehicles. This paper presents mmEMP, a supervised learning approach that enhances radar point clouds using a low-cost camera and an inertial measurement unit (IMU), enabling crowdsourcing training data from commercial vehicles. Bringing the visual-inertial (VI) supervision is challenging due to the spatial agnostic of dynamic objects. Moreover, spurious radar points from the curse of RF multipath make robots misunderstand the scene. mmEMP first devises a dynamic 3D reconstruction algorithm that restores the 3D positions of dynamic features. Then, we design a neural network that densifies radar data and eliminates spurious radar points. We build a new dataset in the real world. Extensive experiments show that mmEMP achieves competitive performance compared with the SOTA approach training by LiDAR's data. In addition, we use the enhanced point cloud to perform object detection, localization, and mapping to demonstrate mmEMP's effectiveness.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. S. Wang, D. Cao, R. Liu, W. Jiang, T. Yao, and C. X. Lu, “Human parsing with joint learning for dynamic mmwave radar point cloud,” ACM Proc. IMWUT, vol. 7, no. 1, pp. 1–22, 2023.
  2. H. Li, R. Liu, S. Wang, W. Jiang, and C. X. Lu, “Pedestrian liveness detection based on mmwave radar and camera fusion,” in IEEE Proc. SECON, 2022, pp. 262–270.
  3. D. Cao, R. Liu, H. Li, S. Wang, W. Jiang, and C. X. Lu, “Cross vision-rf gait re-identification with low-cost rgb-d cameras and mmwave radars,” ACM Proc. IMWUT, vol. 6, no. 3, pp. 1–25, 2022.
  4. K. Cai, B. Wang, and C. X. Lu, “Autoplace: Robust place recognition with single-chip automotive radar,” in IEEE Proc. ICRA, 2022, pp. 2222–2228.
  5. P. Gao, S. Zhang, W. Wang, and C. X. Lu, “Dc-loc: Accurate automotive radar based metric localization with explicit doppler compensation,” in IEEE Proc. ICRA, 2022, pp. 4128–4134.
  6. A. Prabhakara, T. Jin, A. Das, G. Bhatt, L. Kumari, E. Soltanaghai, J. Bilmes, S. Kumar, and A. Rowe, “High resolution point clouds from mmwave radar,” in IEEE Proc. ICRA, 2023, pp. 4135–4142.
  7. M. Jiang, G. Xu, H. Pei, Z. Feng, S. Ma, H. Zhang, and W. Hong, “4d high-resolution imagery of point clouds for automotive mmwave radar,” IEEE Transactions on Intelligent Transportation Systems, pp. 1–15, 2023.
  8. F. Ding, A. Palffy, D. M. Gavrila, and C. X. Lu, “Hidden gems: 4d radar scene flow learning using cross-modal supervision,” in IEEE Proc. CVPR, 2023, pp. 9340–9349.
  9. F. Ding, Z. Pan, Y. Deng, J. Deng, and C. X. Lu, “Self-supervised scene flow estimation with 4-d automotive radar,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 8233–8240, 2022.
  10. H. Dong, Y. Xie, X. Zhang, W. Wang, X. Zhang, and J. He, “Gpsmirror: Expanding accurate gps positioning to shadowed and indoor regions with backscatter,” in Proc. ACM MobiCom, 2023.
  11. C. X. Lu, M. R. U. Saputra, P. Zhao, Y. Almalioglu, P. P. De Gusmao, C. Chen, K. Sun, N. Trigoni, and A. Markham, “milliego: single-chip mmwave radar aided egomotion estimation via deep sensor fusion,” in ACM Proc. SenSys, 2020, pp. 109–122.
  12. C. X. Lu, S. Rosa, P. Zhao, B. Wang, C. Chen, J. A. Stankovic, N. Trigoni, and A. Markham, “See through smoke: robust indoor mapping with low-cost mmwave radar,” in ACM Proc. MobiSys, 2020, pp. 14–27.
  13. Y. Cheng, J. Su, M. Jiang, and Y. Liu, “A novel radar point cloud generation method for robot environment perception,” IEEE Transactions on Robotics, vol. 38, no. 6, pp. 3754–3773, 2022.
  14. T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” IEEE Trans. Robot., vol. 34, no. 4, pp. 1004–1020, 2018.
  15. C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” IEEE Transactions on Robotics, vol. 37, no. 6, pp. 1874–1890, 2021.
  16. K. Qian, Z. He, and X. Zhang, “3d point cloud generation with millimeter-wave radar,” ACM Proc IMWUT, vol. 4, no. 4, pp. 1–23, 2020.
  17. H. Yamada, T. Kobayashi, Y. Yamaguchi, and Y. Sugiyama, “High-resolution 2d sar imaging by the millimeter-wave automobile radar,” in IEEE Proc. CAMA, 2017, pp. 149–150.
  18. M. T. Ghasr, M. J. Horst, M. R. Dvorsky, and R. Zoughi, “Wideband microwave camera for real-time 3-d imaging,” IEEE Transactions on Antennas and Propagation, vol. 65, no. 1, pp. 258–268, 2016.
  19. C. M. Watts, P. Lancaster, A. Pedross-Engel, J. R. Smith, and M. S. Reynolds, “2d and 3d millimeter-wave synthetic aperture radar imaging on a pr2 platform,” in IEEE Proc. IROS, 2016, pp. 4304–4310.
  20. J. Guan, S. Madani, S. Jog, S. Gupta, and H. Hassanieh, “Through fog high-resolution imaging using millimeter wave radar,” in IEEE Proc. CVPR, 2020, pp. 11 464–11 473.
  21. Z. Luo, Q. Zhang, W. Wang, and T. Jiang, “Single-antenna device-to-device localization in smart environments with backscatter,” IEEE Internet of Things Journal, vol. 9, no. 12, pp. 10 121–10 129, 2021.
  22. Y. Yang, J. Liu, T. Huang, Q.-L. Han, G. Ma, and B. Zhu, “Ralibev: Radar and lidar bev fusion learning for anchor box free object detection system,” arXiv preprint arXiv:2211.06108, 2022.
  23. Y. Wang, J. Deng, Y. Li, J. Hu, C. Liu, Y. Zhang, J. Ji, W. Ouyang, and Y. Zhang, “Bi-lrfusion: Bi-directional lidar-radar fusion for 3d dynamic object detection,” in IEEE Proc. CVPR, 2023, pp. 13 394–13 403.
  24. Y. Kim, J. W. Choi, and D. Kum, “Grif net: Gated region of interest fusion network for robust 3d object detection from radar point cloud and monocular image,” in IEEE Proc. IROS, 2020, pp. 10 857–10 864.
  25. R. Nabati and H. Qi, “Centerfusion: Center-based radar and camera fusion for 3d object detection,” in IEEE Proc. WACV, 2021, pp. 1527–1536.
  26. J.-J. Hwang, H. Kretzschmar, J. Manela, S. Rafferty, N. Armstrong-Crews, T. Chen, and D. Anguelov, “Cramnet: Camera-radar fusion with ray-constrained cross-attention for robust 3d object detection,” in Proc. ECCV, 2022, pp. 388–405.
  27. Y. Wang, Z. Jiang, Y. Li, J.-N. Hwang, G. Xing, and H. Liu, “Rodnet: A real-time radar object detection network cross-supervised by camera-radar fused object 3d localization,” IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 4, pp. 954–967, 2021.
  28. J.-T. Huang, C.-L. Lu, P.-K. Chang, C.-I. Huang, C.-C. Hsu, P.-J. Huang, H.-C. Wang et al., “Cross-modal contrastive learning of representations for navigation using lightweight, low-cost millimeter wave radar for adverse environmental conditions,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 3333–3340, 2021.
  29. R. Nabati and H. Qi, “Radar-camera sensor fusion for joint object detection and distance estimation in autonomous vehicles,” arXiv preprint arXiv:2009.08428, 2020.
  30. K. Qian, S. Zhu, X. Zhang, and L. E. Li, “Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals,” in IEEE Proc. CVPR, 2021, pp. 444–453.
  31. S. Song, H. Lim, A. J. Lee, and H. Myung, “Dynavins: A visual-inertial slam for dynamic environments,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 523–11 530, 2022.
  32. Y. Fan, H. Han, Y. Tang, and T. Zhi, “Dynamic objects elimination in slam based on image fusion,” Pattern Recognition Letters, vol. 127, pp. 191–201, 2019.
  33. B. Canovas, M. Rombaut, A. Nègre, D. Pellerin, and S. Olympieff, “Speed and memory efficient dense rgb-d slam in dynamic scenes,” in IEEE Proc. IROS, 2020, pp. 4996–5001.
  34. W. Dai, Y. Zhang, P. Li, Z. Fang, and S. Scherer, “Rgb-d slam in dynamic environments using point correlations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 373–389, 2020.
  35. K. Qiu, T. Qin, W. Gao, and S. Shen, “Tracking 3-d motion of dynamic objects using monocular visual-inertial sensing,” IEEE Transactions on Robotics, vol. 35, no. 4, pp. 799–816, 2019.
  36. Y. Ren, B. Xu, C. L. Choi, and S. Leutenegger, “Visual-inertial multi-instance dynamic slam with object-level relocalisation,” in IEEE Proc. IROS, 2022, pp. 11 055–11 062.
  37. B. Xu, W. Li, D. Tzoumanikas, M. Bloesch, A. Davison, and S. Leutenegger, “Mid-fusion: Octree-based object-level multi-instance dynamic slam,” in 2019 International Conference on Robotics and Automation (ICRA).   IEEE, 2019, pp. 5231–5237.
  38. A. Bell, B. Chambers, and H. Butler, “Point data abstraction library,” https://pdal.io/en/latest/apps/chamfer.html, 2023.
  39. ——, “Point data abstraction library,” https://pdal.io/en/latest/apps/hausdorff.html, 2023.
  40. M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
  41. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” arXiv:2304.02643, 2023.
  42. S. Agarwal, K. Mierle, and Others, “Ceres solver,” http://ceres-solver.org.
  43. P. Luc, C. Couprie, S. Chintala, and J. Verbeek, “Semantic segmentation using adversarial networks,” in NIPS Workshop on Adversarial Training, 2016.
  44. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in IEEE Proc. ICCV, 2017, pp. 2980–2988.
  45. K. Bansal, K. Rungta, S. Zhu, and D. Bharadia, “Pointillism: Accurate 3d bounding box estimation with multi-radars,” in ACM Proc. Sensys, 2020, pp. 340–353.
  46. W. Liu, D. Caruso, E. Ilg, J. Dong, A. I. Mourikis, K. Daniilidis, V. Kumar, and J. Engel, “Tlio: Tight learned inertial odometry,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 5653–5660, 2020.
  47. W. Kabsch, “A solution for the best rotation to relate two sets of vectors,” Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, vol. 32, no. 5, pp. 922–923, 1976.
  48. “Rpdnet,” https://github.com/thucyw/RPDNet, Online; Accessed: 14 Sep., 2023.
  49. D. G. Lowe, “Object recognition from local scale-invariant features,” in IEEE Proc. ICCV, vol. 2, 1999, pp. 1150–1157.
  50. T. Qin, J. Pan, S. Cao, and S. Shen, “A general optimization-based framework for local odometry estimation with multiple sensors,” 2019.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now