Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Robust Human Detection under Visual Degradation via Thermal and mmWave Radar Fusion (2307.03623v1)

Published 7 Jul 2023 in cs.CV and cs.RO

Abstract: The majority of human detection methods rely on the sensor using visible lights (e.g., RGB cameras) but such sensors are limited in scenarios with degraded vision conditions. In this paper, we present a multimodal human detection system that combines portable thermal cameras and single-chip mmWave radars. To mitigate the noisy detection features caused by the low contrast of thermal cameras and the multi-path noise of radar point clouds, we propose a Bayesian feature extractor and a novel uncertainty-guided fusion method that surpasses a variety of competing methods, either single-modal or multi-modal. We evaluate the proposed method on real-world data collection and demonstrate that our approach outperforms the state-of-the-art methods by a large margin.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Introducing the intel® realsense™ depth camera d455, 2023.
  2. Vayyar imaging - home, 2023.
  3. Barf: A new direct and cross-based binary residual feature fusion with uncertainty-aware module for medical image classification. Information Sciences, 577:353–378, 2021.
  4. Uncertaintyfusenet: robust uncertainty-aware hierarchical feature fusion model with ensemble monte carlo dropout for covid-19 detection. Information Fusion, 90:364–381, 2023.
  5. Distant vehicle detection using radar and vision. In 2019 International Conference on Robotics and Automation (ICRA), pages 8311–8317. IEEE, 2019.
  6. Spatial attention fusion for obstacle detection using mmwave radar and vision sensor. Sensors, 20(4):956, 2020.
  7. Rfcam: Uncertainty-aware fusion of camera and wi-fi for real-time human identification with mobile devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(2):1–29, 2022.
  8. Integrated sensing and communication between daily devices and mmwave radars. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems (SenSys), 2022.
  9. 2d car detection in radar data with pointnets. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 61–66. IEEE, 2019.
  10. Global-local feature enhancement network for robust object detection using mmwave radar and camera. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4708–4712. IEEE, 2022.
  11. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.
  12. R. Gade and T. B. Moeslund. Thermal cameras and applications: a survey. Machine vision and applications, 25:245–262, 2014.
  13. Pedestrian detection in thermal images using saliency maps. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 988–997, 2019.
  14. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3828–3838, 2019.
  15. Domain-adaptive pedestrian detection in thermal images. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1660–1664. IEEE, 2019.
  16. Cnn-based thermal infrared person detection by domain adaptation. In Autonomous Systems: Sensors, Vehicles, Security, and the Internet of Everything, volume 10643, pages 38–43. SPIE, 2018.
  17. Cramnet: Camera-radar fusion with ray-constrained cross-attention for robust 3d object detection. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVIII, pages 388–405. Springer, 2022.
  18. ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation, Nov. 2022.
  19. V. John and S. Mita. Rvnet: Deep sensor fusion of monocular camera and radar for image-based obstacle detection in challenging environments. In Pacific-Rim Symposium on Image and Video Technology, pages 351–364. Springer, 2019.
  20. A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
  21. Thermal object detection in difficult weather conditions using yolo. IEEE Access, 8:125459–125476, 2020.
  22. S. Lee. Deep learning on radar centric 3d object detection. arXiv preprint arXiv:2003.00851, 2020.
  23. Pedestrian liveness detection based on mmwave radar and camera fusion. In 2022 19th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pages 262–270. IEEE, 2022.
  24. Yolo-firi: Improved yolov5 for infrared image object detection. IEEE Access, 9:141861–141875, 2021.
  25. Modality-agnostic learning for radar-lidar fusion in vehicle detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 908–917, 2022.
  26. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
  27. Wavoice: A noise-resistant multi-modal speech recognition system fusing mmwave and audio signals. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems (SenSys), 2021.
  28. Salient object detection for rgb-d image by single stream recurrent convolution neural network. Neurocomputing, 363:46–57, 2019.
  29. Getting to know low-light images with the exclusively dark dataset. Computer Vision and Image Understanding, 178:30–42, 2019.
  30. milliego: single-chip mmwave radar aided egomotion estimation via deep sensor fusion. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems, pages 109–122, 2020.
  31. Fusion technology of radar and rgb camera sensors for object detection and tracking and its embedded system implementation. In 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1234–1242. IEEE, 2020.
  32. M. Meyer and G. Kuschk. Deep learning based 3d object detection for automotive radar and camera. In 2019 16th European Radar Conference (EuRAD), pages 133–136, 2019.
  33. Graph convolutional networks for 3d object detection on radar data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3060–3069, 2021.
  34. R. Nabati and H. Qi. Centerfusion: Center-based radar and camera fusion for 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536, 2021.
  35. Loci: privacy-aware, device-free, low-power localization of multiple persons using ir sensors. In 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pages 121–132. IEEE, 2020.
  36. A deep learning-based radar and camera sensor fusion architecture for object detection. In 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), pages 1–7. IEEE, 2019.
  37. NVIDIA. Nvidia jetson agx xavier developer kit, 2023.
  38. NVIDIA. Nvidia jetson xavier nx developer kit, 2023.
  39. R. Peng and M. L. Sichitiu. Angle of arrival localization for wireless sensor networks. In 2006 3rd Annual IEEE Communications Society on Sensor and Ad Hoc Communications and Networks, volume 1, pages 374–382, 2006.
  40. Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 444–453, 2021.
  41. Ros: an open-source robot operating system. In ICRA workshop on open source software, volume 3, page 5. Kobe, Japan, 2009.
  42. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  43. Millieye: A lightweight mmwave radar and camera fusion system for robust object detection. In Proceedings of the International Conference on Internet-of-Things Design and Implementation, IoTDI ’21, page 145–157, New York, NY, USA, 2021. Association for Computing Machinery.
  44. Improved orientation estimation and detection with hybrid object detection networks for automotive radar. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pages 111–117, 2022.
  45. Multispectral pedestrian detection using deep fusion convolutional neural networks. In ESANN, volume 587, pages 509–514, 2016.
  46. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11794–11803, 2021.
  47. Rodnet: Radar object detection using cross-modal supervision. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 504–513, 2021.
  48. Bayesian triplet loss: Uncertainty quantification in image retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12158–12168, 2021.
  49. Detectron2, 2019.
  50. Scene-aware learning network for radar object detection. In Proceedings of the 2021 International Conference on Multimedia Retrieval, ICMR ’21, page 573–579, New York, NY, USA, 2021. Association for Computing Machinery.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now