Papers
Topics
Authors
Recent
Search
2000 character limit reached

MMW-Carry: Enhancing Carry Object Detection through Millimeter-Wave Radar-Camera Fusion

Published 24 Feb 2024 in eess.SP | (2402.15897v1)

Abstract: This paper introduces MMW-Carry, a system designed to predict the probability of individuals carrying various objects using millimeter-wave radar signals, complemented by camera input. The primary goal of MMW-Carry is to provide a rapid and cost-effective preliminary screening solution, specifically tailored for non-super-sensitive scenarios. Overall, MMW-Carry achieves significant advancements in two crucial aspects. Firstly, it addresses localization challenges in complex indoor environments caused by multi-path reflections, enhancing the system's overall robustness. This is accomplished by the integration of camera-based human detection, tracking, and the radar-camera plane transformation for obtaining subjects' spatial occupancy region, followed by a zooming-in operation on the radar images. Secondly, the system performance is elevated by leveraging long-term observation of a subject. This is realized through the intelligent fusion of neural network results from multiple different-view radar images of an in-track moving subject and their carried objects, facilitated by a proposed knowledge-transfer module. Our experiment results demonstrate that MMW-Carry detects objects with an average error rate of 25.22\% false positives and a 21.71\% missing rate for individuals moving randomly in a large indoor space, carrying the common-in-everyday-life objects, both in open carry or concealed ways. These findings affirm MMW-Carry's potential to extend its capabilities to detect a broader range of objects for diverse applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. H.-M. Chen, S. Lee, R. Rao, M.-A. Slamani, and P. Varshney, “Imaging for concealed weapon detection: a tutorial overview of development in imaging sensors and processing,” IEEE Signal Processing Magazine, vol. 22, no. 2, pp. 52–61, 2005.
  2. TSA. (2022) Imaging technology. [Online]. Available: https://web.archive.org/web/20100106043039/http://www.tsa.gov/approach/tech/imaging_technology.shtm
  3. D. M. M. Roomi and R.Rajashankari, “Detection of concealed weapons in x-ray images using fuzzy k-nn,” International Journal of Computer Science, Engineering and Information Technology (IJCSEIT), vol. 2, no. 2, pp. 187–196, Apr. 2012.
  4. P. Varshney, H.-M. Chen, L. Ramac, M. Uner, D. Ferris, and M. Alford, “Registration and fusion of infrared and millimeter wave images for concealed weapon detection,” in Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), vol. 3, 1999, pp. 532–536 vol.3.
  5. K. B. Cooper, R. J. Dengler, N. Llombart, T. Bryllert, G. Chattopadhyay, E. Schlecht, J. Gill, C. Lee, A. Skalare, I. Mehdi, and P. H. Siegel, “Penetrating 3-d imaging at 4- and 25-m range using a submillimeter-wave radar,” IEEE Transactions on Microwave Theory and Techniques, vol. 56, no. 12, pp. 2771–2778, 2008.
  6. P. S. K. Bandyopadhyay, B. Datta, and S. Roy, “Identifications of concealed weapon in a human body,” 2012, available at https://arxiv.org/ftp/arxiv/papers/1210/1210.5653.pdf.
  7. X. Gao, G. Xing, S. Roy, and H. Liu, “Ramp-cnn: A novel neural network for enhanced automotive radar object recognition,” IEEE Sensors Journal, vol. 21, no. 4, pp. 5119–5132, 2021.
  8. X. Gao, H. Liu, S. Roy, G. Xing, A. Alansari, and Y. Luo, “Learning to detect open carry and concealed object with 77 ghz radar,” IEEE journal of selected topics in signal processing, vol. 16, no. 4, pp. 791–803, 2022.
  9. S. Yeom, D.-S. Lee, J.-Y. Son, and S.-H. Kim, “Concealed object detection using passive millimeter wave imaging,” in 2010 4th International Universal Communication Symposium, 2010, pp. 383–386.
  10. J. Liu, K. Zhang, Z. Sun, Q. Wu, W. He, and H. Wang, “Concealed object detection and recognition system based on millimeter wave fmcw radar,” Applied Sciences, vol. 11, no. 19, 2021.
  11. X. Zhuge and A. G. Yarovoy, “A sparse aperture mimo-sar-based uwb imaging system for concealed weapon detection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 49, no. 1, pp. 509–518, 2011.
  12. X. Wang, S. Gou, X. Wang, Y. Zhao, and L. Zhang, “Patch-based gaussian mixture model for concealed object detection in millimeter-wave images,” in TENCON 2018 - 2018 IEEE Region 10 Conference, 2018, pp. 2522–2527.
  13. T. Zheng, Z. Chen, J. Luo, L. Ke, C. Zhao, and Y. Yang, “Siwa: See into walls via deep uwb radar,” in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, ser. MobiCom ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 323–336. [Online]. Available: https://doi.org/10.1145/3447993.3483258
  14. X. Gao, S. Roy, and G. Xing, “Mimo-sar: A hierarchical high-resolution imaging algorithm for mmwave fmcw radar in autonomous driving,” IEEE Transactions on Vehicular Technology, vol. 70, no. 8, pp. 7322–7334, 2021.
  15. X. Gao, S. Roy, G. Xing, and S. Jin, “Perception through 2d-mimo fmcw automotive radar under adverse weather,” in 2021 IEEE International Conference on Autonomous Systems (ICAS), 2021, pp. 1–5.
  16. X. Gao, G. Xing, S. Roy, and H. Liu, “Experiments with mmwave automotive radar test-bed,” in 2019 53rd Asilomar Conference on Signals, Systems, and Computers, 2019, pp. 1–6.
  17. Y. Li, Z. Peng, R. Pal, and C. Li, “Potential active shooter detection based on radar micro-doppler and range-doppler analysis using artificial neural network,” IEEE Sensors Journal, vol. 19, no. 3, pp. 1052–1063, 2019.
  18. Z. Zhang, X. Di, Y. Xu, and J. Tian, “Concealed dangerous object detection based on a 77ghz radar,” in 2018 IEEE International Workshop on Electromagnetics:Applications and Student Innovation Competition (iWEM), 2018, pp. 1–2.
  19. A. Sonny, A. Kumar, and L. R. Cenkeramaddi, “Carry object detection utilizing mmwave radar sensors and ensemble-based extra tree classifiers on the edge computing systems,” IEEE Sensors Journal, vol. 23, no. 17, pp. 20 137–20 149, 2023.
  20. T. Liu, Y. Zhao, Y. Wei, Y. Zhao, and S. Wei, “Concealed object detection for activate millimeter wave image,” IEEE Transactions on Industrial Electronics, vol. 66, no. 12, pp. 9909–9917, 2019.
  21. C. Wang, J. Shi, Z. Zhou, L. Li, Y. Zhou, and X. Yang, “Concealed object detection for millimeter-wave images with normalized accumulation map,” IEEE Sensors Journal, vol. 21, no. 5, pp. 6468–6475, 2021.
  22. M. T. Bhatti, M. G. Khan, M. Aslam, and M. J. Fiaz, “Weapon detection in real-time cctv videos using deep learning,” IEEE Access, vol. 9, pp. 34 366–34 382, 2021.
  23. O. Bazgir, D. Nolte, S. R. Dhruba, Y. Li, C. Li, S. Ghosh, and R. Pal, “Active shooter detection in multiple-person scenario using rf-based machine vision,” IEEE Sensors Journal, vol. 21, no. 3, pp. 3609–3622, 2021.
  24. 3GPP and 38.101, “5g; nr; user equipment (ue) radio transmission and reception,” Part 2: Range 2 Standalone, 2 Version 15.2.0 Release 15, 2018.
  25. R. Nitzberg, “Constant-false-alarm-rate signal processors for several types of interference,” IEEE Transactions on Aerospace and Electronic Systems, no. 1, pp. 27–34, 1972.
  26. A. Coates and A. Y. Ng, “Multi-camera object detection for robotics,” in 2010 IEEE International Conference on Robotics and Automation, 2010, pp. 412–419.
  27. X. Gao, S. Roy, and L. Zhang, “Static background removal in vehicular radar: Filtering in azimuth-elevation-doppler domain,” arXiv preprint arXiv:2307.01444, 2023.
  28. H. Krim and M. Viberg, “Two decades of array signal processing research: the parametric approach,” IEEE Signal Processing Magazine, vol. 13, no. 4, pp. 67–94, 1996.
  29. P.-J. Chung, M. Viberg, and J. Yu, “Chapter 14 - doa estimation methods and algorithms,” in Academic Press Library in Signal Processing: Volume 3, ser. Academic Press Library in Signal Processing, A. M. Zoubir, M. Viberg, R. Chellappa, and S. Theodoridis, Eds.   Elsevier, 2014, vol. 3, pp. 599–650. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B978012411597200014X
  30. J. McWirter and T. Shepherd, “A systolic array processor for mvdr beamforming,” in IEE Colloquium on Adaptive Antennas, 1990, pp. 4/1–4/2.
  31. R. Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE Transactions on Antennas and Propagation, vol. 34, no. 3, pp. 276–280, 1986.
  32. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part based models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 9, pp. 1627–1645, 2010.
  33. R. E. Kalman, “A New Approach to Linear Filtering and Prediction Problems,” Journal of Basic Engineering, vol. 82, no. 1, pp. 35–45, 03 1960. [Online]. Available: https://doi.org/10.1115/1.3662552
  34. X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 10 359–10 366.
  35. R. Jonker and A. Volgenant, “A shortest augmenting path algorithm for dense and sparse linear assignment problems,” Computing, vol. 38, pp. 325–340, 2005.
  36. S. Sugimoto, H. Tateda, H. Takahashi, and M. Okutomi, “Obstacle detection using millimeter-wave radar and its visualization on image sequence,” in Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., vol. 3, 2004, pp. 342–345 Vol.3.
  37. J. Oh, K.-S. Kim, M. Park, and S. Kim, “A comparative study on camera-radar calibration methods,” in 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2018, pp. 1057–1062.
  38. J. Li and P. Stoica, “Mimo radar with colocated antennas,” IEEE Signal Processing Magazine, vol. 24, no. 5, pp. 106–114, 2007.
  39. M. A. Richards, “Fundamentals of radar signal processing.”   US: McGraw-Hill Professional, 2005, pp. –1. [Online]. Available: https://mhebooklibrary.com/doi/book/10.1036/0071444742
  40. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  41. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” 2020. [Online]. Available: https://arxiv.org/abs/2005.12872
  42. X. Gao, S. Roy, H. Liu, Y. Luo, and G. Xing, “Raw adc data of 2d-mimo mmwave radar for carry object detection,” 2022. [Online]. Available: https://dx.doi.org/10.21227/begn-ye78
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up