Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
12 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
37 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Dataset and Benchmark: Novel Sensors for Autonomous Vehicle Perception (2401.13853v1)

Published 24 Jan 2024 in cs.RO and cs.CV

Abstract: Conventional cameras employed in autonomous vehicle (AV) systems support many perception tasks, but are challenged by low-light or high dynamic range scenes, adverse weather, and fast motion. Novel sensors, such as event and thermal cameras, offer capabilities with the potential to address these scenarios, but they remain to be fully exploited. This paper introduces the Novel Sensors for Autonomous Vehicle Perception (NSAVP) dataset to facilitate future research on this topic. The dataset was captured with a platform including stereo event, thermal, monochrome, and RGB cameras as well as a high precision navigation system providing ground truth poses. The data was collected by repeatedly driving two ~8 km routes and includes varied lighting conditions and opposing viewpoint perspectives. We provide benchmarking experiments on the task of place recognition to demonstrate challenges and opportunities for novel sensors to enhance critical AV perception tasks. To our knowledge, the NSAVP dataset is the first to include stereo thermal cameras together with stereo event and monochrome cameras. The dataset and supporting software suite is available at: https://umautobots.github.io/nsavp

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. The International Journal of Robotics Research 39(12): 1367–1376.
  2. Alonso I and Murillo AC (2019) EV-SegNet: Semantic segmentation for event-based cameras. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Long Beach, CA, USA: IEEE, pp. 1624–1633.
  3. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, pp. 5297–5307.
  4. arXiv preprint arXiv:1711.01458 .
  5. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Vancouver, BC, Canada: IEEE, pp. 4016–4023.
  6. IEEE Transactions on Intelligent Vehicles : 1–10.
  7. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Long Beach, CA, USA: IEEE, pp. 1666–1675.
  8. IEEE Transactions on Intelligent Transportation Systems 19(3): 934–948.
  9. arXiv preprint arXiv:2001.08499 .
  10. Sensors 23(7): 3479.
  11. Farooq MA, Shariff W and Corcoran P (2023) Evaluation of thermal imaging on embedded GPU platforms for application in vehicular assistance systems. IEEE Transactions on Intelligent Vehicles 8(2): 1130–1144.
  12. Fischer T and Milford M (2020) Event-based visual place recognition with ensembles of temporal windows. IEEE Robotics and Automation Letters 5(4): 6924–6931.
  13. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, HI, USA: IEEE, pp. 2973–2983.
  14. Furgale P, Rehder J and Siegwart R (2013) Unified temporal and spatial calibration for multi-sensor systems. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. Tokyo, Japan: IEEE, pp. 1280–1286.
  15. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(1): 154–180.
  16. Garg S, Suenderhauf N and Milford M (2018) LoST? Appearance-invariant place recognition for opposite viewpoints using visual semantics. In: Proceedings of Robotics: Science and Systems. Pittsburgh, Pennsylvania: Robotics: Science and Systems Foundation.
  17. IEEE Robotics and Automation Letters 6(3): 4947–4954.
  18. The International Journal of Robotics Research 32(11): 1231–1237.
  19. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Vancouver, BC, Canada: IEEE, pp. 5108–5115.
  20. In: 2021 European Conference on Mobile Robots (ECMR). Bonn, Germany: IEEE, pp. 1–6.
  21. In: 2023 European Conference on Mobile Robots (ECMR). Coimbra, Portugal: IEEE, pp. 1–6.
  22. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). Rhodes, Greece: IEEE, pp. 1–6.
  23. Hu Y, Liu SC and Delbruck T (2021) v2e: From video frames to realistic DVS events. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Nashville, TN, USA: IEEE, pp. 1312–1321.
  24. arXiv preprint arXiv:2306.09014 .
  25. Remote Sensing and Spatial Information Sciences 37(B5): 829–834.
  26. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE, pp. 1037–1045.
  27. In: Infrared Technology and Applications XLV. Baltimore, MD, USA: SPIE, pp. 322–334.
  28. IEEE Robotics and Automation Letters 7(3): 6282–6289.
  29. Ligocki A, Jelinek A and Zalud L (2020) Brno urban dataset - the new data for self-driving agents and mapping tasks. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). Paris, France: IEEE, pp. 3284–3290.
  30. Matthies L and Shafer S (1987) Error modeling in stereo navigation. IEEE Journal on Robotics and Automation 3(3): 239–248.
  31. Milford MJ and Wyeth GF (2012) SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights. In: 2012 IEEE International Conference on Robotics and Automation. Saint Paul, MN, USA: IEEE, pp. 1643–1649.
  32. Sensors 15(6): 13851–13873.
  33. IEEE Transactions on Intelligent Transportation Systems : 1–11.
  34. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Nashville, TN, USA: IEEE, pp. 1403–1409.
  35. Olson E (2011) AprilTag: A robust and flexible visual fiducial system. In: 2011 IEEE International Conference on Robotics and Automation. Shanghai, China: IEEE, pp. 3400–3407.
  36. In: Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., pp. 16639–16652.
  37. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(6): 1964–1980.
  38. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Long Beach, CA, USA: IEEE, pp. 1684–1693.
  39. Shin U, Park J and Kweon IS (2023) Deep depth estimation from thermal image. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, BC, Canada: IEEE, pp. 1043–1053.
  40. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, pp. 1731–1740.
  41. Quantitative InfraRed Thermography Journal 14(2): 193–205.
  42. In: European Conference on Computer Vision. Tel Aviv, Israel: Springer, pp. 341–357.
  43. In: Proceedings of the on Thematic Workshops of ACM Multimedia. Mountain View, CA, USA: ACM, pp. 35–43.
  44. Teledyne FLIR (2018) Teledyne FLIR thermal dataset for algorithm training. Available at: https://www.flir.com/oem/adas/adas-dataset-form/ (accessed 5 December 2023).
  45. Torres SN and Hayat MM (2003) Kalman filtering for adaptive nonuniformity correction in infrared focal-plane arrays. Journal of the Optical Society of America A 20(3): 470–480.
  46. Tumas P, Nowosielski A and Serackis A (2020) Pedestrian detection in severe weather conditions. IEEE Access 8: 62775–62784.
  47. In: 11th International Conference on Quantitative Infrared Thermography. Naples, Italy, pp. 1–10.
  48. Valverde FR, Hurtado JV and Valada A (2021) There is more than meets the eye: Self-supervised multi-object detection and tracking with sound by distilling multimodal knowledge. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA: IEEE, pp. 11612–11621.
  49. Vertens J, Zürn J and Burgard W (2020) HeatNet: Bridging the day-night domain gap in semantic segmentation with thermal images. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas, NV, USA: IEEE, pp. 8461–8468.
  50. Infrared Physics & Technology 96: 199–208.
  51. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Kyoto, Japan: IEEE, pp. 3857–3864.
  52. arXiv preprint arXiv:2309.00962 .
  53. ISPRS Journal of Photogrammetry and Remote Sensing 196: 146–177.
  54. IEEE Robotics and Automation Letters 3(3): 2032–2039.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now