CLIPping the Limits: Finding the Sweet Spot for Relevant Images in Automated Driving Systems Perception Testing (2404.05309v2)
Abstract: Perception systems, especially cameras, are the eyes of automated driving systems. Ensuring that they function reliably and robustly is therefore an important building block in the automation of vehicles. There are various approaches to test the perception of automated driving systems. Ultimately, however, it always comes down to the investigation of the behavior of perception systems under specific input data. Camera images are a crucial part of the input data. Image data sets are therefore collected for the testing of automated driving systems, but it is non-trivial to find specific images in these data sets. Thanks to recent developments in neural networks, there are now methods for sorting the images in a data set according to their similarity to a prompt in natural language. In order to further automate the provision of search results, we make a contribution by automating the threshold definition in these sorted results and returning only the images relevant to the prompt as a result. Our focus is on preventing false positives and false negatives equally. It is also important that our method is robust and in the case that our assumptions are not fulfilled, we provide a fallback solution.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” Feb. 2021. [Online]. Available: http://arxiv.org/abs/2103.00020
- X. Xia, Z. Meng, X. Han, H. Li, T. Tsukiji, R. Xu, Z. Zheng, and J. Ma, “An automated driving systems data acquisition and analytics platform,” Transportation Research Part C: Emerging Technologies, vol. 151, p. 104120, June 2023. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0968090X23001092
- C. King, L. Ries, J. Langner, and E. Sax, “A Taxonomy and Survey on Validation Approaches for Automated Driving Systems,” in 2020 IEEE International Symposium on Systems Engineering (ISSE). Vienna, Austria: IEEE, Oct. 2020, pp. 1–8. [Online]. Available: https://ieeexplore.ieee.org/document/9272219/
- M. Liu, E. Yurtsever, X. Zhou, J. Fossaert, Y. Cui, B. L. Zagar, and A. C. Knoll, “A Survey on Autonomous Driving Datasets: Data Statistic, Annotation, and Outlook,” Jan. 2024. [Online]. Available: http://arxiv.org/abs/2401.01454
- M. Alkhawlani and M. Elmogy, “Text-based, Content-based, and Semantic-based Image Retrievals: A Survey,” vol. 04, no. 01, 2015.
- M. Stefanini, M. Cornia, L. Baraldi, S. Cascianelli, G. Fiameni, and R. Cucchiara, “From Show to Tell: A Survey on Deep Learning-Based Image Captioning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 539–559, Jan. 2023. [Online]. Available: https://ieeexplore.ieee.org/document/9706348/
- M. Naito, C. Miyajima, T. Nishino, N. Kitaoka, and K. Takeda, “A browsing and retrieval system for driving data,” in 2010 IEEE Intelligent Vehicles Symposium. La Jolla, CA, USA: IEEE, June 2010, pp. 1159–1165. [Online]. Available: http://ieeexplore.ieee.org/document/5547999/
- L. Klitzke, C. Koch, A. Haja, and F. Köster, “Real-world Test Drive Vehicle Data Management System for Validation of Automated Driving Systems:,” in Proceedings of the 5th International Conference on Vehicle Technology and Intelligent Transport Systems. Heraklion, Crete, Greece: SCITEPRESS - Science and Technology Publications, 2019, pp. 171–180. [Online]. Available: https://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0007720501710180
- P. Rigoll, L. Ries, and E. Sax, “Scalable Data Set Distillation for the Development of Automated Driving Functions,” in 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). Macau, China: IEEE, Oct. 2022, pp. 3139–3145. [Online]. Available: https://ieeexplore.ieee.org/document/9921868/
- M. S. Lew, N. Sebe, C. Djeraba, and R. Jain, “Content-based multimedia information retrieval: State of the art and challenges,” ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 2, no. 1, pp. 1–19, Feb. 2006. [Online]. Available: https://dl.acm.org/doi/10.1145/1126004.1126005
- P. Rigoll, P. Petersen, H. Stage, L. Ries, and E. Sax, “Focus on the Challenges: Analysis of a User-friendly Data Search Approach with CLIP in the Automotive Domain,” in 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC). Bilbao, Spain: IEEE, Sept. 2023, pp. 168–174. [Online]. Available: https://ieeexplore.ieee.org/document/10422271/
- P. Rigoll, J. Langner, and E. Sax, “Unveiling Objects with SOLA: An Annotation-Free Image Search on the Object Level for Automotive Data Sets,” Dec. 2023. [Online]. Available: http://arxiv.org/abs/2312.01860
- National Institute of Standards and Technology (NIST), “COSINE DISTANCE, COSINE SIMILARITY, ANGULAR COSINE DISTANCE, ANGULAR COSINE SIMILARITY,” Mar. 2023. [Online]. Available: https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/cosdist.htm
- C. Sakaridis, D. Dai, and L. Van Gool, “ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada: IEEE, Oct. 2021, pp. 10 745–10 755. [Online]. Available: https://ieeexplore.ieee.org/document/9711067/
- H. Wang, Y. Zhan, L. Liu, L. Ding, and J. Yu, “Balanced Similarity with Auxiliary Prompts: Towards Alleviating Text-to-Image Retrieval Bias for CLIP in Zero-shot Learning,” Feb. 2024. [Online]. Available: http://arxiv.org/abs/2402.18400
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.