Papers
Topics
Authors
Recent
Search
2000 character limit reached

Beyond the Mud: Datasets and Benchmarks for Computer Vision in Off-Road Racing

Published 12 Feb 2024 in cs.CV | (2402.08025v1)

Abstract: Despite significant progress in optical character recognition (OCR) and computer vision systems, robustly recognizing text and identifying people in images taken in unconstrained \emph{in-the-wild} environments remain an ongoing challenge. However, such obstacles must be overcome in practical applications of vision systems, such as identifying racers in photos taken during off-road racing events. To this end, we introduce two new challenging real-world datasets - the off-road motorcycle Racer Number Dataset (RND) and the Muddy Racer re-iDentification Dataset (MUDD) - to highlight the shortcomings of current methods and drive advances in OCR and person re-identification (ReID) under extreme conditions. These two datasets feature over 6,300 images taken during off-road competitions which exhibit a variety of factors that undermine even modern vision systems, namely mud, complex poses, and motion blur. We establish benchmark performance on both datasets using state-of-the-art models. Off-the-shelf models transfer poorly, reaching only 15% end-to-end (E2E) F1 score on text spotting, and 33% rank-1 accuracy on ReID. Fine-tuning yields major improvements, bringing model performance to 53% F1 score for E2E text spotting and 79% rank-1 accuracy on ReID, but still falls short of good performance. Our analysis exposes open problems in real-world OCR and ReID that necessitate domain-targeted techniques. With these datasets and analysis of model limitations, we aim to foster innovations in handling real-world conditions like mud and complex poses to drive progress in robust computer vision. All data was sourced from PerformancePhoto.co, a website used by professional motorsports photographers, racers, and fans. The top-performing text spotting and ReID models are deployed on this platform to power real-time race photo search.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Zubaer Ahammed. 2018. Basketball player identification by jersey and number recognition. Ph. D. Dissertation. Brac University.
  2. Automatic Team Assignment and Jersey Number Recognition in Football Videos. INTELLIGENT AUTOMATION AND SOFT COMPUTING 36, 3 (2023), 2669–2684.
  3. Automatic number plate detection in vehicles using faster R-CNN. In 2020 International conference on system, computation, automation and networking (ICSCAN). IEEE, 1–6.
  4. DocFormer: End-to-End Transformer for Document Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 993–1003.
  5. Rowel Atienza. 2021. Vision Transformer for Fast and Efficient Scene Text Recognition. In Document Analysis and Recognition – ICDAR 2021, Josep Lladós, Daniel Lopresti, and Seiichi Uchida (Eds.). Springer International Publishing, Cham, 319–334.
  6. Character region awareness for text detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9365–9374.
  7. Darwin Bautista and Rowel Atienza. 2022. Scene Text Recognition with Permuted Autoregressive Sequence Models. In European Conference on Computer Vision. Springer Nature Switzerland, Cham, 178–196. https://doi.org/10.1007/978-3-031-19815-1_11
  8. Racing Bib Numbers Recognition.. In BMVC. 1–10.
  9. Knock, knock. Who’s there?–Identifying football player jersey numbers with synthetic data. arXiv preprint arXiv:2203.00734 (2022).
  10. Noppakun Boonsim. 2018. Racing bib number localization on complex backgrounds. WSEAS Transactions on Systems and Control 13 (2018), 226–231.
  11. Rung-Ching Chen et al. 2019a. Automatic License Plate Recognition via sliding-window darknet-YOLO deep learning. Image and Vision Computing 87 (2019), 47–56.
  12. Text Recognition in the Wild: A Survey. ACM Comput. Surv. 54, 2, Article 42 (mar 2021), 35 pages. https://doi.org/10.1145/3440756
  13. Learning resolution-invariant deep representations for person re-identification. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 8215–8222.
  14. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In Proceedings of the iEEE conference on computer vision and pattern recognition. 1335–1344.
  15. Chee Kheng Ch’ng and Chee Seng Chan. 2017. Total-text: A comprehensive dataset for scene text detection and recognition. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), Vol. 1. IEEE, 935–942.
  16. Yeong-Jun Cho and Kuk-Jin Yoon. 2016. Improving person re-identification via pose-aware multi-shot matching. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1354–1362.
  17. Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 994–1003.
  18. Soccer Player Recognition using Artificial Intelligence and Computer Vision. In 2022 IEEE International Conference on Electro Information Technology (eIT). IEEE, 477–481.
  19. Unsupervised person re-identification: Clustering and fine-tuning. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14, 4 (2018), 1–18.
  20. Person re-identification by symmetry-driven accumulation of local features. In 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 2360–2367.
  21. The HDA+ data set for research on fully automated re-identification systems. In Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part III 13. Springer, 241–255.
  22. More: a large-scale motorcycle re-identification dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 4034–4043.
  23. Masato Fujitake. 2024. Dtrocr: Decoder-only transformer for optical character recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 8025–8035.
  24. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430 (2021).
  25. Soccer player recognition using spatial constellation features and jersey number recognition. Computer Vision and Image Understanding 159 (2017), 105–115.
  26. Soccer Jersey Number Recognition Using Convolutional Neural Networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops.
  27. SoccerNet 2022 challenges results. In Proceedings of the 5th International ACM Workshop on Multimedia Content Analysis in Sports. 75–86.
  28. A systematic evaluation and benchmark for person re-identification: Features, metrics, and datasets. IEEE transactions on pattern analysis and machine intelligence 41, 3 (2018), 523–536.
  29. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  30. Transreid: Transformer-based object re-identification. In Proceedings of the IEEE/CVF international conference on computer vision. 15013–15022.
  31. TGCRBNW: A Dataset for Runner Bib Number Detection (and Recognition) in the Wild. In 2020 25th International Conference on Pattern Recognition (ICPR). 9445–9451. https://doi.org/10.1109/ICPR48806.2021.9412220
  32. Adversarially occluded samples for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5098–5107.
  33. Swintextspotter: Scene text spotting via better synergy between text detection and text recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4593–4603.
  34. Person re-identification with end-to-end scene text recognition. In Computer Vision: Second CCF Chinese Conference, CCCV 2017, Tianjin, China, October 11–14, 2017, Proceedings, Part III. Springer, 363–374.
  35. ICDAR 2013 robust reading competition. In 2013 12th international conference on document analysis and recognition. IEEE, 1484–1493.
  36. Open images v5 text annotation and yet another mask text spotter. In Asian Conference on Machine Learning. PMLR, 379–389.
  37. Automatic number plate recognition for motorcyclists riding without helmet. In 2018 International Conference on Current Trends towards Converging Technologies (ICCTCT). IEEE, 1–6.
  38. A robust real-time automatic license plate recognition based on the YOLO detector. In 2018 international joint conference on neural networks (ijcnn). IEEE, 1–10.
  39. An efficient and layout-independent automatic license plate recognition system based on the YOLO detector. IET Intelligent Transport Systems 15, 4 (2021), 483–503.
  40. Extraction and recognition of license plates of motorcycles and vehicles on highways. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., Vol. 4. IEEE, 356–359.
  41. SNIDER: Single noisy image denoising and rectification for improving license plate recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 0–0.
  42. Jiaze Li and Bin Liu. 2022. Rider Re-identification Based on Pyramid Attention. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, 81–93.
  43. Global-local temporal representations for video person re-identification. In Proceedings of the IEEE/CVF international conference on computer vision. 3958–3967.
  44. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740–755.
  45. Hengyue Liu and Bir Bhanu. 2019. Pose-guided R-CNN for jersey number recognition in sports. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 0–0.
  46. Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes. In Proceedings of the European conference on computer vision (ECCV). 67–83.
  47. Stefano Messelodi and Carla Maria Modena. 2013. Scene text recognition and tracking to identify athletes in sport videos. Multimedia tools and applications 63, 2 (2013), 521–545.
  48. CRNN based jersey-bib number/text recognition in sports and marathon images. In 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 1149–1156.
  49. Reading digits in natural images with unsupervised feature learning. (2011).
  50. TGC20ReId: A dataset for sport event re-identification in the wild. Pattern Recognition Letters 138 (2020), 355–361. https://doi.org/10.1016/j.patrec.2020.08.003
  51. Character Time-series Matching For Robust License Plate Recognition. In 2022 International Conference on Multimedia Analysis and Pattern Recognition (MAPR). IEEE, 1–6.
  52. Performance measures and a data set for multi-target, multi-camera tracking. In European conference on computer vision. Springer, 17–35.
  53. A review on various methodologies used for vehicle classification, helmet detection and number plate recognition. Evolutionary Intelligence 14, 2 (2021), 979–987.
  54. Player number localization and recognition in soccer video using hsv color space and internal contours. International Journal of Electrical and Computer Engineering 2, 7 (2008), 1408–1412.
  55. Helmet Detection And Number Plate Recognition Using Deep Learning. In 2022 IEEE Region 10 Symposium (TENSYMP). IEEE, 1–6.
  56. William Robson Schwartz and Larry S Davis. 2009. Learning discriminative appearance-based models using partial least squares. In 2009 XXII Brazilian symposium on computer graphics and image processing. IEEE, 322–329.
  57. Automated license plate recognition: a survey on methods and techniques. IEEE Access 9 (2020), 11203–11225.
  58. ASTER: An Attentional Scene Text Recognizer with Flexible Rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 9 (2019), 2035–2048. https://doi.org/10.1109/TPAMI.2018.2848939
  59. Transferring a semantic representation for person re-identification and search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4184–4193.
  60. A new multi-modal approach to bib number/text detection and recognition in Marathon images. pattern recognition 61 (2017), 479–491.
  61. Sergio Montazzolli Silva and Claudio Rosito Jung. 2018. License plate detection and recognition in unconstrained scenarios. In Proceedings of the European conference on computer vision (ECCV). 580–596.
  62. Detecting text in natural image with connectionist text proposal network. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer, 56–72.
  63. Deepsportradar-v1: Computer vision dataset for sports understanding with high quality annotations. In Proceedings of the 5th International ACM Workshop on Multimedia Content Analysis in Sports. 1–8.
  64. Multi-task learning for jersey number recognition in ice hockey. In Proceedings of the 4th International Workshop on Multimedia Content Analysis in Sports. 11–15.
  65. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140 (2016).
  66. End-to-end scene text recognition. In 2011 International conference on computer vision. IEEE, 1457–1464.
  67. Kai Wang and Serge Belongie. 2010. Word spotting in the wild. In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 11. Springer, 591–604.
  68. Convolutional Recurrent Neural Networks for Text Classification. In 2019 International Joint Conference on Neural Networks (IJCNN). 1–6. https://doi.org/10.1109/IJCNN.2019.8852406
  69. Re-identification of pedestrians with variable occlusion and scale. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, 1876–1882.
  70. Person re-identification by discriminative selection in video ranking. IEEE transactions on pattern analysis and machine intelligence 38, 12 (2016), 2501–2514.
  71. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 79–88.
  72. Athlete number detection on the basis of their face images. In 2017 International Conference on Biometrics and Kansei Engineering (ICBAKE). IEEE, 84–89.
  73. End-to-end deep learning for person search. arXiv preprint arXiv:1604.01850 2, 2 (2016), 4.
  74. Multiple players tracking and identification using group detection and player number recognition in sports video. In IECON 2013-39th Annual Conference of the IEEE Industrial Electronics Society. IEEE, 2442–2446.
  75. Deep learning for person re-identification: A survey and outlook. IEEE transactions on pattern analysis and machine intelligence 44, 6 (2021), 2872–2893.
  76. DeepSolo: Let Transformer Decoder With Explicit Points Solo for Text Spotting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 19348–19357.
  77. Deep metric learning for person re-identification. In 2014 22nd international conference on pattern recognition. IEEE, 34–39.
  78. Bike-person re-identification: a benchmark and a comprehensive evaluation. IEEE Access 6 (2018), 56059–56068.
  79. CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model. arXiv:2305.14014 [cs.CV]
  80. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision. 1116–1124.
  81. Person re-identification: Past, present and future. arXiv preprint arXiv:1610.02984 (2016).
  82. Omni-scale feature learning for person re-identification. In Proceedings of the IEEE/CVF international conference on computer vision. 3702–3712.
  83. EAST: An Efficient and Accurate Scene Text Detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now