Surveying You Only Look Once (YOLO) Multispectral Object Detection Advancements, Applications And Challenges (2409.12977v1)

Abstract: Multispectral imaging and deep learning have emerged as powerful tools supporting diverse use cases from autonomous vehicles, to agriculture, infrastructure monitoring and environmental assessment. The combination of these technologies has led to significant advancements in object detection, classification, and segmentation tasks in the non-visible light spectrum. This paper considers 400 total papers, reviewing 200 in detail to provide an authoritative meta-review of multispectral imaging technologies, deep learning models, and their applications, considering the evolution and adaptation of You Only Look Once (YOLO) methods. Ground-based collection is the most prevalent approach, totaling 63% of the papers reviewed, although uncrewed aerial systems (UAS) for YOLO-multispectral applications have doubled since 2020. The most prevalent sensor fusion is Red-Green-Blue (RGB) with Long-Wave Infrared (LWIR), comprising 39% of the literature. YOLOv5 remains the most used variant for adaption to multispectral applications, consisting of 33% of all modified YOLO models reviewed. 58% of multispectral-YOLO research is being conducted in China, with broadly similar research quality to other countries (with a mean journal impact factor of 4.45 versus 4.36 for papers not originating from Chinese institutions). Future research needs to focus on (i) developing adaptive YOLO architectures capable of handling diverse spectral inputs that do not require extensive architectural modifications, (ii) exploring methods to generate large synthetic multispectral datasets, (iii) advancing multispectral YOLO transfer learning techniques to address dataset scarcity, and (iv) innovating fusion research with other sensor types beyond RGB and LWIR.

• The paper surveys around 400 studies from 2020-2024, providing a comprehensive review of advancements in YOLO-based multispectral object detection methods.
• A key finding is the prevalence of RGB-LWIR fusion and the dominance of YOLOv5 as the most used YOLO variant in surveyed multispectral applications.
• A major challenge highlighted is the scarcity of publicly available annotated multispectral datasets, limiting standardization and relying on custom collections.

The paper, titled "Surveying You Only Look Once (YOLO) Multispectral Object Detection Advancements, Applications and Challenges," offers a comprehensive review of the advancements in multispectral object detection using the YOLO (You Only Look Once) architecture, a popular convolutional neural network (CNN) model known for its real-time object detection capabilities. This work surveys around 400 papers, reviewing 200 in detail, to provide an authoritative overview of multispectral imaging technologies and their intersection with deep learning models, with a notable focus on YOLO adaptations from 2020 to 2024.

Key Survey Insights

Multispectral Imaging and YOLO

• Sensor Fusion: The survey highlights that RGB (Red-Green-Blue) and LWIR (Long-Wave Infrared) fusion is prevalent in the literature, with 39% of works focusing on this combination to enhance detection capabilities in various visibility conditions.
• YOLO Variants: YOLOv5 emerges as the most employed variant for multispectral applications, representing 33% of the modified YOLO models surveyed. The survey showcases the adaptability of YOLO architectures, particularly the trend of moving from single to dual-stream models to process multispectral data effectively.

Geographic and Research Trends

• Chinese Research Dominance: A significant majority (58%) of the research reviewed originates from Chinese institutions, underscoring their strong focus on this field. This is reflected in the slight edge in average journal impact factor for papers from China compared to those from other countries.
• Platform Utilization: Ground-based collection is the most common platform, used in 63% of the reviewed studies, followed by a noteworthy increase in the employment of unmanned aerial systems (UAS).

Key Architectural Innovations

• Dual-Stream Architectures: Advances such as MOD-YOLO and GMD-YOLO, employing dual-stream networks for separate processing of visible and thermal data, exemplify key adaptations to enhance multispectral object detection. These models improve mAP performance by 4.8% and 3.6%, respectively, over traditional YOLO frameworks, showcasing significant architectural refinement.
• Attention Mechanisms and Transformer Integrations: Innovative approaches using these technologies allow more sophisticated data fusion and adaptability to varying conditions, with models like TF-YOLO leveraging transformer fusion to achieve superior illumination adaptability, outperforming YOLOv7 by significant mAP margins.

Challenges and Future Directions

• Dataset Scarcity: The paper points out a critical challenge in the limited availability of publicly accessible annotated multispectral datasets, emphasizing the reliance on custom datasets, which limits standardization in evaluation metrics.
• Proposed Future Research:
  • Development of architectures that adapt flexibly to varied spectral inputs.
  • Generation of large synthetic datasets through methods like GANs and physics-based modeling to overcome dataset scarcity.
  • Advancement of transfer learning and unsupervised learning to reduce dependency on large labeled datasets.
  • Exploration of sensor fusion beyond traditional RGB-LWIR combinations for more comprehensive multispectral capabilities.

Conclusion

The extensive review reveals significant YOLO multispectral object detection advancements, highlighting architectural innovation, particularly in improving detection under challenging environmental conditions and integrating additional spectral data. Future research directions are recommended to further address existing challenges and expand the applicability of these technologies across diverse domains, reinforcing the pivotal role of multispectral object detection in multiple industries, including agriculture, autonomous vehicles, and defense.