FlightScope: An Experimental Comparative Review of Aircraft Detection Algorithms in Satellite Imagery

Abstract: Object detection in remotely sensed satellite pictures is fundamental in many fields such as biophysical, and environmental monitoring. While deep learning algorithms are constantly evolving, they have been mostly implemented and tested on popular ground-based taken photos. This paper critically evaluates and compares a suite of advanced object detection algorithms customized for the task of identifying aircraft within satellite imagery. Using the large HRPlanesV2 dataset, together with a rigorous validation with the GDIT dataset, this research encompasses an array of methodologies including YOLO versions 5 and 8, Faster RCNN, CenterNet, RetinaNet, RTMDet, and DETR, all trained from scratch. This exhaustive training and validation study reveal YOLOv5 as the preeminent model for the specific case of identifying airplanes from remote sensing data, showcasing high precision and adaptability across diverse imaging conditions. This research highlight the nuanced performance landscapes of these algorithms, with YOLOv5 emerging as a robust solution for aerial object detection, underlining its importance through superior mean average precision, Recall, and Intersection over Union scores. The findings described here underscore the fundamental role of algorithm selection aligned with the specific demands of satellite imagery analysis and extend a comprehensive framework to evaluate model efficacy. The benchmark toolkit and codes, available via https://github.com/toelt-llc/FlightScope_Bench, aims to further exploration and innovation in the realm of remote sensing object detection, paving the way for improved analytical methodologies in satellite imagery applications.

• The paper demonstrates that YOLOv5 achieves higher AP, mAP, and IoU metrics compared to other algorithms for aircraft detection in satellite imagery.
• The study employed HRPlanesV2 and GDIT datasets with standardized training on NVIDIA RTX A6000 GPUs, ensuring a robust and fair comparison.
• Inference analysis reveals that advanced YOLO models minimize false positives and achieve high recall, significantly enhancing remote sensing accuracy.

FlightScope: An Experimental Comparative Review of Aircraft Detection Algorithms in Satellite Imagery

The paper "FlightScope: An Experimental Comparative Review of Aircraft Detection Algorithms in Satellite Imagery" provides an exhaustive evaluation of advanced object detection algorithms tailored for aircraft identification within satellite imagery. This research explores and contrasts models including YOLOv5, YOLOv8, Faster RCNN, CenterNet, RetinaNet, RTMDet, and DETR.

Methodology

The study utilizes the HRPlanesV2 dataset for training, which features a significant variety of aircraft and conditions, supplemented by validation and testing on the GDIT dataset to ensure robust evaluation across different imaging scenarios (Figure 1).

Figure 1: Flowchart of the FlightScope comparative study: The training is performed on HRPlanesV2 dataset and the Validation and Test conducted on HRPlanesV2 and GDIT Aerial airport datasets.

The training setup leverages significant computational resources, including three NVIDIA RTX A6000 GPUs, enabling extensive parameter tuning and hyperparameter exploration across the models. Training configurations are standardized, allowing a fair comparison of model performance.

Results

The evaluation metrics used in this study include Average Precision (AP), Recall, and Intersection over Union (IoU), metrics crucial for assessing performance in object detection tasks.

The performance analysis reveals that YOLOv5 consistently outperforms other methods, demonstrating high mean average precision (mAP) and mAP50 (Figure 2).

Figure 2: Comparison of bounding box mean average precision (mAP) curves for trained object detection algorithms. To the left raw figures of the curves, the right figures are magnifications from epoch 450 to 500. (a) Represents the mAP. (b) Illustrates the mAP50.

In contrast, SSD struggles in maintaining competitive mAP and mAP50 scores, indicating limitations in detecting aircraft accurately in diverse satellite imagery conditions. Bounding box and total loss curves (Figure 3) further substantiate the training performance of the models, highlighting their convergence behavior.

Figure 3: Bounding box loss.

Analysis on GDIT Dataset

Performance evaluation on the GDIT dataset reinforces the robustness of YOLOv5 and YOLOv8, with YOLOv5 achieving the highest AP and IoU scores across various image subsets (Figure 4).

Figure 4: Estimated evaluation metrics when inferencing the 8 models (initially trained on HRPlanesV2) on all images from unseen subsets 'Train' (a), 'Test' (b) and 'Validation' (c) from GDIT aircraft dataset.

Inference examples illustrate the models' capabilities and challenges in detecting aircraft, with YOLO models notably minimizing false positives and achieving high recall in diverse conditions (Figures 13 and 14).

Figure 5: Inference examples of YOLOv8, DETR, SSD and CenterNet on unseen images from Google Earth HRPlanesv2 dataset and Airbus GDIT. FP' stands for False Positives,ND' for No Detection and `IE' for Inaccurate Estimation.

Figure 6: Inference examples of YOLOv5, RTMDet, RetinaNet and Faster-RCNN on another set of unseen images from Google Earth HRPlanesv2 dataset and Airbus GDIT. FP' stands for False Positives,ND' for No Detection.

Conclusion

This exhaustive comparative assessment demonstrates that the YOLOv5 emerges as the most suitable algorithm for detecting aircraft in satellite imagery, largely due to its superior performance in AP, recall, and IoU scores. These findings underscore the critical importance of algorithm selection in satellite image analysis, with broader implications for future advancements in remote sensing and object detection methodologies.