Anti-UAV Benchmark for Detection & Tracking

• Anti-UAV Benchmark is a rigorously constructed dataset and evaluation protocol designed to benchmark algorithms for UAV detection, localization, and tracking.
• It features detailed annotations, varied environmental conditions, and multi-modal sensor data to simulate operational challenges.
• Evaluations use metrics like IoU, mAP, AUC, and precision to objectively assess algorithm performance and highlight strengths and limitations.

An anti-UAV benchmark is a rigorously constructed dataset and associated evaluation protocol for the development and comparison of algorithms dedicated to the detection, localization, and tracking of unmanned aerial vehicles (UAVs) in operationally realistic environments. These benchmarks are indispensable for advancing automated aerial security, enabling objective assessment of both detection and tracking algorithms under systematically varied conditions such as lighting, background complexity, object scale, and sequence duration.

1. Benchmark Structure and Dataset Composition

Anti-UAV benchmarks are characterized by carefully curated data splits, annotation protocols, and environmental variability. For example, the DUT Anti-UAV benchmark comprises two complementary high-quality visible-light subsets: a detection subset with 10,000 manually annotated images (5,243 train, 2,621 val, 2,245 test; single "UAV" class), and a tracking subset of 20 videos totaling 24,804 frames (average ∼1,240 frames/video), captured across urban, rural, and skyline backgrounds, and annotated per-frame with tight bounding-boxes cross-checked by trained annotators. Object appearance varies from 2×10⁻⁶ up to 0.7 of image area, simulating a diverse range of distances and target signatures. Lighting spans day, night, dawn, and dusk conditions (Zhao et al., 2022).

Other benchmarks, such as MMAUD, focus on multi-modal sensor fusion, providing 50 sequences (1,700 s, ∼28 min) with synchronized stereo RGB, multiple LiDARs, radar, and audio arrays, and 3D sub-centimeter Leica ground truth for detection, classification (5 UAV types), and trajectory estimation, specifically accommodating real-world scenarios with ambient machinery noise and urban clutter (Yuan et al., 2024).

2. Annotation Formats and Environmental Diversity

Annotations are standardized to support replicable algorithm development. In DUT Anti-UAV, all bounding boxes share a (x, y, width, height) coordinate format; classes are limited to a single "UAV" label, and annotation was performed by trained teams with consistency cross-checks. MMAUD expands annotation to 2D COCO-style bounding boxes, 3D point-cloud centroids, and COCO/NumPy for pose and class, including full confidence and timestamp metadata. In all cases, environmental variables such as lighting (day/night/dusk/dawn), object scale (far/persistent target), and background (urban, rural, skyline, machinery noise) are chosen to maximize challenge diversity. For tracking, DUT Anti-UAV includes both short-term and long-term video sequences, with ground-truth bounding box available for initialization; in MMAUD, every sensor stream is rigidly time-synced to high-precision Leica total-station ground truth.

3. Evaluation Protocols and Metrics

Evaluation is formally specified to ensure unambiguous and fair assessment:

• Detection: Intersection over Union (IoU) and mean Average Precision (mAP) are adopted as primary metrics. For a predicted box $B_p$ and ground-truth $B_g$, $$\text{IoU} = \frac{|B_p \cap B_g|}{|B_p \cup B_g|}, \qquad \text{mAP} = \frac{1}{C}\sum_{c=1}^C \mathrm{AP}_c$$ where $\mathrm{AP}_c$ is area under the precision–recall curve for class $c$.
• Tracking: Success (AUC, area under overlap-threshold curve), Precision (percentage of center-location errors < threshold, e.g., 20 px), and Normalized Precision (area under error curve normalized to [0, 0.5] of image size) are standard. For multi-object scenarios, the Multiple Object Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP), and IDF1 are reported:

$$\mathrm{MOTA} = 1 - \frac{\sum_t(\mathrm{FN}_t + \mathrm{FP}_t + \mathrm{IDSW}_t)}{\sum_t \mathrm{GT}_t},\quad \mathrm{MOTP} = \frac{\sum_{t,i} d_{t,i}}{\sum_t c_t}$$

where $d_{t,i}$ denotes spatial error between predictions and ground truths, and $c_t$ is count of correct matches at time $t$. (Zhao et al., 2022, Yuan et al., 2024)

4. Baseline Algorithms and Implementation

Typical anti-UAV benchmarks provide baseline results using leading detection and tracking architectures:

• Detection: Two-stage (Faster R-CNN, Cascade R-CNN, ATSS) and one-stage (SSD, YOLOX, YOLOv5, YOLOv12) detectors with standard backbones (VGG16, ResNet-18/50, DarkNet).
• Tracking: Short-term trackers such as SiamFC, SiamRPN++, ECO, ATOM, DiMP, and TransT; long-term trackers including SPLT and LTMU; plus data association methods when extending to multi-object tracking.
• Training Regimen: Detection models use batch size 16/GPU, 12 epochs, initial learning rate 0.002 with stepwise decay, random horizontal flips and multiscale color jitter. Anchor configurations are tuned to small UAV sizes, e.g., scales {32, 64, 128}. VSI metric optimization ensures detector sensitivity to small, low-contrast UAVs.
• Fusion: Tracking is augmented by fusing detector outputs when tracker confidence is low—a simple threshold-based rule that triggers detection when the tracker's score falls below $\tau_t$. Pseudo-code:

  if score_t < τ_t: detections = D.detect(frame_t) if score_d > max(τ_d, score_t): result = bbox_d else: result = bbox_t else: result = bbox_t

  Performance is robust to $\tau \in [0.5,0.95]$ (Zhao et al., 2022).

5. Performance Analysis and Limitations

Empirical evaluation reveals that two-stage detectors (e.g., Cascade R-CNN + ResNet50) yield highest mAP (0.683 at 10.7 FPS), whereas one-stage detectors (e.g., YOLOX + DarkNet) achieve greater throughput (51.3 FPS) at the expense of accuracy (mAP = 0.552). When fusing detection and tracking—e.g., LTMU tracker with detection fallback—tracking success improves significantly (from 0.608 to 0.664, normalized precision from 0.783 to 0.865).

Failure modes are consistent across benchmarks:

• Extremely distant or small UAVs (<0.1% image area) degrade both detection and tracking confidence.
• Background clutter (wires, branches) increases false positives in one-stage detection.
• Motion blur and night scenes reduce both detection and tracking reliability.
• Computationally, detection–tracking fusion incurs low overhead (detector invoked in ~20% of frames), and one-stage detectors enable real-time speed on GPUs.

Acknowledged limitations include: single-class labeling (precluding multi-object scenarios), restriction to visible-light imagery (no IR or radar), and heuristic confidence thresholding that may not generalize (Zhao et al., 2022).

6. Recommendations and Future Directions

Key findings suggest that high-precision detectors are essential for rescuing trackers under target loss, and that diverse, challenging datasets improve generalization. Fusion of even elementary detector and tracker modules delivers consistent performance gains.

Suggested future work includes:

• Expanding benchmarks to multi-class and multi-object tracking (e.g., distinguishing birds from UAVs).
• Integrating multi-modal sensing (e.g., infrared, LiDAR) for adverse lighting and night operations.
• Adaptive, learned thresholding for tracker–detector collaboration.
• Moving toward end-to-end detection–tracking networks, for example leveraging transformers specifically for small aerial targets (Zhao et al., 2022, Yuan et al., 2024).

The anti-UAV benchmark thus provides a rigorous, extensible foundation for the quantitative assessment and development of robust drone detection and tracking algorithms in dynamic, real-world environments.