UAVBench: Benchmarking for Aerial Robotics

• Unmanned Aerial Vehicle Benchmarks are standardized datasets and protocols that enable systematic evaluation of UAV algorithms in both simulated and real-world environments.
• They incorporate diverse sensor modalities such as RGB, thermal, LiDAR, radar, and RF, paired with detailed annotations and precise evaluation metrics like IoU, mAP, and tracking success rates.
• UAVBench supports comprehensive analysis in tracking, detection, 3D perception, and agentic reasoning, paving the way for robust advancements in drone autonomy and control.

Unmanned Aerial Vehicle Benchmarks ("UAVBench") denote a family of standardized datasets, protocols, and experimental test suites for streaming the development, evaluation, and comparative analysis of perception, optimization, communication, and autonomy methods in drone and aerial robotics research. UAVBench frameworks encompass both real-world and synthetic data, multi-modal sensing, scenario generation, agentic reasoning, and domain-specific evaluation metrics, supporting robust empirical scrutiny of UAV algorithms and systems across a range of operational environments, hardware, and software stacks.

1. Dataset Architectures and Modalities

Leading UAVBench initiatives provide comprehensive multi-modal datasets suitable for tracking, detection, identification, 3D reconstruction, path planning, and agentic reasoning. For example, the Anti-UAV benchmark consists of 318 unaligned RGB–thermal video pairs totaling ∼586K frames, each frame manually annotated with bounding boxes, existence flags, and seven per-frame attributes (OV, OC, FM, SV, LI, TC, LR) (Jiang et al., 2021). Modalities across benchmarks include visible-band RGB, thermal IR, LiDAR, radar, DVS (Dynamic Vision Sensor), spectrometer-based hyperspectral cubes, and raw RF signal captures (Shi et al., 12 Mar 2025, Lekhak et al., 3 Oct 2025, Zou et al., 27 Nov 2025).

Dataset splits are defined to support cross-validation: Anti-UAV provides training/validation/test divisions (160/67/91 video pairs), whereas UAV-MM3D offers 400K synchronized multi-sensor synthetic frames partitioned by scene, modality, and weather (Zou et al., 27 Nov 2025). WebUAV-3M delivers 3.3M annotated frames over 4,500 videos, leveraging semi-automatic annotation (SATA) for large-scale bounding-box generation (Zhang et al., 2022).

A unified feature of advanced UAVBench datasets is rich attribute-level annotation: Anti-UAV and CST Anti-UAV encode both box-level and multi-attribute flags per frame, enabling fine-grained challenge analysis (tiny targets, scale/thermal variation, occlusion) (Xie et al., 31 Jul 2025).

2. Evaluation Metrics and Annotation Protocols

Modern UAVBench protocols standardize evaluation through precise metrics:

Tracking Metrics:

• Precision: $$P = (1/N) \sum_{i=1}^N \mathbb{I}(\mathrm{dist}(b_i^{pred},b_i^{gt}) < \tau)$$, typically $\tau = 20$ px.
• Success Rate (IoU): $$S = (1/N)\sum_{i=1}^N \mathbb{I}(\mathrm{IoU}(b_i^{pred},b_i^{gt}) > T)$$, for $T=0.5$ or AUC over $T\in[0,1]$.
• UAV-state accuracy (mSA): mSA = $(1/T)\sum_{t=1}^T$ [IoU if visible, $p_t$ if absent].

Detection and Identification:

• Mean Average Precision (mAP) and its variants across size categories (AP_S, AP_M, AP_L).
• Multiple Object Tracking Accuracy (MOTA), IDF1, and error statistics for association tasks (Isaac-Medina et al., 2021, Zhang et al., 2022).

RF-based Identification:

• Overall Accuracy (ACC), Precision, Recall, F1-score, and SNR-stratified accuracies; key recommendation: spectrogram window STFTP ≈ 256, ViT-L16 architecture (Shi et al., 12 Mar 2025).

3D Perception Benchmarks:

• Average Precision (AP), Mean AP (mAP), MOT accuracy, geometric reconstruction error ($E_{geom}$), and photometric consistency error ($E_{photo}$), with explicit trajectory forecasting metrics (ADE, FDE) (Zou et al., 27 Nov 2025, Ye et al., 14 Oct 2024).

Optimization/Planning:

• Relative error, Friedman ranks, Wilcoxon and Holm–Bonferroni post-hoc tests across functional evaluation budgets and variable path planning dimensions (Shehadeh et al., 24 Jan 2025).

All annotations are governed by progressive, multi-stage QA protocols (e.g., Anti-UAV’s three-stage annotation workflow, CST Anti-UAV’s multi-pass attribute checking by expert annotators) to minimize semantic drift and labeling errors.

3. Benchmark Task Coverage and Challenges

UAVBench suites are designed to exercise broad functional capabilities, including but not limited to:

• Single/multi-object tracking across realistic, adverse, and multi-modal environments.
• Object detection and category discrimination under tiny-object, high-occlusion, and thermal crossover regimes.
• RF signal identification and fingerprinting (RFUAV) across 37 UAV types and multi-SNR spectra (Shi et al., 12 Mar 2025).
• 3D object detection, 6-DoF pose estimation, and collaborative tracking/detection in multi-UAV swarms (UAV3D, UAV-MM3D) (Ye et al., 14 Oct 2024, Zou et al., 27 Nov 2025).
• Landmine and UXO detection via hyperspectral imaging with pixel-level fusion to EMI sensor data (Lekhak et al., 3 Oct 2025).
• Path planning in topologically diverse terrains or obstacle fields, evaluated via landscape analysis (ELA) and numeric benchmarks (Shehadeh et al., 24 Jan 2025).
• Reasoning, planning, and scenario-based cognition in agentic autonomous systems, with validated LLM-generated scenarios and multi-style MCQs (Ferrag et al., 14 Nov 2025, Guo et al., 23 May 2025, Yao et al., 28 Aug 2024).

Scenario variation is explicit: WebUAV-3M and UAV-C introduce scenario-constrained subtests such as low light, long-term occlusions, small targets, adversarial noise, and common corruption (rain, fog, blur, sensor noise), with performance drops systematically analyzed across trackers (e.g., 68% drop under zoom blur, 19% under composite Rain-Defocus) (Liu et al., 18 Mar 2024).

4. Algorithmic Baselines and Comparative Analyses

UAVBench protocols typically include extensive baseline comparisons:

• Tracking/detection: SiamR-CNN, GlobalTrack (QG-RCNN, long-term), PrDiMP, KeepTrack, and transformer-based OSTrack/ToMP; best mSA on CST Anti-UAV is 35.92% with GlobalTrack, sharply lower than Anti-UAV410’s 66.42% (Xie et al., 31 Jul 2025, Jiang et al., 2021).
• RF identification: ViT-L16 achieves 58.16% accuracy using "Hot" spectrograms (SNR>10 dB: 99.70%) (Shi et al., 12 Mar 2025).
• 3D Perception: LGFusionNet surpasses alternative fusion methods for pose estimation (rotation error reduced by 41%, position error by 26%) and 3D AP by 54% on UAV-MM3D (Zou et al., 27 Nov 2025).
• Optimization: Adaptive evolutionary algorithms (EA4eig, APGSK, ELSHADE) dominate UAV path planning landscapes; deterministic DIRECT-type methods stagnate in multimodal topologies (Shehadeh et al., 24 Jan 2025).
• Agentic Reasoning: BEDI and UAVBench_MCQ document SOTA VLMs (GPT-4o, Qwen3, Gemini, Claude) with up to 98% on cyber-physical questions, but lower performance on ethics-aware, energy/resource, and integrated reasoning (best Balanced Style Score: Qwen3 235B at 0.74) (Ferrag et al., 14 Nov 2025, Guo et al., 23 May 2025).
• Compression: Learned codecs (EEV) offer 19.8% average BD-rate reduction vs OpenDVC, and outperform HEVC in most outdoor UAV clips, but not for fish-eye/distorted indoor scenes (Jia et al., 2023).

Attribute-wise breakdowns, ablation studies, and per-corruption radar profiles further inform challenges and research directions.

5. Benchmark Suite Standardization and Best Practices

Recent UAVBench design handbooks (DECISIVE, UA-1 PH2) formalize multi-domain sUAS evaluation protocols across urban/subterranean, indoor/outdoor, GNSS-/vision-/RF-denied settings (Norton et al., 2023, Norton et al., 29 Jan 2025). The standard suite is modular:

• Communications (BLOS/NLOS, latency, interference)
• Field Readiness (endurance, takeoff/landing success, room/building clearing, noise)
• Obstacle Avoidance and Collision Resilience (MASI, $\Delta V$)
• Navigation (traversal error, confined spaces, aperture/channel tests)
• Mapping (2D/3D accuracy, coverage, acuity)
• Autonomy (contextual/non-contextual regressions)
• Trust and Situation Awareness (survey protocols, interface-attention SEEV metrics)

All methods are accompanied by canonical metric definitions (LaTeX), statistical confidence reporting (Wilson intervals), open-source apparatus templates, and environment parameter logs. Comprehensive qualitative surveys supplement quantitative test data.

Table: Key UAVBench Datasets and Modalities

Benchmark	Core Modality	Frames / Size	Attributes / Tasks
Anti-UAV	RGB-Thermal	586K	7 attributes, state accuracy
RFUAV	Raw RF (IQ)	1.3TB	37 UAVs, fingerprint f (5D)
CST Anti-UAV	Thermal	240K	6 manual attributes, tiny UAV
UAV-MM3D	RGB/IR/LiDAR/Radar	400K	2D/3D boxes, 6-DoF, tracking
WebUAV-3M	RGB + Text/Audio	3.3M	223 categories, UTUSC
UAV3D	Multi-RGB, BEV	500K	3D boxes, 3D MOT, fusion
UAVLight	Multitemporal RGB	∼5K/18 scenes	Paired illumination slots
BEDI/UAVBench_MCQ	Real/Sim/LLM	50K scenarios	Reasoning styles, metrics

6. Implications, Limitations, and Future Directions

UAVBench frameworks elucidate the fundamental gaps in current methods: extreme drop under adverse/corrupted conditions, poor generalization to tiny targets or ethical/action reasoning, and domain transfer limitations of learned video and perception codecs (Jiang et al., 2021, Xie et al., 31 Jul 2025, Jia et al., 2023, Ferrag et al., 14 Nov 2025, Yao et al., 28 Aug 2024). The increasing prevalence of agentic and multimodal paradigms (GNN fusion, VLM reasoning, multimodal tracking) suggests a clear trajectory toward fully embodied, robust UAV intelligence.

Recommended future work includes:

• Domain-adaptive and adversarial-robust modeling;
• Real-to-sim-to-real transfer pipelines (simulation, augmentation, pre-training);
• Extension of benchmarks to GNSS-denied, multi-UAV, ethical-policy, and dynamic obstacle scenarios;
• Joint benchmarking of perception, planning, and agentic reasoning under physical and environmental constraints;
• Full open reproducibility via GitHub dataset/code releases.

UAVBench provides an empirical and standardizing foundation for the comparative evaluation of UAV algorithms in both academic and applied contexts.