Anti-UAV Multi-Modal Benchmark

Updated 8 January 2026

Anti-UAV Multi-Modal Benchmark is a rigorously designed dataset and protocol that integrates stereo vision, LiDAR, radar, and audio for detecting, classifying, and tracking UAVs.
It defines tasks like 2D/3D detection, UAV-type classification, and trajectory estimation using standardized metrics such as mAP, RMSE, and F1 to ensure reliable evaluation.
The benchmark supports real-world and adversarial testing conditions, driving advances in sensor fusion, domain generalization, and robust multi-modal algorithm design.

An Anti-UAV Multi-Modal Benchmark is a rigorously designed dataset and protocol suite for evaluating algorithms targeting the detection, classification, and tracking of unmanned aerial vehicles (UAVs) using synchronized data from multiple sensor modalities. These benchmarks address real-world threat scenarios by leveraging diverse sensing—such as stereo vision, LiDAR, radar, and audio arrays—often combined with meticulously aligned multi-sensor ground truth. The rise of miniature and commercial drones with low visual or acoustic signature has catalyzed intensive research in both the academic and applied security sectors, necessitating robust, multi-modal benchmarking for anti-UAV technologies (Yuan et al., 2024, Dong et al., 16 Apr 2025, Zhang et al., 8 Dec 2025).

1. Sensor Modalities and Dataset Composition

Anti-UAV multi-modal benchmarks feature synchronized, high-resolution data streams from multiple sensor types mounted on static or mobile platforms:

Stereo Vision: Dual global-shutter color cameras (e.g., PIXEL-XYZ AR135-130T400 at 2560×960 px, 30 Hz) provide panoramic 180° fisheye coverage, enabling depth estimation up to ~20 m and object visibility out to 100 m (Yuan et al., 2024).
LiDAR: Dual units (e.g., Livox Avia conic 3D, Livox Mid360 peripheral 3D) supply both conic and 360° horizontal scans at 10 Hz, with ranges spanning 70–300 m, capturing both upward and lateral UAV movement (Yuan et al., 2024, Deng et al., 2024).
Radar: 4D mmWave systems (e.g., Oculii Eagle ETH04, 77 GHz, 120°×30° FoV) output 15 Hz point clouds for long-range tracking (up to 350 m), robust to weather and lighting (Yuan et al., 2024).
Audio Arrays: Cross-shaped omnidirectional microphone configurations (e.g., Hikvision DS-VM1, 0–44.1 kHz, 41.8 kHz sampling rate/channel) facilitate time-difference-of-arrival estimation of acoustic drone signatures in the presence of confounding ambient sounds such as heavy machinery and wind (Yuan et al., 2024, Dong et al., 16 Apr 2025).
Other Modalities: Synthetic and simulation-based benchmarks introduce IR/thermal, event cameras (DVS), and RF, supporting investigations under adverse or occluded environments (Zou et al., 27 Nov 2025, Dong et al., 16 Apr 2025).

Comprehensive benchmarks like MMAUD and MMUAD typically consist of 1,000+ sequences and tens of thousands of synchronized frames, fully annotated for detection (2D/3D bounding boxes), class (UAV type), and trajectory (per-frame ground truth, e.g., Leica MS60 Total Station at 5 Hz with millimeter-level error) (Yuan et al., 2024, Deng et al., 2024, Zou et al., 27 Nov 2025).

2. Task Definitions and Evaluation Metrics

Multi-modal anti-UAV benchmarks prescribe several core supervised and self-supervised tasks:

2D/3D Detection: Predict axis-aligned bounding boxes and class labels from single or fused modalities. Principal metrics include mean Average Precision at IoU≥0.5 ([email protected]), F1, Precision/Recall, and Intersection-over-Union (IoU). For 3D, mAP@[.5:.95] over bounding spheres or cuboids is standard (Yuan et al., 2024, Zou et al., 27 Nov 2025, Dong et al., 16 Apr 2025).
UAV-Type Classification: Assign each detected instance or sequence to one of K discrete UAV classes (e.g., DJI Mavic 2, Avata, Matrice 300). Evaluation is based on accuracy, per-class recall, and confusion matrices, with cross-entropy loss for training (Yuan et al., 2024, Deng et al., 2024).
Trajectory/State Estimation: Estimate UAV 3D position $\hat p_t$ at each timestamp and output a temporally consistent trajectory. Common metrics: Mean Squared Error (MSE), Root-Mean-Squared Error (RMSE), Mean Absolute Error (MAE), Multi-Object Tracking Accuracy (MOTA), and Multi-Object Tracking Precision (MOTP) (Yuan et al., 2024, Deng et al., 2024, Zou et al., 27 Nov 2025).
Multi-Scenario/Adversarial Evaluation: Advanced protocols (e.g., UTUSC in WebUAV-3M) introduce scenario-constrained subtests: low light, small targets, occlusion, high-speed motion, and adversarial examples (e.g., GPS spoofing, camouflage) (Zhang et al., 2022).

Typical evaluation splits are 60%/20%/20% (train/val/test) or stratified by scenario, with real-time constraints (<33 ms/frame; ≈30 FPS) on embedded and enterprise hardware (Yuan et al., 2024, Zhang et al., 2022).

3. Benchmark Datasets and Annotation Protocols

Key multi-modal anti-UAV benchmarks include:

Dataset	Modalities	Scale/Annotations	Key Features
MMAUD (Yuan et al., 2024)	Stereo RGB, 3D LiDAR, Radar, Audio	50 sequences/51k frames	Leica ground truth, full scenario simulation, ambient industrial noise
MMUAD (UG2+) (Deng et al., 2024)	Stereo, multiple LiDARs, Radar, Audio	177 sequences, extreme weather/lighting	Classification and pose estimation leaderboards, sequence-level annotation
Anti-UAV (Jiang et al., 2021), UAUTrack (Ren et al., 2 Dec 2025)	RGB, IR	>300 video pairs, 580k boxes	Dense manual bbox, multi-modal fusion, scene complexity sampling
UAV-MM3D (Zou et al., 27 Nov 2025)	RGB, IR, LiDAR, Radar, DVS	400k frames (synthetic)	Full 6-DoF pose and trajectory, multi-UAV/sim-to-real domain randomization
WebUAV-3M (Zhang et al., 2022)	RGB, Language, Audio	4,500 videos, 3.3M frames	Semi-automatic SATA annotation, language/audio enrichment, scenario subtests

Annotation protocols emphasize multi-round QC, synchronization to hardware timestamps (<1 ms drift), and provision of 2D/3D labels plus absent/occluded flags. Recent benchmarks augment data with meta-attributes (e.g., camera motion, illumination, fast motion, distractors) and natural-language prompts to support vision-language modeling (Zhang et al., 8 Dec 2025, Zhang et al., 2022).

4. Algorithmic Baselines and Performance

Benchmarks report comprehensive baselines across classical, deep, and multi-modal fusion paradigms:

2D Detection: CNN detectors—YOLOv5 (84.85% mAP50), YOLOX (85.90%), SSD, CenterNet—demonstrate >80% mAP50 at real-time rates (30 FPS). Long-range, low-contrast targets remain the hardest regime (Yuan et al., 2024).
3D Trajectory Estimation: Backbone CNNs (ResNet50, VGG16/19, YOLOv4) enable sub-meter errors (RMSE ≈0.5 m) on visual data; audio-only regressors show significantly degraded accuracy (≈2.6 m RMSE) under ambient noise (Yuan et al., 2024, Deng et al., 2024).
Classification: EfficientNet-B7, with sequence fusion and ROI cropping, achieves >81% accuracy, outperforming single-frame baselines by ≥10–15 percentage points (Deng et al., 2024).
Multi-Modal Fusion: Methods fusing RGB, LiDAR, radar, and audio outperform single-modality models in challenging conditions, with LiDAR-guided mid-attention boosting 3D AP from 20.15% to 26.72% in synthetic UAV-MM3D (Zou et al., 27 Nov 2025).
Tracking and Vision-Language: Unified transformer pipelines (e.g., UAUTrack) and Mamba-based SSMs (MambaSTS, AUC 0.437) leverage multi-modal attention and text prompts for robust drone locking, with semantic priors yielding substantial improvements in distractor-rich scenes (Ren et al., 2 Dec 2025, Zhang et al., 8 Dec 2025).

5. Architectural Methodologies and Data Processing

Modern anti-UAV benchmarks foster the development and evaluation of:

Cross-Modal Transformers: Fusion via attention mechanisms integrates tokens from RGB, TIR, audio, and prompts (e.g., Text Prior Prompt—TPP in UAUTrack) at every transformer layer, improving both robustness and semantic specificity (Ren et al., 2 Dec 2025).
State-Space Modeling: SSMs within tracking architectures (e.g., MambaSTS) model long-term temporal dependencies, propagating scene memory across frames and enabling video-level spatial-temporal-semantic fusion (Zhang et al., 8 Dec 2025).
Unsupervised Clustering: For scenarios lacking per-frame 2D bboxes, dynamic point clustering (e.g., DBSCAN on LiDAR/radar) filters candidate targets via centroid velocity/motion cues, combined with bias-corrected regression for refined center localization (Deng et al., 2024).
Data Augmentation: Benchmarks recommend random rotation, temporal dropout, reversal, and synthetic clutter insertion (rain, birds) to mimic real-world occlusion, weather, and adversarial perturbations (Yuan et al., 2024, Deng et al., 2024).
Ground Truth Alignment: Sensor extrinsics are calibrated using divide-and-conquer approaches (intrinsics/extrinsics via Zhang’s, targetless camera-LiDAR methods, CAD-based microphone and radar mounting, etc), aligning all measurements to a consistent world frame using 4×4 homogeneous transforms (Yuan et al., 2024).

6. Current Challenges and Research Directions

Despite significant advances, multi-modal anti-UAV benchmarks expose enduring challenges:

Multi-modal Fusion Realism: Audio suffers from industrial/machinery confounds; visual modalities degrade under poor illumination or weather; LiDAR/radar require domain-adapted back-ends for robust long-range, multi-uav association (Yuan et al., 2024, Zou et al., 27 Nov 2025).
Generalization Across Scenarios: Transfer from synthetic (UAV-MM3D) to real data remains hindered by domain gaps; scenario subtests (low light, occlusion, adversarial attacks) reveal systematic weaknesses in existing trackers (Zou et al., 27 Nov 2025, Zhang et al., 2022).
Scalability: Real-time processing across all modalities is nontrivial for embedded/edge platforms, especially under scalability constraints (e.g., swarm monitoring or variable camera rates) (Dong et al., 16 Apr 2025, Zhang et al., 8 Dec 2025).
Research Gaps: Identified gaps include stealth/low-signature UAVs under adverse weather/jamming, adversarial robustness, and swarm tracking beyond 20–30 vehicles (Dong et al., 16 Apr 2025).
Open Directions: Benchmarks and surveys recommend dynamic cross-modal weighting, real-time vision-language integration, reinforcement learning for adaptive tracking/interception, and the expansion of benchmarks to include overt stealth/jamming, day/night and multi-swarm test conditions (Dong et al., 16 Apr 2025, Zhang et al., 8 Dec 2025).

7. Benchmark Construction and Best Practices

Practical guidelines for constructing new anti-UAV multi-modal benchmarks, as distilled from accepted challenge formulas and winning pipelines, include:

Sensor Calibration and Synchronization: Use rigidly mounted multi-sensor rigs, hardware timestamping, and cross-sensor transformation (extrinsics/intrinsics) methods for sub-millisecond alignment (Yuan et al., 2024, Deng et al., 2024).
Annotation Protocol: Combine high-precision 3D pose (motion-capture or RTK-GPS) with dense frame-wise 2D/3D annotations and sequence-level class labels; include absent/occlusion flags and meta-attributes (Zhang et al., 2022, Yuan et al., 2024).
Evaluation Protocol: Adopt a lexicographic (pose MSE, –class accuracy) composite ranking, standardized mAP/scenario metrics, and enforce real-time constraints in line with deployment needs (Deng et al., 2024, Ren et al., 2 Dec 2025).
Augmentation and Domain Randomization: Simulate real-world variability via aggressive data augmentation; generate adversarial and rare cases for robust algorithmic stress testing (Deng et al., 2024, Zou et al., 27 Nov 2025).
Open Access and Documentation: Release datasets, calibration rigs, and code with detailed sensor diagrams, annotation guidelines, and processing scripts to ensure reproducibility and community uptake (Yuan et al., 2024, Zhang et al., 8 Dec 2025).

Anti-UAV Multi-Modal Benchmarks thus represent the intersection of hardware integration, annotation methodology, scenario diversity, and algorithmic innovation, driving the development and evaluation of real-world UAV threat countermeasures (Yuan et al., 2024, Deng et al., 2024, Dong et al., 16 Apr 2025).