WebUAV-3M Benchmark for UAV Tracking

• WebUAV-3M is a large-scale multimodal benchmark for deep learning-based UAV tracking, featuring 4,500 videos and 3.3M frames with 223 target categories.
• It employs a semi-automatic target annotation pipeline (SATA) that combines automated predictions with human-in-the-loop corrections to ensure dense, high-quality labels.
• The dataset includes visual, language, and audio modalities with innovative evaluation protocols to assess tracker performance under challenging scenarios like low light, occlusion, and high-speed motion.

WebUAV-3M is a large-scale public benchmark constructed to advance deep learning-based unmanned aerial vehicle (UAV) tracking. It addresses previous limitations in scale, diversity, modalities, and evaluation protocols within UAV tracking research, comprising 3.3 million frames across 4,500 videos, 223 target categories, and a comprehensive set of scenario constraints. Through extensive multimodal annotation and innovative evaluation protocols, WebUAV-3M supports the development and rigorous assessment of modern UAV trackers, especially in challenging, long-tail, and multi-scenario contexts (Zhang et al., 2022).

1. Dataset Scale, Structure, and Diversity

WebUAV-3M consists of 4,500 UAV-captured videos containing approximately 3.3 million frames (28.9 hours at 30 fps). Each video contains between 40 and 18,841 frames, with a mean of 710 frames per video. Target diversity is expressly designed, featuring 223 target categories and 63 motion types, grouped into 12 superclasses (e.g., person, building, vehicle, vessel, aircraft, animal, artifact, plant). The category frequency exhibits a pronounced long-tail distribution; for example, “person” has 1,305 videos, while rare classes like “balloon” have as few as 4.

The dataset is divided to stress both generalization and fair quantitative evaluation:

Split	Videos	Frames (approx.)	Target Classes	Motion Classes
Training	3,520	2.6 million	208	59
Validation	200	–	–	–
Test	780	0.6 million	120	36

A deliberate minimization of category overlap between training and test splits helps highlight tracker generalization for unseen object and motion types (Zhang et al., 2022).

2. Semi-Automatic Target Annotation (SATA) Pipeline

WebUAV-3M is annotated via an efficient, scalable semi-automatic target annotation (SATA) pipeline, enabling dense labeling of 3.3 million frames within three months. SATA operates as follows:

• Initialization: A human annotator draws a bounding box in the first frame to “ground” the tracker.
• Short-Rollout: The tracker predicts boxes for subsequent frames in real time.
• Human-in-the-loop Verification: Annotators accept, correct, or adjust predictions. When prediction quality degrades, the tracker is interactively retrained on recent corrections.
• Verification: Each annotation passes three successive rounds of human verification to ensure high quality.

This approach alternates automated prediction with human correction, yielding dense annotations with high temporal consistency while achieving both scalability and accuracy (Zhang et al., 2022).

3. Multimodal Annotation Scheme

Each WebUAV-3M video includes both visual and non-visual modalities to facilitate research beyond pure visual tracking:

• Natural Language Specifications: One English sentence per video (average 8–12 words, ≈800 unique-word vocabulary) describes the object class, distinctive attributes, position, motion, and surroundings (e.g., “a small red drone hovering steadily above green fields”).
• Audio Descriptions: Each video receives two Balabolka-generated TTS audio descriptions (male and female voices), totaling 9,000 audio files. The average duration per audio clip is approximately 5 seconds, matching each video’s sentence length.

A plausible implication is that this multi-modality supports exploration of language and audio cues for multi-modal UAV tracking and data fusion algorithms (Zhang et al., 2022).

4. Evaluation Protocol and Scenario Constraints

The UAV Tracking-Under-Scenario Constraint (UTUSC) protocol replaces previous binary or global attribute annotation with per-frame, quantitative scenario indicators. Each indicator describes a distinct challenge, guiding both model development and scenario-based benchmarking:

• Low light: Average luminance over a $4\times$ bounding-box area $$\Omega_t = \tfrac{1}{3}\sum_{c\in\{R,G,B\}} I_c(x,y)$$.
• Long-term occlusion: Duration of consecutive occlusion frames.
• Small target: Square root of the bounding-box area, $\Xi_t = \sqrt{w_t h_t}$.
• High-speed motion: Normalized velocity, $$\Delta_t = \frac{\|p_t - p_{t-1}\|}{\sqrt{s_{t-1}s_t} (T_t - T_{t-1})}$$, with $s_t = \sqrt{w_t h_t}$.
• Target distortions: No-reference IQA score, $\Psi_t = f_\theta(\text{crop}(F_t))$.
• Dual-dynamic disturbances: Indicator $\Phi_t$ for abrupt camera/target motion.
• Adversarial examples: Magnitude of adversarial perturbation, $$I_{k+1}^j = I_k + \epsilon \psi(I_k, \eta^j),\ \|I_{k+1}^j-I_0\|_2\le M$$.

Seven scenario-based 100-video subtests are constructed: low light, long-term occlusion, small targets, high-speed motion, target distortions, dual-dynamic disturbances, and adversarial examples. Each subtest spans 10–12 superclasses, 39–49 target classes, and 10–14 motion types.

Evaluation metrics consist of precision at 20 px, normalized precision, success (IoU) and area-under-curve (AUC), complete success (cAUC; incorporating IoU, location, and aspect ratio), and mean accuracy (mAcc; penalizes false positives during absent frames) (Zhang et al., 2022).

5. Baseline Tracker Performance Analysis

Forty-three representative trackers are benchmarked on WebUAV-3M under UTUSC, revealing nuanced trade-offs and failure modes:

• Top five trackers (by cAUC/mAcc/Precision):

Tracker	Pre	nPre	AUC	cAUC	mAcc
AlphaRefine	0.753	0.643	0.593	0.562	0.602
KeepTrack	0.710	0.603	0.543	0.512	0.550
PrDiMP	0.674	0.575	0.514	–	–
RPT	–	–	0.495	–	–
ECO	–	–	–	–	–

• Real-time vs. accuracy: SiamRPN (∼143 fps) and KCF (132 fps CPU) offer the highest speed but lag in accuracy, while AlphaRefine achieves top accuracy at 42 fps and KeepTrack at 34 fps.
• Scenario robustness: All trackers degrade as task difficulty increases. TransT exhibits resilience to occlusion, presumably due to global attention mechanisms. PrDiMP demonstrates superior handling of high-speed motion attributed to uncertainty modeling.
• Adversarial robustness: Most trackers exhibit minor drops under moderate adversarial magnitudes, but transformer-based trackers (e.g., TransT) lose >20% accuracy at $M>6000$ (Zhang et al., 2022).

6. Challenges, Insights, and Research Trajectories

WebUAV-3M highlights a range of persistent and emerging challenges:

• Effective low-light and nighttime tracking capabilities remain limited in current approaches.
• CNN and transformer-based trackers demonstrate vulnerability to adversarial perturbations, indicating a need for enhanced defense mechanisms.
• Initial gains from multi-modal fusion (vision, language, and audio) are limited, suggesting that further methodological innovation is needed, with large-scale multimodal benchmarks like WebUAV-3M playing a critical role in this endeavor.
• The pronounced long-tail class distribution (Zipf’s law) mandates novel approaches to rare-category generalization.
• Real-time constraints inherent to UAV platforms demand further investigation into efficiency versus accuracy trade-offs (Zhang et al., 2022).

All dataset resources, the SATA toolkit, protocol details, and baseline results are publicly accessible at https://github.com/983632847/WebUAV-3M.