UAVIT-1M: UAV Visual Instruction Tuning

• UAVIT-1M is a large-scale low-altitude UAV dataset with about 1.24M instruction-tuning annotations from 789k images spanning 11 distinct tasks.
• It addresses the domain gap by providing both image‐ and region‐level instructions tailored for real-world UAV applications like rescue, traffic management, and industrial inspection.
• The dataset integrates diverse public sources with rigorous quality control and weighted sampling to improve semantic understanding in multimodal large language models.

UAVIT-1M is a large-scale low-altitude UAV visual instruction-tuning dataset introduced together with UAVBench to evaluate and improve multimodal LLMs (MLLMs) on low-altitude vision-language tasks. It is presented as a dataset for both image-level and region-level understanding in real-world UAV imagery, with approximately 1.24 million instruction-tuning conversations built from 789k images, 11 distinct tasks, and about 2,000 spatial-resolution types. Its stated purpose is to remedy the mismatch between current MLLM capabilities and the requirements of real-world low-altitude UAV applications, where viewpoint, scale, object density, flight altitude, weather, and spatial reasoning differ substantially from both Internet imagery and satellite remote sensing (Zhan et al., 15 Mar 2026).

1. Position within low-altitude UAV vision-language research

UAVIT-1M is the training counterpart to UAVBench. In the paper’s formulation, UAVBench diagnoses the limitations of existing MLLMs on low-altitude UAV imagery, while UAVIT-1M provides the domain-specific supervision needed for instruction tuning. The paired benchmark uses only source-dataset test sets and contains 966k test samples over 261k images across 10 tasks and 43 test units, whereas UAVIT-1M is constructed only from source training and validation data to avoid label leakage. The split policy is explicitly stated: the test set of each source dataset is used for UAVBench, and the training or validation set is used for the instruction fine-tuning data (Zhan et al., 15 Mar 2026).

The paper frames this dataset as necessary because low-altitude UAV imagery is not adequately covered by either generic natural-image MLLM resources or satellite-oriented remote-sensing resources. That claim is supported by a domain-gap analysis using 1k randomly sampled images each from UAV data, ImageNet natural images, and DIOR satellite remote-sensing images over 20 random trials. The reported Frechet Inception Distance values are UAV↔RS $=0.60\pm0.002$, UAV↔Natural $=0.5\pm0.004$, and RS↔Natural $=0.7\pm0.007$. The corresponding cosine similarities are UAV↔RS $=0.7\pm0.002$, UAV↔Natural $=0.81\pm0.001$, and RS↔Natural $=0.7\pm0.002$. In the paper’s interpretation, low-altitude UAV data is closer to natural images than to satellite remote sensing, which helps explain why remote-sensing-specialized MLLMs may underperform on low-altitude UAV tasks.

The intended application scope is broad within low-altitude operations. The paper associates the dataset with emergency rescue, industrial inspection, urban governance, traffic management, agricultural environmental protection, delivery, patrol, disaster response, and human-drone interaction. Those uses are not encoded as separate benchmark tracks, but they motivate the dataset’s emphasis on diverse scenes, weather, altitudes, and task granularities.

2. Dataset scale, task structure, and visual characteristics

UAVIT-1M contains approximately 1.24 million instruction-tuning annotations derived from 789k images. The paper also reports “a total of 1.2M instruction-tuning data and 789k images,” while the per-task totals in Table 4 sum to 1.24M. The dataset spans 1,993 image-resolution types, with widths ranging from 110 to 6000 pixels and heights ranging from 83 to 6000 pixels. It covers 11 tasks divided into image-level and region-level instruction-following settings (Zhan et al., 15 Mar 2026).

The task composition is as follows.

Task	Images	Annotations
Image Classification (Cls)	22,521	22,521
Detailed Classification (D.Cls)	3,757	3,757
Detailed Image Captioning (D.Cap)	37,854	113,562
Image Captioning (Cap)	10,961	54,805
Visual Question Answering (VQA)	72,449	172,838
Target Counting (Count)	73,939	156,241
Region Classification (R.Cls)	181,873	272,463
Region Captioning (R.Cap)	116,995	116,995
Region VQA (R.VQA)	95,203	95,203
Region Detection (Det)	81,557	140,207
Visual Grounding (VG)	92,074	92,074

The task distribution is explicitly imbalanced because it inherits the scale differences of the source datasets. The largest task is Region Classification at 21.9% of UAVIT-1M, and the smallest is Detailed Classification at 0.4%. The paper does not treat this imbalance as an accident; instead, it addresses it during fine-tuning through weighted sampling.

The visual domain is defined by pure real-world low-altitude UAV imagery rather than synthetic renderings. The authors state that synthetic visual data was excluded. The source mixture covers rural areas, cities, parking lots, residential areas, universities, disaster-stricken areas, streets, and transportation hubs. Weather and lighting conditions include rainy weather, foggy weather, night, low light, and motion blur. Object position is center-biased: normalized target centers follow a central Gaussian-like distribution around $x=0.5$, which the paper attributes to realistic UAV operating practice, since operators tend to keep objects of interest near the image center.

Altitude variation is another defining characteristic. The paper uses low $(<30\text{m})$, mid $(\ge 30\text{m and } \le 70\text{m})$, and high $(>70\text{m})$ categories. It also analyzes object size using normalized box-area ratios, with small $=0.5\pm0.004$0, normal $=0.5\pm0.004$1, and large $=0.5\pm0.004$2 bins. UAVBench and UAVIT-1M are reported to have similar object-size distributions, although the benchmark is more challenging for small objects, especially in Region Captioning and Region Classification.

3. Source datasets, normalization, and quality control

UAVIT-1M was built from 21 selected datasets drawn from an initial pool of 30 large-scale mainstream UAV datasets. The processing pipeline has four stages: data preparation, data integration, data generation, and quality control. In the preparation stage, the authors manually reviewed the available sources and retained only public datasets deemed suitable for real-world low-altitude visual understanding. For UAV123, which includes synthetic sequences, the Unreal4-rendered sequences were deleted. For web-crawled sources marked with a dagger in Table 1, including 9,688 videos from ERA, MOD20, and WebUAV-3M, the retained videos were manually reviewed and confirmed as real-world captures (Zhan et al., 15 Mar 2026).

The 21 selected sources are AeroScapes, AU-AIR, CapERA, DroneVehicle, ERA, FloodNet, GeoText-1652, HazyDet, MOD20, RefDrone, RDDTS, Semantic Drone, UAV123, UAVDT-M, UAVDT-S, UAVid, UDD, UDV DIT, VDD, VisDrone2019-DET, and WebUAV-3M. These sources cover semantic segmentation, object detection, video captioning, event recognition, natural language-guided geolocalization, action recognition, referring expression comprehension, tracking, multi-object tracking, and image-text retrieval.

A central integration problem was annotation heterogeneity. The paper identifies two major forms: differing bounding-box formats such as YOLO versus COCO, and inconsistent category names across datasets. To address category inconsistency, the authors manually created a semantically rich mapped category dictionary with 100 object categories. The dictionary explicitly groups synonyms or semantically overlapping labels, with examples including “people,” “person,” and “human,” as well as broader-scope relations such as “bus” under “public transport.” During training, random mapping among equivalent category names is performed. The stated purpose is to improve semantic generalization and reduce naming bias.

Task generation combines direct data-structure transformation, Gemini 2.5 Pro generation, and manual annotation, with reported proportions of 79.7%, 11.1%, and 9.2%, respectively. Quality control is extensive. Manual annotation uses a double-check mechanism in which each processed sample is cross-reviewed by a second expert, and disagreements are arbitrated by a third senior annotator. LLM-generated data for image captioning and region VQA receives full review to avoid hallucinations and errors. The paper states that 115.7k samples were manually reviewed and that only 0.39% of reviewed samples required manual correction. For existing-dataset expansion, two rounds of random manual quality scoring were performed, sampling 0.6% of the samples from each task in each round and reviewing approximately 20k samples. The final checker rejection rate was 0.05%.

4. Instruction design and the 11-task formulation

UAVIT-1M is organized as a unified visual instruction-tuning corpus rather than as a collection of unrelated annotation files. The conversation template is described as <image> Image Features \n [Task Identifier] Instruction. Explicit task identifiers such as [img cap], [reg vqa], and [vg] are prepended to the instruction. Region coordinates in prompts are represented as $=0.5\pm0.004$3, corresponding to the top-left and bottom-right vertices of the region. To support different MLLMs, the dataset provides multiple coordinate encodings: absolute coordinates, coordinates scaled to hundredths, and coordinates scaled to thousandths (Zhan et al., 15 Mar 2026).

Instruction diversification is performed with DeepSeek. The paper gives the example of expanding “Count the number of buses.” into variants such as “How many buses are there?” and “Provide the quantity of buses.” The released conversation format is single-turn: a short “human” prompt paired with a “gpt” answer in JSON-style conversation structure. The paper does not describe a multi-turn curriculum.

The image-level tasks include classification, captioning, VQA, and counting. Image Classification asks the model to assign the entire image to one event or scene class. Detailed Classification asks for all object categories contained in the image in multi-label textual form. Image Captioning requests a brief description, while Detailed Image Captioning requests a richer paragraph-style description; the paper illustrates this with a campus scene described in terms of building appearance, vegetation, paths, roads, and parking lots. VQA includes presence questions, comparison questions, weather questions, and altitude questions, including both categorical altitude ranges and numeric altitude estimation. Target Counting asks for the count of a queried category as an integer.

The region-level tasks are explicitly spatial. Region VQA asks questions about a specified region, such as the color of a building denoted by coordinates. Region Captioning asks for a description of a specified region. Region Classification asks for the class of a specified region. Region Detection takes a category query and returns all matching boxes. Visual Grounding takes a referring expression and asks for the corresponding location. This two-level design is one of the dataset’s central properties: whole-image semantics and region-specific reasoning are jointly represented within a single instruction-tuning framework.

5. Fine-tuning use, model formulation, and empirical effects

The paper fine-tunes three 7B MLLMs on UAVIT-1M: LLaVA1.5-UAV, MiniGPTv2-UAV, and GeoChat-UAV. MiniGPTv2-UAV uses EVA-CLIP ViT with input resolution 448, a linear projector, and a LLaMA-2-Chat-7B backbone fine-tuned with LoRA. LLaVA1.5-UAV uses CLIP ViT-L/14 at 336 resolution, an MLP alignment layer, and Vicuna-v1.5-7B. GeoChat-UAV uses CLIP ViT-L/14 at 336, an MLP alignment layer that remains frozen, and Vicuna-v1.5-7B. All experiments are reported on 4 NVIDIA L20 48G GPUs; the paper additionally states that 2 L20 GPUs or 4 RTX 3090 24G GPUs can also support the experiments (Zhan et al., 15 Mar 2026).

The training formulation is standard autoregressive multimodal instruction tuning. If the visual encoder extracts $=0.5\pm0.004$4, then the projected visual tokens are

$=0.5\pm0.004$5

With language instruction embedding $=0.5\pm0.004$6, the concatenated sequence $=0.5\pm0.004$7 is processed by the LLM. Cross-modal attention is written as

$=0.5\pm0.004$8

and the autoregressive loss is

$=0.5\pm0.004$9

The paper also describes weighted sampling for task balancing, but only verbally: each task’s sampling frequency is set inversely to its proportion so that large tasks do not dominate and small tasks are not ignored.

The reported empirical effect is strong on semantic understanding and captioning. Before fine-tuning, open-source MLLMs are said to perform poorly across UAVBench, especially on altitude estimation, detailed classification, counting, and detection. After fine-tuning on UAVIT-1M, the adapted models improve markedly on many tasks. On image classification and VQA, MiniGPTv2-UAV reaches 93.06 on ERA classification and 89.96 and 90.16 on presence and comparison VQA, with flying-altitude recognition 66.79 and altitude-estimation accuracy 51.97 for integer answers and 14.49 for decimal answers. GeoChat-UAV’s weather VQA on UAVDT-S rises from 77.98 to 99.14, altitude recognition rises from 36.51 to 70.05, and integer altitude estimation rises from 0.00 to 48.10.

On Detailed Classification, gains are particularly large. MiniGPTv2-UAV reaches 87.73 F1 on AeroScapes, 94.31 F1 and 91.33 mAP on VDD, and 98.55 F1 and 97.14 mAP on UDD. On image captioning, GeoChat-UAV becomes the strongest adapted captioner in the reported results, with CapERA BLEU-4 31.1, METEOR 25.5, ROUGE 52.2, CIDEr 115.1, and SPICE 24.7, while on UDV DIT it achieves BLEU-4 16.1, CIDEr 35.3, and SPICE 16.6. For region-level semantics, fine-tuned models become strong on Region Classification and competitive on Region Captioning. The paper states that, for image-level tasks, fine-tuned low-altitude models can surpass the advanced closed-source model, and that for region-level tasks they can achieve best performance on Region Classification and Region Captioning.

Ablation studies tie these effects directly to dataset design. Adding task identifiers improves overall performance and multi-task efficiency. Reducing data scale from 100% to 50% to 10% consistently reduces performance. Training on only one granularity level hurts the other and often hurts the retained one as well. For MiniGPTv2-UAV, full 100% data with both image-level and region-level supervision gives Cls 85.69, VQA 97.35, Count 25.46, R.VQA 50.65, R.Cls 90.62, and VG 28.27; reducing the data to 10% lowers these to 79.32, 88.60, 18.51, 41.88, 80.53, and 17.65.

6. Limitations, biases, and distinction from similarly named datasets

The paper is explicit that UAVIT-1M does not solve all low-altitude UAV MLLM problems. The weakest area after fine-tuning remains region-level localization, especially detection and grounding. Even after tuning, Region Detection [email protected] remains very low: MiniGPTv2-UAV, the strongest among the tuned models on this metric, reaches 7.09 AP on AU-AIR, 1.47 on DroneVehicle, 1.27 on UAVDT-M, 0.99 on HazyDet normal, and 0.91 on HazyDet hazy. Visual Grounding improves only modestly. The paper interprets this as evidence that current MLLM architectures still lack the fine-grained spatial precision required for UAV region localization (Zhan et al., 15 Mar 2026).

Other limitations are also stated. Performance is unstable across tasks; the authors argue that future systems should become more balanced all-rounders. The current benchmark and tuning corpus focus primarily on RGB images, and the paper explicitly suggests future extension to infrared, SAR, multi- or hyperspectral, and temporal data. The dataset has a realistic but nontrivial center bias because UAV operators tend to keep targets near the image center. The paper also notes the trade-off between generalist multimodal versatility and specialist perception precision: classical detectors such as YOLO or DETR can achieve 40–50% [email protected] on VisDrone-2019-DET, substantially above current MLLM detection performance. No dedicated licensing terms for the aggregated sources are given in the provided text, although the paper states that the benchmark, dataset, and fine-tuned checkpoints will be released publicly.

A recurrent source of confusion is the name. UAVIT-1M should not be conflated with InsViE-1M, which is a 1,019,593-triplet instruction-based video editing dataset composed of source video, edited video, and instruction triplets for video editing rather than low-altitude UAV visual instruction tuning (Wu et al., 26 Mar 2025). Nor should it be confused with the UAV-based VNIR hyperspectral benchmark for landmine and UXO detection, which is a specialized hyperspectral remote-sensing dataset with 32 flight lines, a final $=0.7\pm0.007$0 reflectance cube, and approximately 143 surrogate targets, rather than a million-scale UAV vision-language corpus (Lekhak et al., 3 Oct 2025).

In that sense, UAVIT-1M occupies a specific position in the dataset landscape. It is neither a generic million-video editing corpus nor a specialized airborne hyperspectral detection release. It is a low-altitude UAV instruction-tuning dataset built from 21 public real-world datasets, normalized into a unified annotation space, expanded into 11 image- and region-level tasks, and empirically shown to improve MLLM semantic understanding of low-altitude UAV imagery while leaving counting, grounding, and precise localization as open problems.