DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Images

Abstract: Object detection for Unmanned Aerial Vehicles (UAVs) working in open and dynamic environments is a highly challenging task. While Vision-LLMs (VLMs) have offered a powerful solution for universal object detection, adapting them to UAV scenarios remains non-trivial due to a substantial domain gap between VLM pre-training data and aerial imagery. The prevailing Parameter-Efficient Fine-Tuning (PEFT) methods prove ineffective in bridging this gap, as VLMs' "natural-scene, foreground-dominant" visual priors misalign with the "bird's-eye-view, background-dominant, small-object" characteristics of UAV data. To address this issue, we propose DroneFINE, a novel PEFT paradigm comprising two domain-aware complementary modules tailored for VLM-based drone image detectors. Specifically, a data-dependent, foreground-aware, and multi-path adaptation mechanism named HyperAdapter is designed, which overcomes the static structural constraints of PEFT. In addition, a background suppression algorithm named SemanticGate is developed. It is a text-conditioned guidance strategy that employs background vocabulary to actively guide the model in suppressing responses from irrelevant regions. Extensive experiments on VisDrone and UAVDT demonstrate that DroneFINE significantly outperforms existing PEFT methods and achieves performance comparable to full fine-tuning while substantially reducing the number of trainable parameters.

• The paper introduces HyperAdapter and SemanticGate, combining foreground-aware dynamic convolutions with text-conditioned background suppression to address aerial domain gaps in vision-language detectors.
• DroneFINE-T matches full fine-tuning on VisDrone at 38.4 mAP and 61.2 mAP50 using 1.54M trainable parameters, while outperforming it on UAVDT with 26.5 mAP and 43.6 mAP50.
• The method improves small-object detection without large-scale aerial pre-training, preserves COCO generalization, and adds only 0.43 FPS of inference overhead on an NVIDIA 2080 Ti.

Motivation and problem statement

DroneFINE addresses the failure of standard Parameter-Efficient Fine-Tuning (PEFT) when adapting Vision-LLM (VLM) detectors to UAV imagery. The authors build on GroundingDINO and identify a substantial domain gap: VLM pre-training corpora (Objects365, GLIP, GRIT, V3Det) are ground-level, foreground-dominant scenes, whereas drone imagery is bird's-eye-view, background-dominant, and populated by small objects. Zero-shot evaluation confirms this gap—categories such as "people" and "awning-tricycle" yield only 2%–20% mAP$_{50}$ on VisDrone. The paper's central empirical claim is that conventional PEFT methods (LoRA, Adapter, VPT, NormTuning, Mona) fail not merely because of limited capacity but because of two specific structural deficiencies: low-rank updates cannot capture the high-rank weight changes required for aerial adaptation, and static parameter structures cannot suppress pervasive background noise.

The evidence for the first deficiency is an SVD analysis of FFN weights after full fine-tuning, showing that maintaining low approximation error on UAV data requires a remarkably high rank, particularly in deeper layers. This motivates moving beyond fixed low-rank linear adapters toward dynamic, data-dependent structures.

Method

HyperAdapter

HyperAdapter is a serial adapter inserted after attention and MLP layers in the frozen visual backbone. Its key mechanism is foreground-aware dynamic convolution generation:

• Foreground-aware aggregation: $M$ learnable queries aggregate global foreground features $\mathbf{z}$ via cross-attention over flattened input features. Because the queries are optimized across the training set, they encode dataset-level priors while cross-attention captures intra-image context. This avoids auxiliary density-prediction heads.
• Multi-path dynamic kernels: A single shared hypernetwork $H$ maps $\mathbf{z}$ to weights $\mathbf{W}_{dyn}$ that parameterize parallel $1\times1$, $3\times3$, and $5\times5$ depthwise separable branches operating in a rank-64 bottleneck space. Parameter cost is controlled through compressed hypernetwork hidden dimensions, depthwise separability, and group-shared parameters.

The multi-path design serves both scale handling of small objects and richer gradient feedback to the generator. Feature visualizations show the attention concentrating on vehicles and pedestrians amid complex backgrounds.

SemanticGate

SemanticGate exploits an asymmetric observation: although foreground features align poorly with text due to the domain gap, dominant background elements remain highly detectable by the VLM's text branch. Rather than ignoring the background, SemanticGate actively identifies and suppresses it.

A four-stage pipeline constructs a 20-item background vocabulary covering streetscapes, natural terrains, and atmospheric conditions: sampling UAV images for distribution analysis, LLM-based candidate generation (GPT-4), empirical detectability filtering via DINO-X grounding, and manual review. Embeddings are pre-computed to avoid runtime overhead.

At query selection time, the module computes max similarity scores against original text ($\mathbf{s}_o$) and background vocabulary ($M$0), then applies three LSTM-inspired gates—a Protection Gate preserving high-confidence foreground candidates, an Inhibition Gate activating suppression for background-dominant candidates, and a Candidate Adjustment term bounded by $M$1—to produce adjusted scores $M$2 used for top-$M$3 query selection. An InfoNCE-style contrastive loss on concatenated text embeddings enforces separability between target and background tokens, which the gating mechanism's correctness depends on.

Quantitative results

With GroundingDINO-Swin-T as the base and only the decoder/head plus adapters trainable, DroneFINE-T trains 1.54M parameters versus 27.5M for full fine-tuning (5.6%), yet matches full fine-tuning on VisDrone (38.4 mAP / 61.2 mAP$M$4 vs. 38.4/61.3). On UAVDT it surpasses full fine-tuning (26.5/43.6 vs. 22.4/38.6), where full tuning overfits due to severe inter-frame redundancy. Against LoRA at matched rank, the method gains +1.7 mAP and +2.0 mAP$M$5 on VisDrone.

Method	Params	VisDrone mAP/mAP$M$6	UAVDT mAP/mAP$M$7
Zero-shot	0	16.5 / 26.9	9.5 / 18.5
Baseline (decoder+head)	0	37.1 / 59.6	24.7 / 40.9
Full FT	27.5M	38.4 / 61.3	22.4 / 38.6
LoRA	1.13M	36.7 / 59.3	25.0 / 40.8
Mona	1.4M	37.9 / 60.5	22.2 / 37.6
CoOp	0.012M	37.3 / 60.0	24.9 / 41.4
DroneFINE-T	1.54M	38.4 / 61.2	26.5 / 43.6

Against external detectors, DroneFINE-L (Swin-L) reaches 42.3 mAP / 65.8 mAP$M$8 on VisDrone, exceeding prior SOTA by 2.3 mAP and 3.4 mAP$M$9; on UAVDT, DroneFINE-T achieves SOTA mAP, surpassing ESOD by 2.9 points, though its mAP$\mathbf{z}$0 trails ESOD, which the authors attribute to ESOD's foreground re-cropping prioritizing loose localization. Notably, these results are obtained without large-scale aerial pre-training (e.g., LAE-1M), unlike competing VLM baselines.

Ablations and analysis

Component ablations attribute consistent gains to both modules: HyperAdapter alone adds +1.5 mAP$\mathbf{z}$1 on VisDrone and +1.0 on UAVDT; SemanticGate alone adds +1.2 and +1.1 respectively; the full model reaches 38.4/61.2 and 26.5/43.6. Removing foreground awareness from HyperAdapter costs 0.9 mAP$\mathbf{z}$2 on both datasets, indicating dynamic convolution alone is insufficient. Replacing the learned foreground aggregation with ESOD's ObjSeeker density-map head yields weaker results (37.3/60.0), supporting the claim that implicit query-based aggregation is both more effective and cheaper than auxiliary prediction heads.

Rank analysis shows rank 64 is optimal (rank 96 degrades performance), and at larger budgets LoRA ($\mathbf{z}$3, 2.26M params) provides no gain over baseline while HyperAdapter at 1.54M params reaches 38.1/61.1—directly substantiating the high-rank-capacity argument. Background-vocabulary ablations show the full 20-word vocabulary outperforms size-matched random words by 1.0 mAP / 1.4 mAP$\mathbf{z}$4, and scaling random vocabularies from 5 to 20 words yields no consistent gain, confirming that complementary background semantics—not vocabulary size—drive the improvement.

Query-level statistics clarify the mechanism: after SemanticGate, GT-box IoU coverage improves on 19.3% of images and degrades on 6.0%, while foreground similarity actually decreases on 23.9% of images. The gains therefore arise from re-ranking queries via background scores rather than amplifying foreground responses. A single-gate variant loses 0.7 mAP$\mathbf{z}$5, supporting the three-gate design.

On COCO, DroneFINE retains 44.9 mAP versus 42.5 for full fine-tuning and 41.9 for Mona, demonstrating anti-forgetting of pre-trained generalization; inference overhead is only 0.43 FPS on a 2080 Ti.

Limitations and open questions

Several caveats bear on the results. The background vocabulary is constructed through a dataset-informed pipeline (sampling, GPT-4 generation, DINO-X validation, manual review); its transferability to aerial domains beyond streetscapes, terrain, and atmosphere is untested, and the manual review step introduces a non-trivial human dependency. The gating mechanism explicitly assumes $\mathbf{z}$6 and $\mathbf{z}$7 are meaningful and distinct—an assumption enforced only indirectly via the contrastive loss, whose removal degrades VisDrone performance though not UAVDT mAP$\mathbf{z}$8. The mAP$\mathbf{z}$9 deficit relative to ESOD on UAVDT remains unresolved beyond an attribution to differing localization priorities. Evaluation is confined to two datasets and one base detector (GroundingDINO); whether the domain-gap diagnosis and the two-module remedy generalize to other VLM detectors or other remote-sensing regimes is left open. Finally, the paper does not report training-cost comparisons beyond parameter counts, so wall-clock efficiency claims rest primarily on inference FPS.

Conclusion

DroneFINE reframes PEFT for VLM-based drone detection around two domain-specific requirements: representational capacity sufficient for high-rank adaptation, and active suppression of background interference. Its two components—foreground-aware hypernetwork-generated multi-path convolutions and text-conditioned background gating in query selection—together match full fine-tuning on VisDrone with 5.6% of the trainable parameters, exceed full fine-tuning on UAVDT, preserve COCO generalization better than full tuning, and set state-of-the-art results among compared methods on both benchmarks. The work establishes that targeted architectural awareness, rather than scale of adaptation, can close the ground-to-aerial domain gap for open-vocabulary detectors.