UAUTrack: Unified Transformer Anti-UAV Tracker

• UAUTrack is a unified transformer-based framework for multimodal single-object tracking in anti-UAV scenarios, integrating RGB, TIR, and text prompts.
• It employs multi-head self and cross-modal attention mechanisms to effectively fuse features, achieving superior performance on benchmark datasets.
• The framework uses a text prior prompt strategy to steer the network toward drone-specific features, enhancing accuracy and real-time efficiency.

UAUTrack is a unified transformer-based framework for multimodal single-object tracking, designed specifically for Anti-UAV (unmanned aerial vehicle) scenarios. It integrates RGB, thermal infrared (TIR), and text-based prompts into a single-stream, single-stage, end-to-end architecture. The system employs multi-head attention for both unimodal and cross-modal fusion and leverages a text prior prompt strategy to steer the network toward drone-specific feature representations. UAUTrack achieves state-of-the-art results across major Anti-UAV tracking benchmarks while maintaining practical efficiency in real-time applications (Ren et al., 2 Dec 2025).

1. Unified Architecture and Input Encoding

UAUTrack is built on a single-stream, single-stage transformer pipeline capable of ingesting various modality combinations without architectural changes. The framework pre-processes input as follows:

• RGB only: $I_{\rm RGB} \in \mathbb R^{H \times W \times 3}$
• TIR only (pseudo-colored): $I_{\rm TIR} \in \mathbb R^{H \times W \times 3}$
• RGB-TIR fusion: concatenation $$I_{\rm RT} = [I_{\rm RGB}; I_{\rm TIR}] \in \mathbb R^{H \times W \times 6}$$

For all modes, the system casts image pairs to $I_U \in \mathbb R^{H \times W \times 6}$. Both template and search regions are split into non-overlapping 16 × 16 patches and linearly projected to D-dimensional embeddings. Positional and token-type embeddings are added, producing tokens for each modality and spatial region.

The backbone consists of a stack of N transformer encoder layers, which jointly process the concatenated tokens from both visual and prompt sources. The final token sequence feeds into a lightweight detection head that outputs per-token classification scores and bounding-box regressions.

2. Multimodal Feature Fusion via Attention

Feature fusion in UAUTrack is accomplished using a combination of standard self-attention and cross-modal attention within the transformer encoders:

• Single-modality self-attention operates on subsets of tokens (RGB or TIR alone), computing:

$$A_S^M = \mathrm{Softmax}\left( \frac{[Q_t^M; Q_s^M][K_t^M; K_s^M]^\top}{\sqrt{d_k}} \right) [V_t^M; V_s^M]\,, \quad M\in\{R,T\}$$

• Cross-modal attention (RGB–TIR fusion) combines queries from RGB with keys/values from TIR:

$$A_C = \mathrm{Softmax}\left( \frac{[Q_t^R; Q_s^R][K_t^T; K_s^T]^\top}{\sqrt{d_k}} \right)[V_t^T; V_s^T]$$

This results in token-level fusion between RGB and TIR representations.

After each round of self- and cross-attention, all tokens are concatenated and forwarded to the next encoder, ensuring effective merging of multimodal cues at every layer.

3. Text Prior Prompt Strategy

UAUTrack incorporates semantic guidance with a text prior prompt. For each tracking instance, a prompt of the form “track a $G$ drone” (with $G$ as a size category based on bounding-box diagonal) is generated. The following process is used:

• The prompt is tokenized and encoded with a CLIP-L text encoder, yielding initial prompt tokens $H_L^0$.
• Stacked text transformer layers generate deeper representations:

$H_L^i = L_i(H_L^{i-1}), \quad i=1 \ldots K$

• Prompt tokens are linearly projected, partitioned into template and search subsets, and prepended to their respective visual token streams in every encoder layer:

$$H_F^i = E^{i-1}([p_{L_t}^{i-1}; H_t^{i-1}; p_{L_s}^{i-1}; H_s^{i-1}])$$

This mechanism modulates attention in a target-driven fashion, focusing the transformer on drone-like features and behaviors.

4. Loss Functions and Optimization

UAUTrack is trained with a composite objective encapsulating detection and prompt-guidance components: $$L_{\rm total} = \lambda_{\rm cls} L_{\rm cls} + \lambda_{\rm giou} L_{\rm giou} + \lambda_{L1} L_{L1} + \lambda_{\rm task} L_{\rm task}$$ where

• $L_{\rm cls}$: weighted focal loss for token-wise classification,
• $L_{\rm giou}$: generalized IoU loss for bounding box overlap,
• $L_{L1}$: L1 loss for coordinate regression,
• $L_{\rm task}$: prompt alignment loss driving consistency between text and image-token representations.

Training uses default weights ($\lambda_{\rm cls}=1$, $\lambda_{\rm giou}=2$, $\lambda_{L1}=5$, $\lambda_{\rm task}=1$), AdamW optimizer, initial learning rate $10^{-4}$, and 20 epochs with batch size 32 and 60K samples/epoch.

Online template updates occur every 25 frames if detection confidence exceeds 0.7; a Hanning window penalization is used for smoothness.

5. Experimental Results and Benchmark Performance

UAUTrack’s evaluation spans major Anti-UAV visual tracking datasets:

Dataset	Modalities	AUC (%)	Precision (%)	P_norm (%)	State Accuracy (%)	FPS
Anti-UAV	RGB+TIR	68.8	89.7	89.0	71.9	45
Anti-UAV410	TIR only	64.2	85.0	82.9	66.3	45
DUT Anti-UAV	RGB only	65.8	87.3	91.8	—	45

• On Anti-UAV, overall state accuracy reaches 74.0%, surpassing previous best results by ∼1.7%.
• On Anti-UAV410, UAUTrack achieves 5–6× higher frame rates at near state-of-the-art accuracy compared to SiamDT and GlobalTrack.
• DUT Anti-UAV performance improves AUC by 1.6% compared to the second-best method.

Ablation studies demonstrate the additive contributions of full fine-tuning and the text prompt: full fine-tuning yields the largest accuracy gain, and the text prior prompt further improves AUC by ≈1.5%. Combining unified multimodal fusion and text priors provides an additional ∼4.4% AUC advantage over single-modality systems.

6. Implementation Details and Ablation Insights

UAUTrack is implemented in PyTorch 2.1, utilizing Fast-iTPN as a backbone (pretrained with SUTrack for 180 epochs), and runs on two NVIDIA RTX 6000 ADA GPUs. Inference and fine-tuning hyperparameters are set to favor both accuracy and efficiency; batch resizing and search/template crops are chosen for optimal transformer throughput.

Ablations confirm that:

• Freezing the encoder yields substantially lower accuracy (AUC 61.6%),
• LoRA-adapter adaptation offers moderate improvement but falls short of full fine-tuning,
• The text prior prompt is essential, with a measurable increase in both AUC and precision,
• The complete pipeline combining text and multimodal fusion is necessary to reach the reported state-of-the-art.

7. Context, Limitations, and Future Directions

UAUTrack marks a significant advance in unified, multimodal, and prompt-driven visual tracking for UAV scenarios, integrating vision and text within a transformer-based end-to-end model and delivering high accuracy across modalities and operational conditions (Ren et al., 2 Dec 2025).

However, certain limitations persist:

• The framework is assessed principally on RGB and TIR data; further study is needed to generalize to other sensor types.
• Current modality fusion assumes availability of (pseudo-)RGB or TIR input, and does not address sequential or asynchronous multisensor data.
• The prompt-guidance mechanism is currently limited to a fixed vocabulary and structure, tied to drone tracking; expanding prompt flexibility and exploring zero-shot adaptation to new target types remain open challenges.

A plausible implication is that prompt-driven modulation in transformer tracking could extend beyond the UAV domain, providing a pathway toward more general, unified, and semantically-aware tracking systems operating across a wide sensor spectrum.