AVTrack: Adaptive and Robust Tracking

• AVTrack is a multifaceted tracking concept that includes real-time UAV visual trackers using adaptive Vision Transformers and audio-visual benchmarks for complex scene analysis.
• It leverages adaptive computation with per-block gating and view-invariant representation learning to maintain efficiency and robustness under occlusion and appearance changes.
• The concept also inspires tracking-by-verification methods that combine fast, per-frame tracking with slower corrective mechanisms for improved accuracy in challenging scenarios.

AVTrack is a name used for several distinct research artifacts in tracking literature. In the materials summarized here, the term refers primarily to two concrete entities: a real-time UAV visual tracker based on adaptive computation and view-invariant Vision Transformers, together with its distilled variant AVTrack-MD (Wu et al., 2024); and a human-centric audio-visual instance segmentation and speaker-tracking benchmark for complex scenes, together with the baselines AVTracker and AVTrackFormer (Wang et al., 1 Jun 2026). In a looser historical sense, “AVTrack”-style tracking also denotes tracking-by-verification systems that separate a fast tracker from a slower verifier, as exemplified by PTAV (Fan et al., 2018). The shared theme across these usages is not a single unified algorithm, but a family resemblance: explicit treatment of robustness under appearance change, ambiguity, or occlusion, coupled with design choices intended to preserve practical efficiency.

1. Terminological scope and research contexts

Within the supplied sources, “AVTrack” is not monosemous. The most explicit use is the title “Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking,” which defines AVTrack as a family of real-time UAV trackers built on Vision Transformers with adaptive block activation and mutual-information-based view-invariant representation learning (Wu et al., 2024). A separate 2026 work uses the same name for “AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes,” a test-only benchmark for audio-visual speaker tracking and audio-visual instance segmentation in dynamic, human-centric videos (Wang et al., 1 Jun 2026).

A broader, weaker usage also appears in the PTAV material, where an “AVTrack”-style system is described as “tracking-by-verification / parallel tracking and verifying”: a fast tracker $T$ runs on every frame, while a verifier $V$ is invoked sparsely to validate and correct drift (Fan et al., 2018). This usage is descriptive rather than titular, but it establishes a conceptual lineage in which “AVTrack” denotes explicit separation of a cheap tracking path and a more discriminative verification or recovery mechanism.

A related but distinct point appears in CTTrack, where “AVTrack” is treated as part of a family of transformer trackers that heavily engineer attention for template-search communication; CTTrack argues that such specialized attention structures can often be understood as subsets of packed self-attention and that representation learning, rather than attention redesign, is the main bottleneck (Song et al., 2023). This suggests that later uses of the AVTrack name sit within a broader debate about how much architectural specialization tracking actually requires.

2. AVTrack as an adaptive, view-invariant UAV tracker

In its most concrete tracker instantiation, AVTrack addresses real-time UAV visual tracking under resource constraints and extreme viewpoint changes (Wu et al., 2024). The stated problem setting emphasizes airborne deployment, where limited onboard compute must coexist with robustness to camera motion, scale change, motion blur, low resolution, occlusion, and clutter. AVTrack therefore combines a single-stream ViT backbone with two design principles: adaptive computation through per-block gating, and view-invariant representation learning through mutual information maximization.

The tracker processes template and search images jointly as a single token sequence. The final ViT output is partitioned into template and search tokens,

$$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$

and a prediction head produces a classification map, offset map, and size map. The coarse target center is selected by

$(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$

and the final box is formed from the offset and size predictions,

$$\{(x_t,y_t);(w,h)\}=\{(x_c,y_c)+\mathbf{o}(x_c,y_c);\ \mathbf{s}(x_c,y_c)\}.$$

Training uses a standard tracking objective composed of focal classification loss, GIoU loss, and $L_1$ regression loss, augmented by sparsity and view-invariance terms: $$\mathcal{L}_{\text{overall}} = \mathcal{L}_{\text{cls}} + \lambda_{\text{iou}}\mathcal{L}_{\text{iou}} + \lambda_{L_1}\mathcal{L}_{L_1} + \gamma \mathcal{L}_{\text{spar}} + \kappa \mathcal{L}_{\text{vir}}$$ with $\lambda_{\text{iou}}=2$, $\lambda_{L_1}=5$, $\gamma=50$, and $V$0 (Wu et al., 2024).

The design objective is not merely accuracy. AVTrack is explicitly presented as a real-time system for mobile UAV platforms, and the reported speeds reflect this emphasis. On a desktop GPU, AVTrack-DeiT runs at 256.8 FPS, AVTrack-ViT at 250.2 FPS, and AVTrack-EVA at 283.7 FPS; on Jetson AGX Xavier, AVTrack-DeiT reaches 42.3 FPS (Wu et al., 2024). This speed regime places the work in a different operating region from heavier transformer trackers that optimize absolute benchmark accuracy without onboard deployment constraints.

3. Activation Module and adaptive computation

The computational adaptation mechanism in AVTrack is the Activation Module (AM), attached to transformer blocks after an initial set of always-active early layers (Wu et al., 2024). For block $V$1, the module computes an activation probability

$V$2

where $V$3 is a compact summary derived from the incoming token set and $V$4 is a block-specific linear layer. If $V$5, the transformer block executes; otherwise, it is skipped and its input tokens are copied forward unchanged. The first $V$6 blocks are always active to avoid degenerate behavior in which the network never forms useful template-search interactions.

To prevent the trivial solution in which all later blocks remain active, AVTrack introduces a block sparsity loss

$V$7

where $V$8 controls the desired average activation level (Wu et al., 2024). This is structurally regular pruning at the block level rather than unstructured token pruning, which the paper presents as implementation-friendly on GPUs and embedded devices.

The reported computational profile reflects this block-level adaptivity. AVTrack-DeiT has dynamic FLOPs in the range 0.97–2.4 GFLOPs and parameters in the range 3.5–7.9 M (Wu et al., 2024). A plausible implication is that AVTrack treats computational budget as an input-conditioned variable rather than a fixed property of the tracker, which is particularly useful for UAV scenes where some frame pairs are easy and others are highly ambiguous.

4. View-invariant representation learning and AVTrack-MD

AVTrack’s second core principle is explicit view-invariant representation learning. During training, the tracker extracts two views of the same target: the template crop $V$9 and the target patch $$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$0 inside the search image $$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$1. Let the final template and target-region search tokens be $$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$2 and $$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$3. AVTrack maximizes mutual information between them using the Jensen–Shannon estimator from Deep InfoMax, defining the view-invariant representation loss as

$$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$4

(Wu et al., 2024). This auxiliary loss is used only during training and adds no inference cost.

Building on AVTrack, AVTrack-MD introduces a multi-teacher knowledge-distillation scheme in which AVTrack-DeiT, AVTrack-ViT, and AVTrack-EVA serve as frozen teachers for a shallower student network with half the number of ViT blocks (Wu et al., 2024). Teacher features are averaged,

$$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$5

softened with temperature $$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$6, and matched to student features by maximizing mutual information rather than minimizing MSE: $$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$7 The student is trained with

$$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$8

where $$\mathbf{t}_{1:\mathcal{K}}^{\infty}(Z,X)=\mathbf{t}_{\mathcal{K}_Z}^{\infty}(Z,X)\cup \mathbf{t}_{\mathcal{K}_X}^{\infty}(Z,X),$$9 is reported as empirically optimal (Wu et al., 2024).

The main quantitative consequence is a speed-accuracy trade-off that remains close to the teacher baseline. AVTrack-MD-DeiT uses about 1.5 GFLOPs and about 5.3 M parameters, runs at 310.6 FPS on a desktop GPU and 46.1 FPS on Jetson AGX Xavier, and is about 21% faster on GPU than AVTrack-DeiT while incurring only a 0.5% precision drop and a 0.3% success drop on average across five UAV benchmarks (Wu et al., 2024). AVTrack-MD-ViT even exceeds its teacher in the reported aggregate benchmark average, with Avg Prec 83.4 versus 82.9 and Succ 64.0 versus 63.1 (Wu et al., 2024).

5. Empirical performance of the UAV-tracking AVTrack family

The AVTrack family is evaluated on DTB70, UAVDT, VisDrone2018 SOT, UAV123, UAV123@10fps, and WebUAV-3M, using Precision, Success rate, and FPS (Wu et al., 2024). Against 22 lightweight trackers, all AVTrack and AVTrack-MD variants exceed 80% average precision and 62% success in the reported aggregate summary. AVTrack-DeiT reaches Avg Prec 84.1 and Succ 64.4, second only to Aba-ViTrack’s 85.3 and 64.7, while remaining 41% faster on GPU and 18% faster on CPU than Aba-ViTrack (Wu et al., 2024).

The distilled models preserve most of this accuracy. AVTrack-MD-DeiT reports Avg Prec 83.6 and Succ 64.1 while reaching 310.6 FPS on GPU, and AVTrack-MD-EVA is the fastest compared tracker at 334.4 FPS (Wu et al., 2024). On WebUAV-3M, AVTrack-DeiT reports Prec 70.0, Succ 56.4, and AGX.FPS 42.3, whereas AVTrack-MD-DeiT reports Prec 69.4, Succ 55.3, and AGX.FPS 46.1 (Wu et al., 2024). These numbers are presented as evidence that the family is suitable for actual onboard deployment rather than only laboratory evaluation.

When compared with heavier deep trackers, the AVTrack models yield a markedly different accuracy-efficiency profile. The paper reports that AVTrack-DeiT and AVTrack-MD-DeiT remain below the absolute best average precision and success scores of trackers such as HIPTrack, EVPTrack, and ROMTrack, but AVTrack-MD-DeiT runs at 310.6 FPS, versus 32.1 FPS for HIPTrack and 26.1 FPS for EVPTrack (Wu et al., 2024). This suggests that the research contribution is primarily an efficiency-robustness balance for UAV-specific tracking rather than a claim to dominate all unconstrained tracking benchmarks.

6. AVTrack as an audio-visual benchmark for human-centric complex scenes

A separate research line uses the same name for a benchmark rather than a tracker. In “AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes,” AVTrack is a test-only dataset for audio-visual speaker tracking and audio-visual instance segmentation in dynamic real-world videos (Wang et al., 1 Jun 2026). The benchmark contains 871 clips, averages about 54 seconds per clip, and provides 3,120 instance tracklets with dense, temporally consistent masks for sounding humans only (Wang et al., 1 Jun 2026). Its sources include TV series and films, vlogs, animations, reality shows, interviews, and stage performances.

The benchmark is explicitly designed around eight complexity criteria: visual occlusion, relative position change, background switch, camera motion change, multiple instances, multi-turn sounding, audio-visual inconsistency, and instance scale dynamics (Wang et al., 1 Jun 2026). The prevalence of several of these conditions is substantially higher than in AVISeg: visual occlusion 80.9% versus 8.6%, background switch 60.5% versus 5.9%, relative position change 70.7% versus 9.7%, camera motion change 90.5% versus 7.1%, and multi-turn sounding 56.8% versus 16.0% (Wang et al., 1 Jun 2026). The stated goal is to move evaluation away from simple frame-wise audio-visual co-occurrence and toward robust spatiotemporal and cross-modal reasoning.

Evaluation uses TrackEval metrics adapted to mask-level multi-object tracking. The main metrics are HOTA,

$(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$0

Detection Accuracy,

$(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$1

Association Accuracy,

$(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$2

IDF1,

$(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$3

and MOTA,

$(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$4

(Wang et al., 1 Jun 2026). The use of tracking metrics rather than only segmentation AP aligns the benchmark with long-range identity preservation under cross-modal ambiguity.

The benchmark results reveal strong performance degradation for existing methods. Vision-only VIS baselines remain near HOTA 10–11, whereas AVIS methods such as AVISM and ACVIS reach only about 20–21 HOTA (Wang et al., 1 Jun 2026). AVTrackFormer, the paper’s end-to-end transformer baseline, records HOTA 21.47, DetA 22.51, AssA 20.26, IDF1 26.41, and MOTA 4.11 (Wang et al., 1 Jun 2026). The stronger modular baseline AVTracker reaches HOTA 29.08, DetA 31.18, AssA 28.47, IDF1 34.55, and MOTA 16.20 (Wang et al., 1 Jun 2026). Even Gemini 2.5 Pro in zero-shot mode reaches only HOTA 14.4, below the dedicated AVIS baselines (Wang et al., 1 Jun 2026).

This benchmark usage of AVTrack is conceptually distant from the UAV tracker of the same name. One is a testbed for multimodal scene understanding and association; the other is an onboard-efficient single-object visual tracker. The commonality lies in the emphasis on robustness under ambiguity, distractors, and long temporal horizons.

7. Relation to earlier “AVTrack”-style tracking and broader transformer-tracking debates

The PTAV framework provides an earlier conceptual model for an “AVTrack”-style system as asynchronous tracking-by-verification (Fan et al., 2018). PTAV separates a fast tracker $(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$5 and a verifier $(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$6, running in parallel threads. The tracker operates on every frame and issues verification requests every $(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$7 frames, while the verifier checks the candidate via a Siamese CNN and, if necessary, performs local re-detection and returns a correction. The tracking thread does not block on the verifier; instead, it may roll back to a correction frame when feedback arrives. With a default interval $(x_c,y_c)=\arg\max_{(x,y)} \mathbf{p}(x,y),$8, PTAV reports 27 FPS on OTB2015 and a DPR of 86.2%, while its dynamic template pool improves both speed and accuracy over a fixed-template verifier (Fan et al., 2018).

That architecture is not named AVTrack in the paper title, but the supplied material explicitly states that an “AVTrack”-style system is “exactly this: tracking-by-verification / parallel tracking and verifying” (Fan et al., 2018). This historical usage matters because it foregrounds a recurring design pattern: robustness by combining a cheap per-frame mechanism with a more discriminative but slower corrective mechanism.

By contrast, CTTrack uses AVTrack as a representative of attention-engineered transformer trackers and argues that such designs may be over-specialized (Song et al., 2023). CTTrack’s central claim is that packed self-attention over concatenated template and search tokens is sufficient for information aggregation, and that the real bottleneck is feature learning. Its correlative masked modeling objective improves LaSOT AUC from 64.0 to 65.8 without altering inference complexity, and the paper explicitly states that many trackers with cross-attention, hierarchical attention, or mixed attention—including AVTrack-like designs—can be interpreted as subsets of packed self-attention (Song et al., 2023). This suggests that the UAV AVTrack’s distinctive contribution lies less in inventing a novel template-search attention pattern than in combining conditional computation and MI-based regularization.

A plausible synthesis across these usages is that “AVTrack” marks a recurring concern in tracking research: the inadequacy of naïve per-frame matching under real-world complexity. PTAV addresses it by asynchronous verification (Fan et al., 2018), the UAV AVTrack by adaptive computation and view-invariant representation learning (Wu et al., 2024), and the audio-visual AVTrack benchmark by revealing how severely current methods fail once occlusion, multi-turn sounding, and cross-modal inconsistency are made explicit (Wang et al., 1 Jun 2026). The term therefore denotes not one settled architecture, but a set of responses to robustness, ambiguity, and deployment constraints across different tracking domains.