UETrack: Unified Efficient Single Object Tracker

• UETrack is a unified tracking framework that integrates multiple modalities including RGB, depth, thermal, event, and language inputs in a single real-time pipeline.
• It leverages a Token-Pooling-based Mixture-of-Experts and target-aware adaptive distillation mechanism to enhance feature specialization and supervision efficiency.
• UETrack achieves competitive accuracy with low latency, running up to 221 FPS on embedded systems while maintaining under 15M parameters.

UETrack is a unified and efficient framework for single object tracking that is designed to handle multiple modalities within a single student network and a single training pipeline. It is formulated to accept RGB, RGB-Depth, RGB-Thermal, RGB-Event, and RGB+Language inputs without retraining, while maintaining real-time throughput on GPU, CPU, and embedded hardware. The framework addresses two limitations identified in prior efficient trackers: a concentration on RGB-only inputs and the high complexity of many multi-modal designs. Its core technical contributions are a Token-Pooling-based Mixture-of-Experts mechanism for lightweight feature specialization and a Target-aware Adaptive Distillation strategy that applies teacher supervision selectively according to sample characteristics (Kang et al., 2 Mar 2026).

1. Design objectives and unified modality handling

UETrack is motivated by the requirement that modern single object tracking satisfy both high accuracy and very low latency. The framework is explicitly designed around four goals: one-backbone, one-training-pipeline processing across five modality settings; light overall complexity with fewer than 15 M parameters and approximately 3 G FLOPs for the largest version; strong multi-modal feature modeling through a gating-free Mixture-of-Experts design; and further performance gains from distillation only when the teacher outputs are reliable (Kang et al., 2 Mar 2026).

The modality unification strategy is based on a common token interface. For RGB-Depth, RGB-Thermal, and RGB-Event inputs, UETrack stacks the 3-channel auxiliary map with the 3-channel RGB image to form a 6-channel image $I_c \in \mathbb{R}^{H \times W \times 6}$. For pure RGB and RGB+Language settings, it replicates the RGB channels to obtain the same 6-channel tensor, and in the language setting it additionally introduces a single text token from CLIP. Patch embedding consists of convolutional downsampling with stride $4$, followed by an MLP and two conv-merge layers, producing the token matrix $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$. The language token is denoted $T_l \in \mathbb{R}^{1 \times D}$ and is produced by a frozen CLIP text encoder followed by a linear projection. The final sequence $T \in \mathbb{R}^{L \times D}$ concatenates template tokens, search tokens, and $T_l$.

This organization suggests that UETrack treats modality heterogeneity primarily as a problem of input normalization and shared tokenization rather than as a need for distinct modality-specific backbones. In practical terms, that design choice is closely aligned with the framework’s stated emphasis on deployment efficiency.

2. Backbone structure and prediction pipeline

UETrack uses transformer backbones for both student and teacher models. The two share the same stacking of transformer blocks, except that in the student some feed-forward networks are replaced by TP-MoE modules. After the backbone, two separate center heads produce bounding-box classification and regression, one for the student and one for the teacher, yielding $p_s$ and $p_t$, respectively (Kang et al., 2 Mar 2026).

A distinct control module, the Distillation-Control Net or Adaptive Net, receives pooled features from the student and teacher, fuses them, and outputs a binary decision $\alpha \in \{0,1\}$ through Gumbel-Softmax. The binary control has an operational meaning: $\alpha = 1$ applies distillation for that sample, while $4$0 skips it.

The overall pipeline therefore contains three coupled mechanisms: modality-unified token construction, transformer-based tracking with selective TP-MoE replacement in the student, and sample-wise supervision control through adaptive distillation. This suggests that UETrack is not merely a faster backbone, but an integrated training-and-inference framework in which efficiency is distributed across representation, routing, and supervision design.

3. Token-Pooling-based Mixture-of-Experts

UETrack replaces a standard FFN in certain transformer layers with a Token-Pooling-based Mixture-of-Experts module. The notation is as follows: $4$1 denotes the input token sequence, $4$2 is the number of experts, $4$3 are compact expert tokens, $4$4 are continuous routing weights, and $4$5 is the refined output. The module is defined by

$4$6

Here, $4$7 is average pooling in consecutive subspaces of size $4$8, $4$9 is a linear projection plus reshape, $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$0 partitions soft-aggregated tokens into $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$1 groups along the sequence dimension, each $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$2 is a small FFN, and $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$3 concatenates expert outputs.

The stated benefits are threefold. First, the Softmax-based continuous routing removes discrete gating, token sorting, and cross-expert communication. Second, local average pooling preserves spatial locality before expert-level mixing. Third, each expert processes a distinct subset of pooled tokens, while the final output is softly merged back, allowing full parallelism and end-to-end learning (Kang et al., 2 Mar 2026).

Within the taxonomy of MoE-style tracking components, TP-MoE is therefore characterized by soft assignment and token pooling rather than by explicit gate dispatch. The ablations reported for UETrack indicate that this is not only an architectural simplification but also a speed-oriented choice: replacing TP-MoE with classic Gate-MoE leads to a smaller accuracy drop than removing TP-MoE entirely, but incurs a substantial speed penalty.

4. Target-aware Adaptive Distillation

UETrack’s Target-aware Adaptive Distillation mechanism is designed to decide when teacher supervision is helpful. The operational rule is explicit: for easy frames, such as those with a clear view and minimal occlusion, the teacher’s predicted center-heatmap and feature maps are treated as reliable and $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$4; for hard frames involving heavy blur, occlusion, or distractors, $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$5 so that noisy teacher signals are not transferred (Kang et al., 2 Mar 2026).

Let $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$6 denote the student and teacher center-probability maps, and let $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$7 be the one-hot ground truth. Classification, GIoU, L1, and auxiliary SUTrack task losses are aggregated as $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$8, $T_c \in \mathbb{R}^{D \times (H/16 \times W/16)}$9, $T_l \in \mathbb{R}^{1 \times D}$0, and $T_l \in \mathbb{R}^{1 \times D}$1. The distillation terms are

$T_l \in \mathbb{R}^{1 \times D}$2

The overall student objective is

$T_l \in \mathbb{R}^{1 \times D}$3

with $T_l \in \mathbb{R}^{1 \times D}$4, $T_l \in \mathbb{R}^{1 \times D}$5, $T_l \in \mathbb{R}^{1 \times D}$6, and $T_l \in \mathbb{R}^{1 \times D}$7.

The Adaptive-Net loss uses the surrogate prediction

$T_l \in \mathbb{R}^{1 \times D}$8

and $T_l \in \mathbb{R}^{1 \times D}$9 has the same form as the non-distillation part of $T \in \mathbb{R}^{L \times D}$0 but uses $T \in \mathbb{R}^{L \times D}$1 in place of $T \in \mathbb{R}^{L \times D}$2.

This design localizes the distillation decision at the sample level instead of applying teacher supervision uniformly. A plausible implication is that UETrack treats distillation as a conditional regularizer rather than as a universal training prior, which is consistent with the reported visualization result that blurry or occluded frames tend to trigger $T \in \mathbb{R}^{L \times D}$3.

5. Variants, training protocol, and deployment profile

UETrack is reported in three variants. UETrack-B is denoted as $T \in \mathbb{R}^{L \times D}$4; it uses the first 6 layers of the Fast-iTPN-T backbone and places TP-MoE at layer 6 with 8 experts. UETrack-S is denoted as $T \in \mathbb{R}^{L \times D}$5, and UETrack-T as $T \in \mathbb{R}^{L \times D}$6 (Kang et al., 2 Mar 2026).

Variant	Parameters and FLOPs	Speed
UETrack-B	13 M; 3.2 G	163 / 56 / 60 FPS on 2080Ti / i9-14900KF / Jetson AGX
UETrack-S	9 M; 2.5 G	approximately 183 / 68 / 67 FPS
UETrack-T	6 M; 1.8 G	approximately 221 / 83 / 77 FPS

Training uses COCO, LaSOT, GOT-10k, TrackingNet, VASTTrack, DepthTrack, VisEvent, LasHeR, OTB99, and TNL2K. The template size is $T \in \mathbb{R}^{L \times D}$7 and the search size is $T \in \mathbb{R}^{L \times D}$8, with bounding boxes enlarged by factors $T \in \mathbb{R}^{L \times D}$9. Augmentations are horizontal flip and brightness jitter. Optimization uses AdamW with backbone learning rate $T_l$0, learning rate $T_l$1 for the rest, and weight decay $T_l$2. Training runs for 500 epochs of 100k samples each, and the learning rate is multiplied by $T_l$3 after epoch 400. The hardware configuration is 2$T_l$480 GB Tesla A800 GPUs with batch size 128.

At inference time, only the student model is used, together with a standard Hanning-window penalty on search-region scores as in prior trackers. The reported deployment profile is explicitly aimed at resource-constrained settings: the largest model remains below 15 M parameters and is reported to run at approximately 60 FPS on Jetson AGX and CPU. This makes the framework technically notable not only for multimodal breadth but also for its attempt to preserve a single deployable student model across heterogeneous sensors.

6. Benchmark performance and ablation findings

UETrack is evaluated on 12 benchmarks across 3 hardware platforms and is reported to achieve a superior speed-accuracy trade-off compared to previous methods (Kang et al., 2 Mar 2026). For the largest variant, the principal benchmark results are as follows.

Setting	UETrack-B result	Comparative remark
RGB	69.2% AUC on LaSOT, 48.4% AUC on LaSOText, 82.7% AUC on TrackingNet, 72.6% AO on GOT-10k	Beats AsymTrack-B by +4.5%, +3.8%, +2.7%, +4.9%
RGB-Depth	68.3% EAO on VOT-RGBD22, 60.6% F on DepthTrack	Matches or slightly exceeds SUTrack-T on VOT-RGBD22 and is 1.6×–2.4× faster; +2.3% F over EMTrack on DepthTrack
RGB-Thermal	55.5% AUC on LaSHeR, 64.2% MSR on RGBT234	+1.6% / +0.4% vs SUTrack-T; +2.4% / +1.7% vs SDSTrack; 3.9×–18.7× faster
RGB-Event	59.2% AUC, 76.2% P on VisEvent	New state-of-the-art among real-time methods; +0.4% vs SUTrack-T
RGB-Language	58.0% AUC on TNL2K, 61.3% AUC on OTB99	+0.5% AUC on TNL2K vs SeqTrackv2; 7×–28× speedup on CPU/AGX

A representative deployment figure is that UETrack-B achieves 69.2% AUC on LaSOT while running at 163 FPS on GPU, 56 FPS on CPU, and 60 FPS on AGX. The smallest model, UETrack-T, still runs at 221 FPS, 83 FPS, and 77 FPS across the three platforms, with competitive accuracy against much heavier non-real-time trackers.

The ablations are reported as average changes across five representative datasets—LaSOT, DepthTrack, RGBT234, VisEvent, and TNL2K—together with AGX speed. For TP-MoE necessity, the baseline UETrack-B with TP-MoE and no distillation is reported as avg LAUC = 68.5, F = 59.4, …, speed = 60 FPS. Replacing TP-MoE with a standard FFN causes a $T_l$5 accuracy change and increases speed by 3 FPS. Replacing TP-MoE with classic Gate-MoE causes a $T_l$6 accuracy change but decreases speed by 21 FPS to 39 FPS. Removing local aggregation within TP-MoE causes a $T_l$7 accuracy change and increases speed by 1 FPS.

For expert count and insertion position, using 4 experts gives $T_l$8 average performance, 16 experts gives $T_l$9, and 32 experts gives $p_s$0, with the default set to 8 experts. Inserting TP-MoE in the last 2 layers gives $p_s$1, in the last 3 layers $p_s$2, and in even layers $p_s$3.

For distillation, adding KL only gives $p_s$4 average improvement, adding feature-mimic MSE gives $p_s$5, and adding adaptive gating $p_s$6 gives $p_s$7 average improvement over the baseline. Visualization studies further report that different experts attend to the center of object, contour edges, background clutter, and related regions, while TAD decision visualization shows $p_s$8 for blurry or occluded frames and $p_s$9 for clear frames. These results support the interpretation that TP-MoE contributes expert specialization and that adaptive distillation is most useful when teacher outputs are structurally reliable.

7. Limitations, deployment context, and future extensions

Three limitations are explicitly identified. First, TP-MoE is inserted only in deeper layers; early-layer specialization may help further, but is described as risking disruption of low-level features. Second, the CLIP text encoder is frozen, and joint fine-tuning for language is proposed as a possible route to higher performance at extra cost. Third, the framework is limited to single-object tracking; extension to multi-object scenarios requires additional association logic (Kang et al., 2 Mar 2026).

The deployment context emphasized for UETrack is heterogeneous sensing under strict efficiency budgets. The single student model supports RGB, depth, thermal, event, and language inputs and is described as suitable for robotic and embedded applications. In that sense, UETrack occupies a specific niche within single object tracking research: it is not only a multimodal tracker, but a multimodal tracker engineered around a unified deployable runtime.

The future directions listed for the framework are dynamic insertion of TP-MoE in multiple layers with learnable routing, semi-supervised or unsupervised adaptation of the student to field domain shifts, extension to long-term tracking through re-detection and memory, extension to multi-object tracking through lightweight object-management modules, and joint fine-tuning of the frozen language encoder for end-to-end vision-language alignment. Taken together, these directions suggest that UETrack is intended as a baseline unified architecture whose current formulation prioritizes efficiency and modality breadth over full generality in tracking scope.