UniTracker: Unifying Any-Modality Video Tracking

• The paper introduces Un-Track as a single-model, any-modality video object tracker that uses a shared latent embedding for multimodal fusion via low-rank factorization and modal prompting.
• UniTracker is defined by its unified approach to aligning heterogeneous modalities through explicit edge-aware binding and implicit low-rank reconstruction, ensuring robustness across input types.
• Empirical results demonstrate significant performance gains and cross-dataset generalization on benchmarks like DepthTrack, LasHeR, and VisEvent, validating its practical deployment.

UniTracker, called Un-Track in the paper “Single-Model and Any-Modality for Video Object Tracking,” is a single-model, any-modality video object tracker designed to use one set of parameters across heterogeneous sensing modalities such as RGB, depth, thermal, and event data (Wu et al., 2023). Its central premise is that multimodal tracking is constrained by three recurring practical conditions: the heterogeneity of modality-specific representations, the scarcity of large paired multimodal tracking datasets, and the fact that an auxiliary modality may be unavailable at test time. Un-Track addresses these conditions by learning a shared latent embedding that binds modalities into a common representation, and by embedding that representation into a single transformer-based architecture augmented with modal prompting and LoRA-based finetuning.

1. Problem setting and meaning of “any-modality”

Un-Track is formulated for multimodal video object tracking in settings where RGB may be paired with one auxiliary modality, such as depth, thermal, or event data. The method is explicitly motivated by the observation that existing modality-specific trackers, and even several “unified” trackers, often still require separate fine-tuning or separate parameter sets. In deployment terms, this is impractical when the available sensor package changes across datasets or applications.

Within this framework, “any-modality” means that a single trained tracker can accept RGB+depth, RGB+thermal, RGB+event, or even RGB-only with dummy auxiliary input. The paper defines this as a consequence of a shared latent space rather than a hard-coded fusion rule. “Handling missing modalities” therefore does not mean that all sensors are jointly modeled at all times; rather, it means that the tracker is not restricted to one fixed modality pairing and does not require all modalities to be present simultaneously.

A frequent misunderstanding is to read “any-modality” as arbitrary simultaneous multi-auxiliary fusion. The paper does not make that claim. It emphasizes one auxiliary modality at a time in practice, and it assumes access to RGB-X pairs during training. This makes the method unified in parameterization and representation, but not a general solution for unrestricted multi-auxiliary sensor fusion (Wu et al., 2023).

2. Shared latent embedding and cross-modal binding

The core representational idea is that different sensing modalities share some structure, especially object boundaries / edges, even when raw pixel statistics differ. Un-Track therefore learns a common latent space through two complementary mechanisms: explicit edge-aware binding and implicit low-rank shared embedding.

The explicit component computes gradient maps through x/y differences and uses them as a modality-agnostic cue. In the paper’s interpretation, depth, thermal, and event data often emphasize contours and boundaries, and these boundary signals can serve as a common anchor across modalities. This edge cue is denoted by $G$.

The implicit component is framed as low-rank factorization + reconstruction. Instead of directly forcing all modalities into the same feature space, each modality is projected into a low-rank latent space with modality-specific MLP approximators. For depth $D$, thermal $T$, and event $E$, the projections are $D$6 where $\sigma_x$ maps channel dimension $c$ to low-rank dimension $k$, with $k < c$. The gradient feature is also represented in low rank as $G_k$.

The fused low-rank representation is then written as $D$7 and the shared embedding is reconstructed as $D$8 This makes $F$ the shared cross-modal representation used downstream. A central property of this construction is that it is learned using only RGB-X pairs; the method does not require a fully paired RGB-D-T-E training set. This suggests that the intended unification target is representational alignment across partially overlapping modality pairs rather than exhaustive joint supervision across all sensors (Wu et al., 2023).

3. Modal prompting and transformer adaptation

After constructing the shared embedding $D$0, Un-Track performs cross-modal interaction through a modal prompting block. The prompting mechanism classifies RGB tokens by confidence into negative, uncertain, and positive groups through a learnable score function $D$9 where $D$1 is the RGB feature. The prompting policy is token-wise shrinkage and exchange: negative tokens from RGB are replaced by tokens from the auxiliary modality, uncertain tokens are reconstructed using both RGB and the shared modality, and positive tokens are preserved.

The first low-rank RGB prompt feature is $T$0 Uncertain tokens are then processed as $T$1 These are fused in latent space, $T$2 and the final fused output is $T$3 where $D$2 is the similarly processed auxiliary-modal low-rank representation. The paper characterizes this design as efficient because most computation occurs in low-rank space, so the prompting block adds only a small overhead while improving robustness.

Architecturally, Un-Track is built on a transformer-based RGB tracker based on OSTrack. The design includes frozen pretrained backbone / tracker weights as initialization, outer modal prompting for cross-modal feature enhancement, and inner LoRA finetuning for efficient adaptation. The LoRA update is applied to transformer attention weights. For a frozen weight matrix $D$3, two learnable low-rank matrices $D$4 and $D$5 are added as $T$4 The training objective is not replaced by a new bespoke tracking loss; the paper states that training uses the same loss as the baseline tracker OSTrack, with the shared embedding, prompting, and LoRA modules added to that baseline structure (Wu et al., 2023).

4. Training regime, missing modalities, and practical constraints

The training regime is deliberately narrow in one sense and broad in another. It is narrow because the method assumes RGB-X pairs rather than arbitrary missing-label multimodal supervision. It is broad because those pairs are sufficient to learn a shared latent space that can bind different auxiliary modalities to RGB without requiring that all modalities be available together.

This distinction matters for interpreting the system’s robustness. The tracker can operate when an auxiliary input is unavailable because different modality inputs align to a common representation and because the model can fall back to RGB-only with dummy auxiliary input. However, the paper does not claim that missing-modality robustness is obtained from sensor-dropout training over complete multimodal bundles. The robustness arises from shared latent alignment and the single-parameter-set architecture.

The paper also identifies an important hyperparameter sensitivity: the shared embedding is learned from low-rank factorization, so the choice of rank matters. Ablation studies show that too low or too high a rank can degrade performance, although the method is described as robust overall. This makes the low-rank bottleneck a genuine modeling choice rather than a purely implementation-level compression trick.

From a systems perspective, the method emphasizes practicality. Training is reported as feasible on a single 24GB GPU, and the computational overhead is modest relative to the OSTrack baseline. This suggests that the proposed unification is intended not only as an abstract representational result but also as a deployable modification of a strong RGB tracker (Wu et al., 2023).

5. Empirical performance and ablation evidence

Un-Track is trained and evaluated on five benchmark datasets spanning different modalities: DepthTrack for RGB-D, LasHeR for RGB-T, VisEvent for RGB-Event, VOT-RGBD2022 for cross-dataset RGB-D generalization, and RGBT234 for cross-dataset RGB-T generalization.

The abstract reports three headline quantitative claims: +8.1 absolute F-score gain on DepthTrack, with only +2.14 GFLOPs and +6.6M parameters added over the RGB baseline. The efficiency table for RGB-only or dummy-depth input gives the concrete baseline and model sizes: OSTrack at 21.50 GFLOPs and 92.08M params, and Un-Track at 23.64 GFLOPs and 98.73M params.

Benchmark	Reported highlight
DepthTrack	+8.1 absolute F-score gain; best or near-best results
LasHeR	Thermal-specific version sets a new SOTA; uni-model highly competitive
VisEvent	Event-specific version sets a new SOTA; unified version improves over prior unified baselines
VOT-RGBD2022	Uni-model generalizes well across datasets
RGBT234	Uni-model generalizes well across datasets

Beyond the abstract-level numbers, the benchmark summaries are consistent in emphasizing two patterns. First, the single-parameter-set version outperforms prior unified trackers and, in some cases, modality-specific trackers. Second, the model exhibits strong cross-dataset generalization, especially on VOT-RGBD2022 and RGBT234, where it outperforms depth- or thermal-specific baselines and previous unified trackers.

The ablation results support the architectural decomposition presented by the method. Removing the shared embedding hurts performance; replacing the prompting block with a weaker alternative hurts performance; removing LoRA hurts performance; and the low-rank choice materially affects results. These findings are methodologically significant because they indicate that Un-Track is not reducible to a generic multimodal fusion head attached to OSTrack: its reported gains depend on the joint effect of shared embedding learning, token-level prompting, and parameter-efficient transformer adaptation (Wu et al., 2023).

6. Place within unified tracking research

The term “unified” has multiple meanings in the tracking literature, and Un-Track occupies a specific position within that broader landscape. In UCT, unification refers to learning the feature extractor and the tracking process jointly as convolution operations in a real-time visual tracker (Zhu et al., 2017, Zhu et al., 2019). In UTT, unification refers to addressing single object tracking and multiple object tracking with one transformer-based paradigm (Ma et al., 2022). In OmniTracker, the unifying axis is a shared architecture, model weights, and inference pipeline across SOT, VOS, MOT, MOTS, and VIS through a tracking-with-detection formulation (Wang et al., 2023). In OneTracker, unification is organized around a Foundation Tracker and Prompt Tracker so that RGB and RGB+X tracking tasks can be handled within one framework, with non-RGB information treated as prompts (Hong et al., 2024).

Un-Track differs from those formulations because its unification target is not task type or training pipeline alone, but heterogeneous sensing modality within video object tracking. Its contribution is a single-parameter-set transformer tracker for heterogeneous modalities that combines shared latent representation learning via low-rank reconstruction, explicit edge-based cross-modal binding, token-level modal prompting, and LoRA-efficient finetuning. This places it in a narrower but practically consequential part of unified tracking research: the attempt to make one tracker usable across RGB-D, RGB-T, RGB-E, and degraded RGB-only deployment without retraining per modality (Wu et al., 2023).

A plausible implication is that Un-Track represents a shift from modality-specific engineering toward representation-centric unification. At the same time, the paper’s own constraints remain important: the model is learned from RGB-X pairs, emphasizes one auxiliary modality at a time, and depends on rank-sensitive low-rank embedding design. Within those limits, it provides a concrete formulation of single-model, any-modality tracking rather than merely a collection of modality-specific branches under a common name.