---
title: 'UniTracker: Unifying Any-Modality Video Tracking'
url: https://www.emergentmind.com/topics/unitracker
type: topic
---

# UniTracker: Unifying Any-Modality Video Tracking

UniTracker, called **Un-Track** in the paper “Single-Model and Any-Modality for Video Object Tracking,” is a **single-model, any-modality video object tracker** designed to use **one set of parameters** across heterogeneous sensing modalities such as **RGB, depth, thermal, and event** data [2311.15851]. Its central premise is that multimodal tracking is constrained by three recurring practical conditions: the heterogeneity of modality-specific representations, the scarcity of large paired multimodal tracking datasets, and the fact that an auxiliary modality may be unavailable at test time. Un-Track addresses these conditions by learning a **shared latent embedding** that binds modalities into a common representation, and by embedding that representation into a single **transformer-based architecture** augmented with **modal prompting** and **LoRA-based finetuning**.

## 1. Problem setting and meaning of “any-modality”

Un-Track is formulated for multimodal video object tracking in settings where RGB may be paired with one auxiliary modality, such as depth, thermal, or event data. The method is explicitly motivated by the observation that existing modality-specific trackers, and even several “unified” trackers, often still require separate fine-tuning or separate parameter sets. In deployment terms, this is impractical when the available sensor package changes across datasets or applications.

Within this framework, **“any-modality”** means that a single trained tracker can accept **RGB+depth, RGB+thermal, RGB+event, or even RGB-only with dummy auxiliary input**. The paper defines this as a consequence of a shared latent space rather than a hard-coded fusion rule. “Handling missing modalities” therefore does not mean that all sensors are jointly modeled at all times; rather, it means that the tracker is not restricted to one fixed modality pairing and does not require all modalities to be present simultaneously.

A frequent misunderstanding is to read “any-modality” as arbitrary simultaneous multi-auxiliary fusion. The paper does not make that claim. It emphasizes **one auxiliary modality at a time** in practice, and it assumes access to **RGB-X pairs** during training. This makes the method unified in parameterization and representation, but not a general solution for unrestricted multi-auxiliary sensor fusion [2311.15851].

## 2. Shared latent embedding and cross-modal binding

The core representational idea is that different sensing modalities share some structure, especially **object boundaries / edges**, even when raw pixel statistics differ. Un-Track therefore learns a common latent space through two complementary mechanisms: **explicit edge-aware binding** and **implicit low-rank shared embedding**.

The explicit component computes gradient maps through x/y differences and uses them as a modality-agnostic cue. In the paper’s interpretation, depth, thermal, and event data often emphasize contours and boundaries, and these boundary signals can serve as a common anchor across modalities. This edge cue is denoted by \(G\).

The implicit component is framed as **low-rank factorization + reconstruction**. Instead of directly forcing all modalities into the same feature space, each modality is projected into a low-rank latent space with modality-specific MLP approximators. For depth \(D\), thermal \(T\), and event \(E\), the projections are
```latex
D_k = \sigma_d(D), \quad T_k = \sigma_t(T), \quad E_k = \sigma_e(E),
```
where \(\sigma_x\) maps channel dimension \(c\) to low-rank dimension \(k\), with \(k < c\). The gradient feature is also represented in low rank as \(G_k\).

The fused low-rank representation is then written as
```latex
M_k = \varphi_{R_1}([D_k, T_k, E_k]) + \varphi_{R_2}(G_k),
```
and the shared embedding is reconstructed as
```latex
F = \Phi_R(M_k) + G.
```
This makes \(F\) the shared cross-modal representation used downstream. A central property of this construction is that it is learned using only **RGB-X pairs**; the method does **not** require a fully paired RGB-D-T-E training set. This suggests that the intended unification target is representational alignment across partially overlapping modality pairs rather than exhaustive joint supervision across all sensors [2311.15851].

## 3. Modal prompting and transformer adaptation

After constructing the shared embedding \(F\), Un-Track performs cross-modal interaction through a **modal prompting** block. The prompting mechanism classifies RGB tokens by confidence into **negative**, **uncertain**, and **positive** groups through a learnable score function
```latex
m_n, m_u, m_p = s(I),
```
where \(I\) is the RGB feature. The prompting policy is token-wise shrinkage and exchange: **negative tokens** from RGB are replaced by tokens from the auxiliary modality, **uncertain tokens** are reconstructed using both RGB and the shared modality, and **positive tokens** are preserved.

The first low-rank RGB prompt feature is
```latex
I_{l_1} = \sigma_c(m_n \cdot F + m_p \cdot I).
```
Uncertain tokens are then processed as
```latex
I_{l_2} = \sigma_n(m_u \cdot F + m_u \cdot I).
```
These are fused in latent space,
```latex
I_l = \varphi_{P}([I_{l_1}, I_{l_2}]),
```
and the final fused output is
```latex
O = \Phi_P(I_l + F_l),
```
where \(F_l\) is the similarly processed auxiliary-modal low-rank representation. The paper characterizes this design as efficient because most computation occurs in low-rank space, so the prompting block adds only a small overhead while improving robustness.

Architecturally, Un-Track is built on a **transformer-based RGB tracker** based on **OSTrack**. The design includes **frozen pretrained backbone / tracker weights** as initialization, **outer modal prompting** for cross-modal feature enhancement, and **inner LoRA finetuning** for efficient adaptation. The LoRA update is applied to transformer attention weights. For a frozen weight matrix \(W_0 \in \mathbb{R}^{d \times k}\), two learnable low-rank matrices \(B \in \mathbb{R}^{d \times r}\) and \(A \in \mathbb{R}^{r \times k}\) are added as
```latex
h = W_0 + BA.
```
The training objective is not replaced by a new bespoke tracking loss; the paper states that training uses the **same loss as the baseline tracker OSTrack**, with the shared embedding, prompting, and LoRA modules added to that baseline structure [2311.15851].

## 4. Training regime, missing modalities, and practical constraints

The training regime is deliberately narrow in one sense and broad in another. It is narrow because the method assumes **RGB-X pairs** rather than arbitrary missing-label multimodal supervision. It is broad because those pairs are sufficient to learn a shared latent space that can bind different auxiliary modalities to RGB without requiring that **all modalities be available together**.

This distinction matters for interpreting the system’s robustness. The tracker can operate when an auxiliary input is unavailable because different modality inputs align to a common representation and because the model can fall back to **RGB-only with dummy auxiliary input**. However, the paper does not claim that missing-modality robustness is obtained from sensor-dropout training over complete multimodal bundles. The robustness arises from shared latent alignment and the single-parameter-set architecture.

The paper also identifies an important hyperparameter sensitivity: the shared embedding is learned from **low-rank factorization**, so the choice of rank matters. Ablation studies show that **too low or too high a rank can degrade performance**, although the method is described as robust overall. This makes the low-rank bottleneck a genuine modeling choice rather than a purely implementation-level compression trick.

From a systems perspective, the method emphasizes practicality. Training is reported as feasible on a **single 24GB GPU**, and the computational overhead is modest relative to the OSTrack baseline. This suggests that the proposed unification is intended not only as an abstract representational result but also as a deployable modification of a strong RGB tracker [2311.15851].

## 5. Empirical performance and ablation evidence

Un-Track is trained and evaluated on five benchmark datasets spanning different modalities: **DepthTrack** for RGB-D, **LasHeR** for RGB-T, **VisEvent** for RGB-Event, **VOT-RGBD2022** for cross-dataset RGB-D generalization, and **RGBT234** for cross-dataset RGB-T generalization.

The abstract reports three headline quantitative claims: **+8.1 absolute F-score gain** on **DepthTrack**, with only **+2.14 GFLOPs** and **+6.6M parameters** added over the RGB baseline. The efficiency table for RGB-only or dummy-depth input gives the concrete baseline and model sizes: **OSTrack** at **21.50 GFLOPs** and **92.08M params**, and **Un-Track** at **23.64 GFLOPs** and **98.73M params**.

| Benchmark | Reported highlight |
|---|---|
| DepthTrack | **+8.1 absolute F-score** gain; best or near-best results |
| LasHeR | Thermal-specific version sets a new SOTA; uni-model highly competitive |
| VisEvent | Event-specific version sets a new SOTA; unified version improves over prior unified baselines |
| VOT-RGBD2022 | Uni-model generalizes well across datasets |
| RGBT234 | Uni-model generalizes well across datasets |

Beyond the abstract-level numbers, the benchmark summaries are consistent in emphasizing two patterns. First, the **single-parameter-set** version outperforms prior unified trackers and, in some cases, modality-specific trackers. Second, the model exhibits strong **cross-dataset generalization**, especially on **VOT-RGBD2022** and **RGBT234**, where it outperforms depth- or thermal-specific baselines and previous unified trackers.

The ablation results support the architectural decomposition presented by the method. Removing the **shared embedding** hurts performance; replacing the prompting block with a weaker alternative hurts performance; removing **LoRA** hurts performance; and the low-rank choice materially affects results. These findings are methodologically significant because they indicate that Un-Track is not reducible to a generic multimodal fusion head attached to OSTrack: its reported gains depend on the joint effect of shared embedding learning, token-level prompting, and parameter-efficient transformer adaptation [2311.15851].

## 6. Place within unified tracking research

The term **“unified”** has multiple meanings in the tracking literature, and Un-Track occupies a specific position within that broader landscape. In **UCT**, unification refers to learning the feature extractor and the tracking process jointly as convolution operations in a real-time visual tracker [1711.04661; 1908.09445]. In **UTT**, unification refers to addressing **single object tracking** and **multiple object tracking** with one transformer-based paradigm [2203.15175]. In **OmniTracker**, the unifying axis is a shared architecture, model weights, and inference pipeline across **SOT, VOS, MOT, MOTS, and VIS** through a tracking-with-detection formulation [2303.12079]. In **OneTracker**, unification is organized around a **Foundation Tracker** and **Prompt Tracker** so that **RGB** and **RGB+X** tracking tasks can be handled within one framework, with non-RGB information treated as prompts [2403.09634].

Un-Track differs from those formulations because its unification target is not task type or training pipeline alone, but **heterogeneous sensing modality** within video object tracking. Its contribution is a **single-parameter-set transformer tracker for heterogeneous modalities** that combines **shared latent representation learning** via low-rank reconstruction, **explicit edge-based cross-modal binding**, **token-level modal prompting**, and **LoRA-efficient finetuning**. This places it in a narrower but practically consequential part of unified tracking research: the attempt to make one tracker usable across RGB-D, RGB-T, RGB-E, and degraded RGB-only deployment without retraining per modality [2311.15851].

A plausible implication is that Un-Track represents a shift from modality-specific engineering toward representation-centric unification. At the same time, the paper’s own constraints remain important: the model is learned from **RGB-X pairs**, emphasizes **one auxiliary modality at a time**, and depends on rank-sensitive low-rank embedding design. Within those limits, it provides a concrete formulation of **single-model, any-modality** tracking rather than merely a collection of modality-specific branches under a common name.

Source: https://www.emergentmind.com/topics/unitracker