---
title: 'TSF: Target-Specific Tracked Features'
url: https://www.emergentmind.com/topics/target-specific-tracked-features-tsf
type: topic
---

# TSF: Target-Specific Tracked Features

Target-Specific Tracked Features (TSF) denotes, in its most explicit recent formulation, a set of referent-specific visual features extracted from tracked object regions across a video and injected into a multimodal large language model so that the model receives temporally aligned cues about the same referred object instance across time [2603.12382]. The term is not used uniformly across arXiv literature, however. In some works it refers to target-adaptive tracking representations, in others to target-specific but non-tracked feature learning, and in still others the acronym “TSF” names an unrelated method family such as Time Series Forest [2409.04542]. A rigorous account therefore has to distinguish the SPARROW definition from broader target-conditioned tracking mechanisms and from purely terminological overlaps.

## 1. Definition and terminological scope

In the SPARROW framework, TSF is defined as **referent-specific visual features extracted from tracked object regions across a video and injected into the multimodal input during training**. Its stated purpose is to provide “temporally aligned, referent-specific cues during training” and thereby enable “identity persistence without explicit tracking at inference” [2603.12382]. This definition is narrower than generic feature adaptation: TSF is not any target-aware descriptor, but a specific mechanism built from tracked crops, compacted into tokens, and appended to the language-model input.

The broader literature represented here shows that the phrase “target-specific” often survives while “tracked features” changes meaning. Some tracking papers instantiate TSF-like behavior without using the exact acronym, by dynamically selecting discriminative channels, storing per-track appearance attributes, or constructing template-conditioned fine-grained target descriptors. Other papers use “TSF” for unrelated constructs and must not be conflated with SPARROW’s notion.

| Context | Meaning | Core representation |
|---|---|---|
| SPARROW | Target-Specific Tracked Features | Tracked crop features projected as LLM tokens |
| Visual tracking papers | Target-adaptive tracked representation | Channel subsets, candidate tuples, template memories, or per-track attribute banks |
| Non-tracking papers | Different use of “TSF” | Time-series forests, polynomial features, or sparse signal-strength fields |

This heterogeneity matters because the same initials can denote different objects. A technically precise usage therefore treats **SPARROW TSF** as the canonical definition of “Target-Specific Tracked Features,” while viewing adjacent tracking papers as related feature-design paradigms rather than identical formulations.

## 2. TSF in pixel-grounded video MLLMs

SPARROW introduces TSF to address a specific failure mode of pixel-grounded video MLLMs: **temporal referential inconsistency**. The paper argues that prior systems such as VideoGLaMM, UniPixel, and GLUS often rely on a static segmentation token, typically **[SEG]**, which conveys semantics but lacks temporal context. As a result, models can segment the correct object in easy frames yet fail to preserve identity when motion, occlusion, reappearance, or distractor similarity becomes significant [2603.12382].

The failures are named explicitly. **Spatial drift** denotes gradual movement of the predicted region away from the intended referent. **Identity switches** occur when the model jumps from the correct instance to a similar distractor. **Ambiguous re-identification** arises when a target disappears and later reappears, but the model no longer knows which instance is the intended one. **Unstable initialization** refers to poor first-frame grounding that propagates downstream. SPARROW’s central distinction is between **semantic grounding** and **referential consistency**: a static [SEG] token indicates what category or phrase to ground, whereas TSF is intended to preserve which exact evolving instance is being discussed over the sequence.

SPARROW couples TSF with a separate **dual-prompt** design. TSF is responsible for temporal referential stability; the **[BOX] + [SEG]** pair handles spatial precision and initialization stability. The [BOX] token provides a coarse geometric prior, while [SEG] refines semantic mask grounding. The paper’s ablation on Ref-DAVIS17 shows that these mechanisms are complementary rather than redundant: baseline performance is 69.5 with no TSF and BOX off, 72.4 with train-only TSF and BOX off, 72.5 with BOX on and no TSF, 76.8 with train-only TSF and BOX on, and 77.7 with train+inference TSF and BOX on [2603.12382]. The decomposition supports the interpretation that TSF primarily stabilizes identity over time, whereas BOX chiefly improves geometric localization.

## 3. Construction and integration of TSF tokens

SPARROW constructs TSF from **tracked crops of the referred target**. Given a textual query \(Q\), an offline pseudo-supervision pipeline first detects the referent with **GroundingDINO**, then propagates it through the clip with **CLDTracker**, crops the tracked regions, encodes the crops with the spatial encoder \(\mathcal{F}_g\), compresses them by K-means, projects them into the LLM embedding space through the spatial V\(\to\)L adapter \(\mathcal{W}_g\), and appends the resulting tokens as \(Z_{\text{TSF}}\) [2603.12382].

The paper formalizes the multimodal input as
\[
\mathcal{I} = [\,Q;\,Z_g;\,Z_h;\,Z_{\text{TSF}}\,].
\]
Here \(Q\) is the tokenized query, \(Z_g\) the projected spatial visual features, \(Z_h\) the projected temporal visual features, and \(Z_{\text{TSF}}\) the projected target-specific tracked tokens. TSF is therefore injected at the **LLM input level** rather than into SAM2, an external memory bank, or the [BOX]/[SEG] decoder prompts.

Compression is intentionally aggressive. The tracked candidate boxes \(\mathcal{B}'=\{\mathbf{B}'_1,\ldots,\mathbf{B}'_{K'}\}\) are clustered in a joint visual–spatial feature space, and the sample nearest each centroid is retained to form \(\mathcal{B}=\{\mathbf{B}_1,\ldots,\mathbf{B}_K\}\) with \(K<K'\). In practice SPARROW sets **\(K=4\)**, so TSF adds **4 extra tokens per target**. The appendix reports that several projection paths were tested—no TSF, temporal-only, dual-path concatenation, dual-path summation, and spatial-only—and that **spatial-only projection** performs best, consistent with TSF’s intended role as an object-centric appearance and identity cue rather than a global temporal descriptor [2603.12382].

Training uses TSF primarily as a **training-time injection mechanism**. Stage 1, termed **Target-Specific Information Injection**, updates the V\(\to\)L adapters \((\mathcal{W}_g,\mathcal{W}_h)\), the L\(\to\)V SEG adapter \(\mathcal{W}_s\), and LoRA parameters inside the LLM, while keeping the visual backbone and pixel decoders frozen. The total loss is
\[
\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{mask}}, \qquad
\mathcal{L}_{\text{mask}}=\mathcal{L}_{\text{BCE}}+\mathcal{L}_{\text{DICE}}.
\]
Notably, no standalone TSF-specific temporal consistency loss is introduced; TSF changes the **supervision context** by augmenting the input with temporally aligned referent tokens.

## 4. Empirical behavior, dataset support, and operational limits

SPARROW supports TSF with a curated referential video corpus built from **HC-STVG, VID-Sentence, A2D Sentences, LaSOT, MeViS, GOT-10k, and Ref-SAV**, standardized into \((\text{video}, \text{referring text}, \text{trajectory}, \text{mask})\) tuples. The final corpus contains **30,646 video sequences** and **45,231 question-answer pairs**, and TSF tokens are precomputed offline for the model [2603.12382].

The clearest empirical isolation comes from Ref-DAVIS17. With BOX off, performance rises from **69.5** without TSF to **72.4** with train-only TSF and **75.3** with train+inference TSF. With BOX on, the corresponding values are **72.5**, **76.8**, and **77.7**. The appendix also reports robustness under corrupted pseudo-supervision: **76.8** with clean TSF, **76.0** with box jitter of \(\pm 10\%\), **75.4** with 10% ID-switch injection, **76.0** with 30% trajectory dropout, versus a **72.5** no-TSF baseline when BOX is on. These numbers indicate that even noisy TSF supervision remains above the no-TSF baseline.

At the system level, the full SPARROW method—attributed jointly to **TSF pretraining and dual-prompt design**—improves up to **+8.9 \(\mathcal{J\&F}\)** on MeViS, up to **+7.3 \(\mathcal{J\&F}\)** on Ref-DAVIS17, about **+5 mIoU** on VidSTG, and up to **+5.4 CLAIR** on VideoGCG. Practical costs are also stated explicitly. Offline TSF generation over all **30,646** clips is estimated at **51.9 hours on 1×A100**. Stage 1 training costs **12 h on 8×A100**, and full SPARROW adds **+32 GPU-hours** over baseline. Default inference omits TSF and therefore avoids detector/tracker overhead, but optional test-time TSF incurs latency
\[
\Delta t_{\text{TSF}}(n,T)\approx 0.093 + 0.020\,nT \ \text{s},
\]
which is about **18.1 s** for \(n=3\) targets and \(T=300\) frames, or about **20 ms/target/frame** [2603.12382].

These details delimit the method’s scope. TSF materially improves referential stability, yet its quality depends on pseudo-trajectories from GroundingDINO and CLDTracker, and optional inference-time use introduces nontrivial runtime. The paper also leaves some implementation details unspecified, including the exact TSF token embedding dimensionality and the exact definition of the joint visual–spatial clustering feature.

## 5. TSF as a broader tracking design pattern

Outside SPARROW, several tracking papers instantiate target-specific tracked representations without formalizing them as TSF tokens. In **S-KeepTrack**, designed for the **TSFMO** benchmark of small and fast moving objects, the target representation is a candidate tuple combining appearance, position, and confidence. KeepTrack uses \(z_i=f_i+\psi(s_i,c_i)\), while S-KeepTrack adds a low-level branch \(\bar z_i=\bar f_i+\psi'(s_i,c_i)\) and fuses high-level and low-level assignment matrices by \(S^*=\omega S+(1-\omega)\bar S\). The reported best setting is **\(\omega=0.2\)**, and the ablation shows that low-level cues contribute more than high-level cues for these tiny, fast targets [2209.04284]. This establishes a TSF-like principle: when appearance evidence is weak, tracked features must encode **candidate-local appearance, confidence, and position** rather than appearance alone.

**DFST** provides a different formulation. It extends the Adaptive Color Names tracker by dynamically re-ranking features at each frame using foreground pixels from the target box and background pixels from the immediate surround. The tracker operates over **10 Color Names channels** plus grayscale/luminance information, and selects **up to 8** channels after ranking them with a supervised adaptation of Infinite Feature Selection. In this setting, “target-specific” means that the chosen channel subset is the one that best discriminates the current target from its local background in the current frame [1609.01958].

In thermal infrared tracking, **DCFG** constructs a template-conditioned feature representation by dividing template channels into groups, suppressing dominant regions, and enforcing inter-group diversity. The paper reports best performance at **\(N=8\)** groups with suppression coefficient **\(\alpha=0.7\)**, and uses the diversity regularizer
\[
L_{DCFG} = \| coef(F_s) - I \|_2^2
\]
to push feature groups toward distinct discriminative target regions [2504.14311]. A related infrared design, **TSF-SiamMU**, decomposes the template into shallow and deep components and updates it through a multi-template residual rule,
\[
z_{final} = MU\big(Concat(z_{in}, z_{ac}, z_{cu})\big) + z_{in},
\]
thereby combining initial-template stability with current and accumulated appearance adaptation [2308.16676].

In online multi-object tracking, **FeatureSORT** makes target-specific tracked features explicit as **per-track memories**. Each trajectory stores a ReID embedding stack \(\mathcal{E}^{(i)}\), clothes-color stack \(\mathcal{C}^{(i)}\), clothes-style stack \(\mathcal{S}^{(i)}\), and direction state \(\mathcal{D}^{(i)}\), and matches new detections through a combined distance in which motion, edge, color, and style are weighted by \([0.1,\,0.4,\,0.25,\,0.25]\), while IoU and direction act as gates [2407.04249]. This usage is closer to classical target memory than to SPARROW’s token injection, but the conceptual link is direct: the tracked object is represented by persistent, target-specific cues rather than by geometry alone.

## 6. Related but distinct uses of “TSF”

Several arXiv papers use the initials “TSF” in ways that are technically adjacent or completely unrelated to SPARROW’s definition. **Slim-TSF** is explicitly a **Sliding Window Multivariate Time Series Forest** for solar-flare forecasting, built from interval statistics such as mean, standard deviation, and slope, followed by pooled max, min, and mean features. Its target is the temporal evolution of solar active regions over **12-hour observation segments** for predicting whether an **M/X-class** flare will occur in the next **24 hours**; this is not Target-Specific Tracked Features in the SPARROW sense [2409.04542].

**Tensor Machines** learn a small number of explicit **target-specific polynomial features** for supervised prediction. The learned features are products of linear forms,
\[
\phi_{p,i}(\mathbf{x}) = \prod_{j=1}^p \langle \bm{\omega}_j^{p,i}, \mathbf{x} \rangle,
\]
optimized directly from labels rather than sampled as target-agnostic random features [1504.01697]. These features are target-specific, but not tracked.

**CoDT** uses a **target-specific branch** in open-set cross-domain 3D action recognition, where target-domain skeleton sequences are encoded by \(F_1\) and \(\hat F_1\), learned through contrastive learning, online clustering, and collaborative pairwise consistency with a domain-shared branch [2207.09767]. Again, the term refers to target-domain specialization rather than tracked object identity.

Two further cases are closer to tracking but still differ materially from SPARROW TSF. A topological motion-discrimination method classifies small targets at range from **persistent-homology features of normalized sub-tracks**, treating motion dynamics rather than appearance as the discriminative signal [2102.05705]. The **TSSG** model, by contrast, tracks **target signal strengths on a spatial grid** via sparse Kalman filtering, so the state is a sparse field of signal-strength coefficients rather than an identity-bound per-target descriptor [1104.5288].

This suggests a useful editorial distinction. **TSF proper** can be reserved for the SPARROW mechanism: tracked referent crops converted into multimodal tokens for video MLLMs. **TSF-like methods** then describe a broader family of target-adaptive tracking representations: dynamic channel selection, candidate-level appearance–position features, template-conditioned fine-grained descriptors, multi-template infrared representations, per-track attribute banks, topological motion signatures, or sparse target-induced feature fields. Such a distinction preserves the precision of the SPARROW definition while acknowledging the larger design space of target-specific feature tracking [2603.12382].

Source: https://www.emergentmind.com/topics/target-specific-tracked-features-tsf