PromptTrack: Prompt-Conditioned Tracking

• PromptTrack is a family of tracking methods that use auxiliary prompts to condition target localization and identity association.
• It integrates visual, textual, and historical cues to enhance performance in autonomous driving, livestock tracking, and multimodal environments.
• Research demonstrates that prompt-conditioning improves detection and association accuracy while addressing challenges like computational cost and prompt drift.

PromptTrack denotes prompt-conditioned tracking in which an auxiliary prompt conditions target localization or identity association over time. In the cited literature, the term has both a narrow and a broad usage: it names a Transformer baseline for language-prompted 3D multi-object tracking in autonomous driving (Wu et al., 2023), it denotes an unsupervised OWLv2-plus-ByteTrack system in a livestock benchmark (Bibinbe et al., 15 Sep 2025), and it also functions as a broader label for prompt-based tracking methods that use textual, visual, historical, or latent prompt signals to adapt a tracker without relying only on conventional local matching (Yang et al., 2022, Cai et al., 2023, Wang et al., 28 Dec 2025, Zhu et al., 2023, Zhang et al., 15 Oct 2025).

1. Terminological scope and usage

Across recent tracking literature, “PromptTrack” is not a single standardized architecture. It is better understood as a family resemblance term centered on prompt-conditioned tracking, with different subfields instantiating the prompt in different ways.

Usage	Prompt form	Representative source
Autonomous-driving PromptTrack	Natural language prompt for 3D multi-view, multi-frame tracking	(Wu et al., 2023)
Livestock PromptTrack	Text-prompt detection with OWLv2 plus ByteTrack association	(Bibinbe et al., 15 Sep 2025)
Prompt-based multimodal tracking	Input-space visual prompts from multi-modal observations	(Yang et al., 2022)
Prompt-based single-object tracking	Historical masks/features, visual boxes, or instructions	(Cai et al., 2023, Wang et al., 28 Dec 2025, Zhu et al., 2023)
Prompt-based MOT association	Explicit and implicit CLIP-centered prompts for association	(Zhang et al., 15 Oct 2025)
Prompt-guided routing	Routing cues that activate experts rather than prompt tokens	(Tan et al., 7 Jul 2026)

This dispersion of meaning is technically important. In autonomous driving, PromptTrack is an end-to-end camera-only Transformer baseline built on PF-Track, with a prompt reasoning branch that selects prompt-referred 3D tracks (Wu et al., 2023). In livestock tracking, PromptTrack is instead described as using OWLv2 for textual prompt-based detection and ByteTrack for tracking, and is benchmarked as an unsupervised detection-plus-identification pipeline (Bibinbe et al., 15 Sep 2025). In visual tracking more broadly, prompt-based trackers include input-space visual prompting, history-conditioned prompting, instruction-conditioned prompting, and CLIP-based multimodal prompt construction (Yang et al., 2022, Cai et al., 2023, Zhu et al., 2023, Zhang et al., 15 Oct 2025).

A useful unifying characterization is that PromptTrack-style systems relocate part of the tracking problem into a prompt interface. The prompt may specify the target, bias search toward a spatial prior, summarize tracking history, serialize state variables into language, or route computation adaptively. What changes across papers is not the presence of conditioning, but the representational locus of the prompt.

2. Prompt representations and conditioning mechanisms

One major prompt family is input-space visual prompting. ProTrack, introduced for multi-modal single-object tracking, defines the multi-modal input as $X=\{V,A\}$ and constructs a prompted RGB-like image by

$$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$

Tracking then becomes

$$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$

Here the prompt is neither text nor a learned token; it is a colorized and blended visual perturbation designed to let a pre-trained RGB tracker operate on RGB-D, RGB-T, or RGB-E inputs without extra multi-modal training (Yang et al., 2022).

A second family is spatially grounded visual prompting. VPTracker reformulates vision-language tracking as a prompted multimodal reasoning problem for an MLLM. The conventional local-search formulation

$$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$

is replaced by a global formulation

$$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$

Its core prompt is a rectangle drawn directly onto the full current frame at the previous predicted location. This “region-level prompt” is intended to “prioritize region-level recognition and resort to global inference only when necessary,” while preserving full-frame perception (Wang et al., 28 Dec 2025).

A third family is historical prompting. HIPTrack uses refined historical foreground masks and historical visual features of the target as prompts. The refined foreground mask is procedurally the box mask intersected with the candidate-elimination mask, after which the historical prompt encoder combines the 4-channel search-image-plus-mask input with the search feature to generate a prompt value stored in memory. The decoder retrieves historical prompts by negative Euclidean distance between stored keys and current queries, then concatenates the decoded prompt with current search features before prediction (Cai et al., 2023).

A fourth family is instruction prompting. TrackGPT defines Instruction Tracking by replacing explicit box, mask, or referring-expression input with an implicit instruction: $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$ Its LVLM “brain” maps a reference frame and instruction to a text response plus two special outputs,

$$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$

where $\mathbf{Q}_R$ is a referring query and $\mathbf{Q}_p$ is a purport query. Prompting here means intent-conditioned target inference rather than explicit target description (Zhu et al., 2023).

A fifth family is dynamic text-plus-pseudo-token prompting. EPIPTrack introduces explicit prompts and implicit prompts for MOT. The explicit template is “A person with identity [ID] and a [ATTRIBUTE] of [VALUE].” The implicit prompt is

$[X]_1[X]_2[X]_3\ldots[X]_M\ [PART]\ [S^*],$

with $$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$0, four body-part placeholders, and a pseudo-word $$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$1 generated from the visual encoder. Explicit prompts serialize ID, score, speed, and depth; implicit prompts combine shared soft context with instance-conditioned pseudo-words, then pass both through the frozen CLIP text encoder for dynamic association-oriented alignment (Zhang et al., 15 Oct 2025).

A sixth, broader use is prompt-guided routing. SpecTrack does not use prompt tokens or text prompts. Its “Spectral Prompt Router” fuses semantic context, a spatial boundary cue, and a latent channel-variation cue to route search regions to a sparse subset of experts. The paper explicitly positions this as prompt-guided conditional computation rather than prompt tuning in the narrow sense (Tan et al., 7 Jul 2026).

3. Tracking formulations and architectural patterns

PromptTrack in autonomous driving is a query-based, camera-only, Transformer architecture derived from PF-Track. Each object query is

$$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$2

and the full query set is

$$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$3

After decoding and temporal refinement,

$$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$4

PromptTrack performs prompt-conditioned selection through

$$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$5

The prompt reasoning module is implemented with cross-modal attention between refined visual track queries and RoBERTa text embeddings, followed by an MLP that outputs a binary prompt relevance score per query (Wu et al., 2023).

The livestock-benchmark PromptTrack is structurally different. It is described as using OWLv2 for textual prompt-based detection and ByteTrack for tracking, and the benchmark paper does not report a custom identity-propagation module, prompt update mechanism, or prompt-conditioned memory bank. In that usage, PromptTrack is effectively an open-vocabulary tracking-by-detection system in which language drives detection and ByteTrack supplies temporal association (Bibinbe et al., 15 Sep 2025).

HIPTrack exemplifies a plug-and-play augmentation pattern. The feature extraction network is frozen, the historical prompt network and prediction head are trained, and the prompt enters after template-search interaction. The memory bank stores key-value pairs from up to $$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$6 tracked frames, updated every $$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$7 frames in the early stage and then every $$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$8 frames, with FIFO replacement. This design treats prompting as online retrieval of temporally accumulated target evidence rather than as static conditioning (Cai et al., 2023).

TrackGPT exemplifies a reasoning-front-end architecture. The LVLM outputs special tokens $$f(V_t, A_t) = \lambda * Color(A_t) + (1 - \lambda) * Color(V_t).$$9 and $$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$0, which are projected into

$$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$1

A modified SAM mask decoder then produces

$$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$2

and a rethinking mechanism re-invokes the LVLM when the purport score drops below threshold. This architecture separates semantic target inference from efficient temporal propagation (Zhu et al., 2023).

EPIPTrack exemplifies a CLIP-centered association architecture. Its Explicit Prompt Modulator, Implicit Prompt Modulator, Attribute Adapter, Multimodal Interaction Module, and Discriminative Feature Augmentor generate multimodal prompt embeddings that are matched to refined visual embeddings. The additional cost is then used in Track Reassociation and Fusion Refinement inside a standard online tracking-by-detection pipeline (Zhang et al., 15 Oct 2025).

VPTracker occupies a different point in the design space: the MLLM itself is the tracker. The input consists of the template image $$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$3, the prompted full frame $$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$4, and instruction text $$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$5; Qwen3-VL-4B is fully fine-tuned, including vision encoder, alignment layer, and LLM, with no separate motion model, re-detection head, verifier, or auxiliary temporal module (Wang et al., 28 Dec 2025).

4. Benchmark regimes and reported performance

The empirical record attached to PromptTrack-style systems spans markedly different tasks: multi-modal SOT, vision-language tracking, instruction-conditioned segmentation, livestock MAT/MOT, autonomous-driving 3D MOT, and prompt-driven association in generic MOT.

System	Benchmark	Reported result
PromptTrack (Bibinbe et al., 15 Sep 2025)	10-minute pig tracking	IDF1 0.66, MOTA 0.48, F1 score 0.76
PromptTrack (Wu et al., 2023)	NuPrompt	AMOTA 0.127, AMOTP 1.361, RECALL 43.5%, MOTA 0.135, IDS 146
ProTrack (Yang et al., 2022)	CDTB / DepthTrack / LasHeR / RGBT234	F-score 0.757 / 0.578; Precision/Success 50.9/41.9 and 78.6/58.7
HIPTrack (Cai et al., 2023)	LaSOT / GOT-10k	AUC 72.7; AO 77.4
TrackGPT (Zhu et al., 2023)	Refer-DAVIS$$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$6 / InsTrack	$$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$7 66.5; $$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$8 54.9
VPTracker (Wang et al., 28 Dec 2025)	TNL2K / TNLLT	AUC 64.9, PR 71.2, NPR 80.2; SR 62.5, PR 72.9, NPR 73.8
EPIPTrack (Zhang et al., 15 Oct 2025)	MOT17 / MOT20 / DanceTrack	HOTA 67.2 / 65.9 / 68.7
SpecTrack (Tan et al., 7 Jul 2026)	MUST / HOTC20 / GOT-10k	AUC 62.4 at 43.7 FPS; AUC 72.6; AO 79.3

The livestock benchmark is especially revealing for the narrow “PromptTrack” implementation. On a 10-minute video of 15 pigs, PromptTrack achieves $$tracker: \{f(V_t, A_t), f(V_1, A_1), B_1\} \rightarrow B_t.$$9, $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$0, $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$1, $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$2, and $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$3, while its unsupervised detection quality at $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$4 is $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$5, $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$6, and $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$7. The paper’s interpretation is that PromptTrack’s main advantage comes from better detection, and that PromptTrack and Track-Anything outperform idTracker while remaining comparable in some respects to ByteTrack (Bibinbe et al., 15 Sep 2025).

In autonomous driving, PromptTrack + PETR improves over CenterPoint + PETR from AMOTA $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$8 to $$T = \text{Crop}(I_1, B_1), \qquad S_t = \text{Crop}(I_t, B_{t-1}), \qquad B_t = M(T, L, S_t)$$9, AMOTP $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$0 to $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$1, Recall $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$2 to $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$3, and IDS $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$4 to $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$5. The branch ablation further shows that removing prompt reasoning drops AMOTA from $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$6 to $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$7, indicating that language-conditioned selection is not merely decorative but functionally necessary in the prompt-defined task (Wu et al., 2023).

For prompt-conditioned single-object tracking, the quantitative pattern is similar but the mechanism differs. HIPTrack reports LaSOT $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$8 AUC and GOT-10k $$T = \text{Crop}(I_1, B_1), \qquad I_t' = \text{VP}(I_t, B_{t-1}), \qquad B_t = MLLM(T, L', I_t').$$9 AO, with gains over DropTrack, OSTrack, and SeqTrack. VPTracker reports TNL2K $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$0 AUC, $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$1 PR, $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$2 NPR and TNLLT $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$3 SR, $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$4 PR, $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$5 NPR, with a prompted-vs-unprompted ablation on TNL2K of $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$6 against $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$7. The reported gains are modest in that ablation, but consistent across metrics and tied to distractor suppression (Cai et al., 2023, Wang et al., 28 Dec 2025).

Instruction prompting produces a different empirical signature. TrackGPT-13B reaches $$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$8 on Refer-DAVIS$$\{\mathbf{X}_{video}, \mathbf{X}_{inst}\} \rightarrow \mathcal{M}.$$9, reported as a new state of the art, and $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$0 on InsTrack after instruction tuning. The same model without InsTrack tuning obtains $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$1, so the shift from explicit referring expressions to implicit human-intent instructions is measurable and nontrivial (Zhu et al., 2023).

At the association level, EPIPTrack reaches HOTA $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$2, IDF1 $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$3, MOTA $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$4 on MOT17 test; HOTA $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$5, IDF1 $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$6, MOTA $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$7 on MOT20; and HOTA $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$8, IDF1 $$\mathcal{T}, \mathbf{Q}_{R}, \mathbf{Q}_{p}= \mathcal{F}_{Brain}(\mathbf{I}_{R}, \mathbf{X}_{inst}),$$9, MOTA $\mathbf{Q}_R$0 on DanceTrack. The strongest improvements are reported on DanceTrack, which the paper treats as a regime of similar appearance, complex non-linear motion, and frequent interaction (Zhang et al., 15 Oct 2025).

5. Relation to classical tracking paradigms

PromptTrack-style work is best understood not as a replacement for classical tracking, but as a reallocation of modeling burden. Tracktor is the canonical warning against over-ascribing gains to elaborate tracking machinery: it showed that a detector’s own bounding-box regressor could propagate identities across frames by reusing the previous box as a proposal,

$\mathbf{Q}_R$1

and that a strong detector already solved much of benchmark MOT, especially the easy cases (Bergmann et al., 2019).

PromptTrack-like systems depart from this baseline in several distinct ways. ProTrack does not learn a fusion module; it adapts data into an RGB-like prompted image so that a pre-trained RGB tracker can operate across modalities (Yang et al., 2022). HIPTrack does not replace Siamese-style matching; it conditions the prediction head with memory-retrieved historical prompts while keeping the backbone frozen (Cai et al., 2023). VPTracker rejects the local-search assumption itself and converts temporal continuity into a full-frame visual annotation supplied to an MLLM (Wang et al., 28 Dec 2025). EPIPTrack keeps the tracking-by-detection pipeline but augments association with dynamically updated multimodal prompts rather than only motion and ReID similarity (Zhang et al., 15 Oct 2025). PromptTrack in autonomous driving keeps query-based 3D tracking and adds prompt-conditioned object selection rather than redesigning the detector (Wu et al., 2023).

This comparison clarifies what prompting does and does not do. It does not abolish the need for detection quality, motion propagation, or association. Instead, it introduces additional conditioning channels: semantic disambiguation, spatial priors, target-history retrieval, or instruction-level intent. This suggests that prompt mechanisms are most valuable where detector regression or local matching are weak: large displacement, reappearance, visually similar distractors, long-term disappearance, implicit target specification, and crowded identity maintenance. That reading is directly aligned with VPTracker’s critique of local search, HIPTrack’s emphasis on deformation and occlusion, and EPIPTrack’s gains on crowded MOT and DanceTrack (Wang et al., 28 Dec 2025, Cai et al., 2023, Zhang et al., 15 Oct 2025).

A second contrast concerns explicit versus implicit association. Classical online MOT often centers on an affinity matrix plus Hungarian assignment. PromptTrack in autonomous driving remains within this query-based set-prediction tradition, but adds prompt reasoning to identify which tracked objects satisfy a language condition (Wu et al., 2023). EPIPTrack also remains association-centric, but promptifies the identity descriptor itself. TrackGPT goes further: the prompt must first be reasoned into a latent target query before segmentation/tracking can proceed (Zhu et al., 2023). The broader PromptTrack landscape therefore spans both association augmentation and prompt-defined target selection.

6. Limitations, ambiguities, and unresolved directions

A recurring limitation is computational cost. VPTracker processes each frame independently by feeding the full-resolution image into Qwen3-VL-4B and reports no FPS, per-frame latency, or memory footprint; the paper supports reading it as an effective but computationally expensive near-online tracker rather than a demonstrated real-time system (Wang et al., 28 Dec 2025). EPIPTrack is explicitly slower than common MOT baselines: on MOT17-Val, ByteTrack runs at $\mathbf{Q}_R$2 FPS, ByteTrack + ReID at $\mathbf{Q}_R$3 FPS, and ByteTrack + EPIP at $\mathbf{Q}_R$4 FPS, with EPM and IPM dominating runtime (Zhang et al., 15 Oct 2025). The livestock PromptTrack benchmark reports no hardware specification, no FPS, and no memory usage (Bibinbe et al., 15 Sep 2025).

A second limitation is prompt ambiguity and drift. Static text descriptions can become outdated in dynamic scenes, which is one of EPIPTrack’s motivations for explicit and implicit prompt updating (Zhang et al., 15 Oct 2025). MLLM-based prompting can be distracted by visually or semantically similar objects, which is precisely the ambiguity VPTracker’s location-aware rectangle prompt is meant to suppress (Wang et al., 28 Dec 2025). TrackGPT’s use of an LVLM introduces a different issue: the target is not explicitly given, so semantic grounding can become stale, requiring rethinking (Zhu et al., 2023).

A third limitation is incomplete long-term identity maintenance. The pig benchmark states that performance of most approaches decreases over time, especially the F1 score, and that long-term maintenance of identity remains a significant challenge for all tested methods (Bibinbe et al., 15 Sep 2025). HIPTrack reduces but does not eliminate dependence on the quality of stored history; the design still relies on predicted boxes and candidate-elimination masks for prompt construction (Cai et al., 2023). VPTracker likewise depends on the previous localization because the visual prompt is drawn from $\mathbf{Q}_R$5, so a bad prior can bias the MLLM even though the full-frame view partly mitigates it (Wang et al., 28 Dec 2025).

A fourth limitation is under-specification of prompt syntax and output schema. VPTracker does not reproduce a literal prompt string, fixed token template, coordinate serialization format, or structured output grammar, even though the method clearly relies on image-level visual annotation plus instruction text (Wang et al., 28 Dec 2025). The livestock benchmark does not report the exact text prompt used for PromptTrack, the OWLv2 variant, or association thresholds inherited from ByteTrack (Bibinbe et al., 15 Sep 2025). PromptTrack for autonomous driving provides the prompt reasoning equations and thresholds $\mathbf{Q}_R$6, $\mathbf{Q}_R$7, but not the exact Hungarian matching costs (Wu et al., 2023).

A fifth limitation is conceptual: prompts alone are insufficient. EPIPTrack’s ablations show that EPM + IPM alone can fail badly, and that strong visual adaptation is necessary before prompt modules become effective (Zhang et al., 15 Oct 2025). SpecTrack makes a related point from a different angle: the prompt is a routing descriptor that allocates capacity, not a standalone replacement for representation learning (Tan et al., 7 Jul 2026). A plausible implication is that mature PromptTrack systems will continue to combine prompt construction with strong visual backbones, temporal propagation, and explicit association logic rather than relying on prompt insertion alone.

PromptTrack therefore names not a settled method class but an active design space. Its current realizations already cover prompted multimodal adaptation, prompt-conditioned association, full-frame spatial prompting for MLLMs, historical prompt memory, and instruction-conditioned reasoning. The unresolved question is not whether prompts can help tracking; the cited literature shows that they can. The unresolved question is which prompt representation best couples semantic specificity, temporal adaptability, and computational tractability for the particular tracking regime under study.