Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

Abstract: Vision-language tracking guided by natural language specifications leverages high-level semantic cues of target objects to substantially boost tracking accuracy and robustness. Existing studies have verified that adaptively optimizing textual descriptions throughout the tracking process can effectively mitigate the semantic-visual mismatch induced by dynamic variations in target appearance, position, and other inherent attributes. Nevertheless, mainstream methods that directly generate textual information via sequence models or LLMs inevitably suffer from inherent defects, including erroneous target updating, excessive background distraction, and pervasive hallucination artifacts. To address the aforementioned limitations, this paper proposes a novel language dependency parsing mechanism to precisely distill core tracking principal components, encompassing target objects, semantic concepts, and background contextual information. On this basis, we perform component-aware adaptive textual description updates by exploiting the powerful cross-modal understanding capability of the pre-trained vision-LLM Qwen-VL. By integrating the proposed elaborately designed modules into the baseline framework, our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks, including TNL2K, LaSOT, TNLLT, and OTB-LANG. The source code and pre-trained models will be released at https://github.com/Event-AHU/Open_VLTrack.

• The paper introduces a structured text update pipeline that decomposes queries into target, concept, and background triplets to enhance tracking accuracy.
• It employs a dependency parsing module and Qwen-VL refinement with target-conditioned Top-K updates to maintain semantic integrity and counteract noise.
• Empirical results across multiple benchmarks demonstrate improved precision, AUC, and robustness over traditional vision-language trackers.

Dynamic Parsing and Fine-Grained Natural Language Specification for Robust Vision-Language Tracking

Motivation and Context

Vision-language tracking (VLT) leverages natural language specifications to guide visual object tracking by establishing robust cross-modal correspondence between textual cues and visual content. While traditional visual trackers rely on image templates and bounding boxes, VLT supplements target localization with semantic descriptions. This enhances robustness against appearance variations, occlusion, and distractors, critical for applications such as UAV tracking, autonomous driving, and HCI. The field has evolved from early LSTM-based approaches [li2017tracking] and dataset proposals (e.g., TNL2K [wang2021tnl2k]) to unified vision-language frameworks with multimodal feature extractors [ma2024unifying].

Existing strategies for updating text guidance dynamically via LLMs and sequence models have demonstrated efficacy but suffer from three fundamental limitations: (1) erroneous target updating due to unstructured text generation, (2) semantic drift caused by background distraction, and (3) hallucination artifacts in free-form language generation. These deficits are especially pronounced in long-term scenarios with complex appearance and contextual changes.

Figure 1: Fine-grained text update contrasts holistic updating paradigms, showing decomposition into target, concept, and background fields, with selective, structured refinement.

Methodology

Fine-Grained Text Update Paradigm

The paper introduces a structured, progressive text update pipeline for VLT. The approach decomposes the raw language query into three distinct semantic fields: target identity, appearance concepts, and background context. This decomposition utilizes a dependency parsing module based on biaffine attention, trained on UD-EWT [dozat2017deep], to extract syntactic structure and assign functional roles to tokens.

Each triplet is then refined using a Qwen-VL module [wang2024qwen2VL], which operates in a role-aware manner, preserving the invariance of the target identity while rewriting concept and background fields to ensure compactness, discriminativity, and suitability for tracking. The Qwen-VL refinement is further supervised via tracking-aware finetuning, where candidates are selected based on their downstream tracking AUC.

Concept field updates are executed via target-conditioned Top-K visual modulation: cosine similarity between the target embedding and search-region visual tokens identifies the Top-K most relevant evidence, which is integrated back into the concept field using span-local cross-attention. The background and target fields remain unchanged to maintain semantic stability.

(Figure 2)

Figure 2: Architecture overview illustrating the triplet parsing, Qwen refinement, and target-conditioned concept updating for vision-language tracking.

Integration into Tracking Backbone

The final updated triplet is fused with the template and search visual features in a unified tracking backbone. Three separate embeddings (target, concept, background) are used for cross-modal interaction, with the concept embedding dynamically evolving to reflect current visual evidence.

Training is end-to-end, optimized via center-based prediction heads with localization (focal loss) and regression (L1 + GIoU) objectives. The complete pipeline achieves dynamic structured guidance in real-time tracking.

Language Dependency Parsing Details

A Transformer encoder processes the query tokens, followed by biaffine classifiers for dependency arc and label scoring. Rule-based procedures select the target anchor (prioritizing nominative roots, subjects, and objects), extract conceptual modifiers, and isolate background context from spatial expressions. The output is a structured triplet with explicit fields for downstream refinement.

(Figure 3)

Figure 3: Dependency parsing module architecture, converting raw queries into structured triplets for targeted refinement.

Empirical Results

The proposed framework is evaluated across four benchmarks: TNLLT, TNL2K, LaSOT, and OTB99-Lang. Key results include:

• TNLLT: PR=75.0, NPR=78.2, SR=64.5, outperforming DUTrack Li2025dutrack and ReasoningTrack wang2025reasoningtrack. Competitors with higher NPR (e.g., SDTrack) exhibit inferior PR and SR, demonstrating robust absolute localization and overlap.
• TNL2K: PR=74.5, AUC=65.1, best PR among trackers (+3.9 over DUTrack), marginally below ATSTrack in AUC but optimized for precision.
• LaSOT: PR=80.3, NPR=83.1, AUC=71.7, competitive with DUTrack and ATSTrack, showing robust performance for long-term, diverse appearance scenarios.
• OTB99-Lang: PR=94.8, AUC=72.4, competitive AUC, improving over ReasoningTrack and DUTrack in overlap quality.

Qualitative visualizations highlight superior ability to maintain stable target anchoring, adaptive concept updating, and context-aware distractor suppression amid challenging conditions (occlusion, motion blur, scale variation). Attribute-based robustness analysis demonstrates consistent improvements across 12/15 challenge categories.

(Figure 4)

Figure 4: Visualizations of dependency parsing and triplet refinement, preserving stable target identity while reorganizing semantic cues for tracking.

(Figure 5)

Figure 5: Tracking results visualizations and radar charts illustrating attribute-level robustness.

Ablation and Limitations

Comprehensive ablation studies validate the contribution of each module:

• DP-based triplet parsing provides explicit language guidance, boosting PR/NPR/AUC.
• Qwen-VL refinement (LoRA rank-16) achieves optimal tracking-oriented triplet refinement; frozen Qwen decreases performance.
• Target-conditioned sparse Top-K update ($K=8$ recommended) balances relevance and noise, outperforming dense update.
• Concept-only updating yields better results than joint concept-background refinement; background updates introduce instability.

Limitations include computational cost for Qwen-based refinement relative to static queries. Deployment efficiency and long-term reasoning (target reappearance, unreliable evidence) warrant further research. Incorporating historical state, temporal cues, and lightweight module design are potential future directions.

Implications and Future Directions

The approach advances VLT by enforcing fine-grained, structured text evolution that aligns semantic guidance with dynamic visual states. This mitigates semantic-visual misalignment and hallucination artifacts inherent in previous LLM-based paradigms. Theoretical implications include demonstrating that explicit linguistic structure and modular updating provide a more controllable and robust interface for cross-modal tasks.

Practically, the framework supports deployment in real-world, long-term multimodal tracking systems, where adaptive semantic guidance is crucial. The methodology is extensible to settings involving referring multi-object tracking [li2025referMOT], open-vocabulary recognition, and context-aware fusion.

Speculatively, further advancements may involve integrating LLM-chain reasoning [wang2025reasoningtrack], memory-centric modules, and real-time scalable LLM refinement. This will enable more generalizable and efficient multimodal tracking models for unconstrained environments.

Conclusion

Dynamic parsing and fine-grained updating of natural language specifications via structured triplet representations and targeted vision-LLM refinement substantially enhance robustness in VLT. Empirical validation across diverse benchmarks demonstrates consistent improvement in precision, overlap, and attribute-level robustness. The findings underscore the importance of modular, adaptive language guidance that preserves target identity, eliminates distracting context, and aligns textual and visual semantics throughout the tracking process. This paradigm establishes a foundation for future research in structured, adaptive multimodal interaction and cross-modal tracking systems.