RAGTrack: Language-Aware RGBT Tracking

• RAGTrack is a language-aware RGBT tracking framework that integrates visual and textual data to robustly localize objects in varying environments.
• The framework employs unified visual-language modeling, dynamic fusion, and a retrieval-augmented generation module to mitigate appearance changes and modality gaps.
• Benchmark results demonstrate significant gains in precision and success rates over prior methods across multiple RGBT datasets, validating its effective temporal reasoning.

RAGTrack is a language-aware RGB-Thermal (RGBT) tracking framework that extends RGBT benchmarks with textual descriptions and combines unified visual-language modeling, dynamic fusion, and Retrieval-Augmented Generation (RAG) for robust target modeling. Introduced in "RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation" (Li et al., 4 Mar 2026), it addresses a specific deficiency in prior RGBT trackers: reliance on initial-frame visual information alone for target modeling, despite severe appearance variation, redundant search regions, heterogeneous modality gaps, and the practical inconvenience of bounding box-based initialization. In this formulation, language provides abstract object understanding through categories, attributes, and motion states, while Multi-modal LLMs (MLLMs) supply adaptive textual descriptions that can be updated over time rather than remaining static.

1. Problem setting and motivation

RGB-Thermal tracking aims to achieve robust object localization across diverse environmental conditions by fusing visible (RGB) and thermal infrared (TIR) modalities. TIR provides all-weather capability complementary to RGB, but existing RGBT trackers rely solely on initial-frame visual information for target modeling. According to the RAGTrack formulation, this creates several coupled failure modes: a single image template offers limited target information and cannot capture appearance changes across viewpoints; targets are inherently ambiguous, so trackers may overemphasize unstable features and become distracted by background or similar objects; redundant search regions contain substantial background and distractors; and heterogeneous modality gaps hinder effective RGB-TIR correspondence (Li et al., 4 Mar 2026).

RAGTrack frames language-aware RAG as a remedy to these limitations. Language is used as high-level semantic guidance that improves target-background separation and complements visual features. MLLMs are used not merely for one-off captioning, but for temporal linguistic reasoning: they generate adaptive textual descriptions as the target evolves, thereby mitigating misalignment between static texts and dynamic frames. The framework is presented as the first introduction of textual descriptions into RGBT tracking benchmarks, extending existing benchmarks with semantic annotations and then using those annotations for unified visual-language modeling and temporal reasoning.

2. Benchmark extension through textual annotation

A central precondition for RAGTrack is the creation of text-annotated RGBT data. The annotation pipeline is a two-stage generation-refinement process designed to produce textual descriptions aligned to RGBT frames (Li et al., 4 Mar 2026). In the first stage, an MLLM receives the image and bounding box and is prompted to: “Describe the object located in the image at <box> ({x, {y}, {x+w}, {y+h}) </box>. Focus on distinctive visual features, motion patterns, and key identifiers to distinguish it from background elements and distractors. Keep the description in a continuous sentence under 20 words. Avoid mentioning bounding boxes or coordinates. Do not use parentheses for explanations.” This stage is intended to scale linguistic annotations efficiently while emphasizing discriminative appearance and motion cues.

In the second stage, a second prompt is used for normalization and error correction: “Correcting the textual description of the tracking object. Ensure the final output is a continuous sentence under 20 words, logically coherent, does not mention bounding boxes, or coordinates terms, and does not use parentheses for explanations. Do not introduce new details. Output only the integrated description without any additional text. Textual description: [Initial description]”. Human experts then rectify residual hallucinations and errors, remove mixed-language or garbled text, and ensure accuracy. This produces one description per frame during training, while testing uses first-frame descriptions to assess textual reasoning over time.

The scope of the released annotations is explicit. For LasHeR training, comprising 979 sequences, RAGTrack provides 514,081 textual descriptions, one per frame. For testing, it provides first-frame descriptions for all sequences in the LasHeR test set and all sequences in GTOT, RGBT210, and RGBT234, totaling 739 descriptions. Dataset attributes are retained for analysis, including LasHeR’s 19 attributes, while temporal context is handled online through the Context-aware Reasoning Module (CRM), whose RAG cycle generates updated descriptions per frame.

3. Architectural organization

RAGTrack integrates three named components: the Multi-modal Transformer Encoder (MTE), the Adaptive Token Fusion (ATF), and the Context-aware Reasoning Module (CRM) (Li et al., 4 Mar 2026). The overall pipeline tokenizes texts and images with reasoning tokens, performs unified visual-language modeling, dynamically selects target-relevant tokens and exchanges channels, retrieves context from a dynamic knowledge base, performs temporal reasoning, and finally predicts bounding boxes. Updated textual descriptions generated by MLLMs refresh the multi-modal references.

The MTE operates on search images $X$ and template images $Z$ for RGB and TIR, together with a language description $L^t$. The encoder uses parameter-shared branches for both modalities. Templates and search images are transformed into patch tokens by three-stage downsampling, and text is encoded after concatenating a temporal sequence prefix with the current text. The prefix is formatted as “A sequence of a [$*$] object:”, where [$*$] is replaced by learnable tokens. Reasoning tokens, text tokens, template tokens, and search tokens are then concatenated into a unified token sequence and processed by multi-head self-attention and MLP blocks. This yields unified visual-language modeling in which text semantics are intended to enhance visual feature discrimination across both modalities.

ATF addresses two specific issues: search redundancy and modality gaps. Its token-selection stage treats self-attention scores as significance indicators and contextual guides. Search-to-reasoning, search-to-text, search-to-template, and search-to-search attention are aggregated into $\mathbf{A}_m^{\text{total}}$, and a parameter-free selection retains search tokens with higher attention values according to a retention ratio $\gamma$. To mitigate template noise, the template’s center region is used. Its second stage performs channel exchange across RGB and TIR on the basis of cross-modal channel relevance, selecting channels according to an exchange ratio $\sigma$ and then fusing them via a channel MLP. The stated purpose is to focus computation on informative tokens while adaptively aligning channels to maintain semantic consistency across modalities.

CRM implements the temporal RAG cycle. It is defined as a four-stage process: construction, retrieval, augmentation, and generation. In construction, the module maintains a local knowledge base of historical text features and adds a new feature only if its maximum cosine similarity with existing entries is below the threshold $\lambda$. In retrieval, the current text feature queries this knowledge base, top-$k$ relevant features are selected, and search features are refined with intra-modal cross-attention. In augmentation, pooled reasoning, text, and template features are combined by a guidance MLP, and temporal augmentation updates the reasoning token for the next frame. In generation, an MLLM dynamically produces a new target description during inference from the search image and predicted box using the prompt: “Describe the object located in the image at <box>({x,{y},{x+w},{y+h})</box>$$</code>. Focus on distinctive visual features, motion patterns, and key identifiers to distinguish it from background elements and distractors.” CRM therefore treats language not as a fixed auxiliary modality but as a continuously refreshed memory mechanism for temporal coherence.</p> <h2 class='paper-heading' id='training-objective-inference-procedure-and-implementation'>4. Training objective, inference procedure, and implementation</h2> <p>The prediction head outputs three maps: a classification score map$$Z$0, a spatial offset map $Z$1, and a normalized size map $Z$2 (Li et al., 4 Mar 2026). The final box is taken at the position of maximum classification score, with coordinates and size recovered from the corresponding offsets and normalized dimensions. Training uses a multi-task loss that combines classification focal loss with $Z$3 and generalized IoU regression losses, with loss weights $Z$4 and $Z$5. Bounding box labels provide classification, offset, and size targets, and data augmentation includes rotation, translation, and grayscale transformation.

Inference proceeds frame by frame. The first frame supplies the language description and template-search images for RGB and TIR. MTE produces multi-modal tokens; ATF computes attention scores, aggregates $Z$6, retains the top-$Z$7 search tokens, extracts the template center, computes cross-modal channel relevance, exchanges $Z$8 of channels, and fuses RGB-TIR features through an MLP. CRM retrieves top-$Z$9 historical text features, refines search tokens via cross-attention, updates reasoning tokens, performs temporal augmentation, and passes the result to the prediction head. An MLLM then generates an updated description for the current frame using the predicted box; if the cosine-similarity threshold permits, the knowledge base is updated. Multi-modal references are updated during inference with threshold 0.65 and interval 5 frames.

The implementation is specified in detail. The framework is implemented in PyTorch and trained on 4×NVIDIA V100 GPUs with batch size 16. The text encoder is CLIP, the visual backbone is HiViT-B initialized from SOT, and the MLLM is QWen2.5-VL-3B. Template and search resolutions are 128×128 and 256×256, respectively. The feature dimension is $L^t$0, the number of text tokens is $L^t$1, the number of reasoning tokens is $L^t$2, and the learnable token length in the sequence prefix is 2. ATF is deployed at HiViT-B layers 6, 12, 18, and 24; the retention ratio is $L^t$3; channel exchange selects 256 channels per modality; and the exchange ratio is $L^t$4. CRM uses knowledge base size $L^t$5, top-$L^t$6 retrieval with $L^t$7, and similarity threshold $L^t$8. Optimization uses AdamW with learning rate $L^t$9 and weight decay $*$0. The reported speed is 24.3 FPS on V100, with computational cost 62.7G FLOPs. The source code is available at https://github.com/IdolLab/RAGTrack.

5. Benchmarks, quantitative results, and ablation evidence

RAGTrack is evaluated on four RGBT benchmarks: GTOT, RGBT210, RGBT234, and LasHeR (Li et al., 4 Mar 2026). The metrics vary by benchmark: GTOT and RGBT234 use Maximum Precision Rate (MPR) and Maximum Success Rate (MSR); RGBT210 uses Precision Rate (PR) and Success Rate (SR); and LasHeR uses PR, Normalized Precision Rate (NPR), and SR.

Benchmark	Metrics	Reported result
GTOT	MPR, MSR	95.1%, 79.3%
RGBT210	PR, SR	93.2%, 67.1%
RGBT234	MPR, MSR	93.8%, 69.5%
LasHeR	PR, NPR, SR	76.8%, 73.0%, 61.1%

The reported gains over prior methods are explicit. On GTOT, RAGTrack attains MPR 95.1% and MSR 79.3%, corresponding to +1.5% MPR over MoETrack and +4.0% MSR over MambaVT. On RGBT210, it reports PR 93.2% and SR 67.1%, with +2.8% PR over AETrack and +2.3% SR over AINet. On RGBT234, it reaches MPR 93.8% and MSR 69.5%, with +6.9% MPR over SMSTracker and +2.8% MSR over STTrack. On LasHeR, it reports PR 76.8%, NPR 73.0%, and SR 61.1%, with +4.2% PR over TVTracker and +5.4% SR over XTrack. Attribute-based analysis on LasHeR shows notable gains in Total Occlusion, where PR increases by 10.7%, and in Out-of-View, where SR increases by 5.5%; the paper interprets these gains as evidence of CRM’s temporal identity maintenance under appearance variation.

The ablation study on RGBT234 decomposes the framework into its major components. The baseline achieves 87.9 MPR and 64.5 MSR. Adding text-free CRM yields 89.1 and 65.0; adding MTE with CRM* yields 91.1 and 66.7; replacing CRM* with the full language-enabled CRM yields 91.8 and 67.4; and the full MTE+CRM+ATF configuration reaches 93.8 and 69.5. This progression is used to argue that unified visual-language modeling, context-aware reasoning, and dynamic fusion are complementary and essential.

Additional ablations isolate several design choices. ATF performs best when placed at layers 6, 12, 18, and 24, giving 93.8% MPR and 69.5% MSR. Compared with TBSI, BSI, and DFM, ATF attains the best accuracy with fewer parameters: 93.8 MPR, 69.5 MSR, and 101.8M parameters, versus 92.8/67.6/145.9M for TBSI, 93.1/68.2/103.6M for BSI, and 92.7/67.8/110.3M for DFM. In CRM augmentation, the proposed mechanism outperforms Add, Transformer, and Mamba while retaining 101.8M parameters. The token study reports that $*$1 is best and that a learnable prefix length of 2 is best; more tokens introduce redundancy. Hyperparameter studies identify $*$2, $*$3, knowledge base size $*$4, and retrieval count $*$5 as the best settings. Robustness to missing text is also reported: even with 100% missing text, RAGTrack achieves 92.9% MPR and 68.8% MSR, surpassing recent strong baselines.

6. Relation to adjacent work, terminological ambiguity, and limitations

Within RGBT tracking, RAGTrack is positioned against trackers such as TBSI, BAT, SDSTrack, OneTracker, AINet, and STTrack, which fuse RGB and TIR features but rely on the initial-frame template and lack language guidance, treat redundant search regions at the token level in ways that increase background distraction, and do not fully resolve heterogeneous modality gaps (Li et al., 4 Mar 2026). It is also contrasted with RGBL vision-language trackers, which integrate text but often suffer from visual-language misalignment and lack explicit mechanisms for suppressing background and fusing modalities efficiently. RAGTrack differs by introducing text annotations into RGBT benchmarks, by using unified visual-language modeling with a temporal sequence prefix and reasoning tokens, by selecting target-relevant tokens through text-guided attention while performing adaptive channel exchange, and by maintaining a dynamic knowledge base that supports temporal reasoning and updated descriptions.

The term “RAGTrack” is not unique to this RGBT tracker. In the literature on robust retrieval-augmented generation, the RADAR paper uses “RAGTrack” to denote a capability for tracking and safeguarding RAG systems under dynamic corpora and volatile retrieval, with dashboards, belief trajectories, cut-capacity trends, and alerting protocols derived from graph-based selection and Bayesian memory (Chen et al., 21 May 2026). In TREC evaluation literature, “RAGTrack” and “RAG Track” are also used to refer to the RAGTIME track, the “RAG TREC Instrument for Multilingual Evaluation,” which studies long-form report generation from multilingual source documents and evaluates nugget coverage, sentence support, and combined F1, alongside MLIR metrics such as MAP, nDCG@20, Recall@1000, and Judged@20 (Lawrie et al., 10 Feb 2026). These usages are terminologically related through retrieval-augmented generation, but they designate distinct research objects: an RGBT tracker, a dynamic-defense monitoring capability for RAG systems, and a TREC evaluation track for multilingual report generation.

The limitations explicitly or implicitly associated with the RGBT RAGTrack framework are narrower. The paper states that explicit failure cases are not enumerated. Constraints implied by the setup include dependency on the quality of generated and curated textual descriptions, additional computational overhead from MLLM-based generation during inference, and dataset-level modality annotation misalignment that can complicate evaluation. Potential future directions suggested by the framework include expanding language generation to richer context such as attributes and motion states, optimizing MLLM usage for speed, and exploring more advanced cross-modal alignment mechanisms.