PATrack: Progressive Multi-Modal Tracking

• The paper introduces a progressive adaptation strategy that mitigates stacked mismatches in intra-modal, inter-modal, and head levels via lightweight adapter modules.
• It employs Modality-Dependent Adaptation to enhance individual modality features, Cross-Modality Entangled Adaptation to fuse shared cues, and Head Adaptation to align with RGB-pretrained predictions.
• PATrack achieves state-of-the-art or competitive results across benchmarks like LasHeR, RGBT234, and VisEvent while maintaining parameter efficiency.

Progressive Adaptation for Multi-Modal Tracking (PATrack) is a parameter-efficient framework for transferring an RGB-pretrained tracker to RGB+X single-object tracking, where $X$ is thermal, depth, or event. Its central premise is that adapting an RGB foundation tracker to multi-modal tracking is not a single adaptation problem but a stacked mismatch across three levels: within each modality, between modalities, and at the prediction head. PATrack addresses these mismatches through a progressive strategy composed of Modality-Dependent Adaptation (MDA), Cross-modality Entangled Adaptation (CEA), and Head Adaptation (HA), while keeping the RGB-pretrained backbone largely intact and inserting lightweight adapters (Wang et al., 22 Mar 2026).

1. Problem setting and conceptual motivation

PATrack studies multi-modal single-object tracking in the RGB+X setting, specifically RGB+Thermal, RGB+Depth, and RGB+Event. The tracker receives template frames $(T_{\text{RGB}}, T_X)$, search frames $(S_{\text{RGB}}, S_X)$, and the initial target box $B_0$, and predicts the target box in the search frame (Wang et al., 22 Mar 2026).

The motivating claim is that adapting RGB-pretrained trackers to multi-modal tracking is difficult because RGB-pretrained representations are biased toward RGB image statistics, whereas thermal, depth, and event inputs differ sharply from RGB. The paper characterizes this as a stacked mismatch. Within each modality, RGB and $X$ have very different statistics and information density; between modalities, the tracker must exploit complementary and shared cross-modal cues reliably; at the prediction head, a head pretrained on RGB-only features has a strong inductive bias that is not naturally suited to fused multi-modal features (Wang et al., 22 Mar 2026).

This framing places PATrack in a research trajectory that includes prompt-based and adapter-based transfer from RGB-pretrained trackers. ViPT adapts a frozen RGB transformer tracker with modal-relevant prompts and learns only a few trainable parameters, while BAT uses a frozen foundation model plus a universal bi-directional adapter for cross-prompting RGB and thermal branches (Zhu et al., 2023, Cao et al., 2023). PATrack differs by explicitly organizing adaptation as a progression from intra-modal enhancement, to inter-modal entanglement, to task-level decoding alignment (Wang et al., 22 Mar 2026).

2. Progressive architecture and insertion strategy

PATrack is built on the OSTrack RGB tracker as the foundation model. For RGB-only tracking, OSTrack predicts the box as

$$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$

For RGB+X tracking, PATrack extends this to

$$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$

The framework uses a dual-branch architecture, one branch for RGB and one branch for the auxiliary modality $X$. The two branches are processed in parallel, and the paper states that parameters are shared between the RGB and X branches at the backbone level for efficiency (Wang et al., 22 Mar 2026).

After patch embedding, the model produces initial tokens

$H_{\text{RGB}}^0,\quad H_X^0.$

These tokens are fed layer by layer through a dual-stream Transformer with inserted adapters: $$(H^{l}_\text{RGB}, H^{l}_\text{X}) = F_A^l(H^{l-1}_\text{RGB}, H^{l-1}_\text{X}), \quad l=1,\dots,N.$$

The progression is architectural rather than curricular. Most layers use MDA to refine modality-specific features, selected layers use CEA to inject cross-modal shared information, and the final fused search tokens are passed through HA before prediction. The adapter placement is explicit. MDA is inserted at layers

$(T_{\text{RGB}}, T_X)$0

while CEA is inserted at

$(T_{\text{RGB}}, T_X)$1

Operationally, progressive adaptation means that modality-specific enhancement dominates most layers, cross-modal entanglement appears at representative shallow, middle, and deep layers, and the head is adapted only after feature fusion (Wang et al., 22 Mar 2026).

For layers using adapter insertion, the paper gives residual update forms. For MDA-type layers: $(T_{\text{RGB}}, T_X)$2

$(T_{\text{RGB}}, T_X)$3

For CEA-type layers: $(T_{\text{RGB}}, T_X)$4

$(T_{\text{RGB}}, T_X)$5

The paper presents the implementation on OSTrack, but a plausible implication is that the method targets RGB-pretrained Transformer trackers with MSA/MLP residual blocks and a box head (Wang et al., 22 Mar 2026).

3. Modality-Dependent Adaptation (MDA)

Modality-Dependent Adaptation is the intra-modal stage of PATrack. Its purpose is to enhance single-modality representations by decomposing and processing high- and low-frequency information. The paper’s argument is that a modality-specific token contains both high-frequency information, such as edges and target boundaries, and low-frequency information, such as smoother global shape and coarse semantic context; because different modalities degrade differently, explicitly decomposing and enhancing these components improves robustness (Wang et al., 22 Mar 2026).

Each modality is processed independently by its own MDA. Features are first compressed from channel dimension $(T_{\text{RGB}}, T_X)$6 to $(T_{\text{RGB}}, T_X)$7, with

$(T_{\text{RGB}}, T_X)$8

For modality $(T_{\text{RGB}}, T_X)$9, the reduced feature $(S_{\text{RGB}}, S_X)$0 is split channel-wise into

$(S_{\text{RGB}}, S_X)$1

with

$(S_{\text{RGB}}, S_X)$2

The high-frequency branch is then split again into

$(S_{\text{RGB}}, S_X)$3

with

$(S_{\text{RGB}}, S_X)$4

The max-pooling branch is

$(S_{\text{RGB}}, S_X)$5

where $(S_{\text{RGB}}, S_X)$6 is $(S_{\text{RGB}}, S_X)$7, stride $(S_{\text{RGB}}, S_X)$8, padding $(S_{\text{RGB}}, S_X)$9, and $B_0$0 is implemented as a $B_0$1 convolution. The depthwise-convolution branch is

$B_0$2

where $B_0$3 is a $B_0$4 convolution and $B_0$5 is depthwise convolution with kernel size $B_0$6, stride $B_0$7, padding $B_0$8. The low-frequency branch is

$B_0$9

The final MDA output is

$X$0

Although MDA processes one modality at a time, its output is then transferred to the other branch through the residual adapter insertion equations. In that sense, MDA is a modality-specific enhancement module whose output supports the opposite modality branch (Wang et al., 22 Mar 2026).

The ablation evidence identifies MDA as the largest single contributor among PATrack’s three adapters. From Table 7, the baseline has LasHeR SR/PR/NPR $X$1, VisEvent SR/PR $X$2, and DepthTrack F-score $X$3. Adding MDA only raises LasHeR to $X$4, VisEvent to $X$5, and DepthTrack to $X$6 (Wang et al., 22 Mar 2026).

4. Cross-modality Entangled Adaptation (CEA)

Cross-modality Entangled Adaptation is the inter-modal stage of PATrack. If MDA handles modality-specific information, CEA handles shared cross-modal information. The paper argues that shared information across modalities is crucial because it improves generalization and helps the tracker focus on target-consistent cues across RGB and $X$7 (Wang et al., 22 Mar 2026).

Given RGB and $X$8 features $X$9 and $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$0, CEA first reduces dimensionality: $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$1 The reduced features are concatenated and transformed to form a fusion-guided shared signal: $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$2 RGB is then updated by attending to $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$3, guided by this shared fused query: $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$4 and $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$5 is updated symmetrically: $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$6 The outputs are projected back: $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$7

The paper does not spell out the inner $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$8 operator in expanded softmax form, but it explicitly defines the argument-level structure above. This suggests that the distinctive property of CEA is not merely cross-attention, but cross-attention guided by inter-modal shared information rather than by a single branch alone (Wang et al., 22 Mar 2026).

CEA is inserted at layers $$B_{box} = \mathrm{Head}(F(S_{\text{RGB}}, T_{\text{RGB}}, B_o)).$$9 and $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$0. The paper reports that this uneven placement was better than frequent even placement. The measured gains are modest but consistent. On LasHeR, removing CEA yields SR $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$1, PR $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$2, NPR $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$3, while inserting CEA at $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$4 yields SR $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$5, PR $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$6, NPR $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$7. Component ablations show full CEA performs better than versions without the fusion-guided query, without convolutional projections, or without the skip connection (Wang et al., 22 Mar 2026).

This inter-modal design can be situated among related multi-modal trackers that emphasize adaptive cross-modal interaction. APTrack uses an Adaptive Modality Interaction module in a unified RGB-X tracker, while DMTrack uses a Progressive Modality Complementary Adapter with shallow and deep stages for progressive cross-modality prompting (Hu et al., 10 Feb 2025, Li et al., 3 Aug 2025). PATrack’s contribution is to make inter-modal entanglement one stage in a larger progressive hierarchy that also includes explicit intra-modal enhancement and head adaptation (Wang et al., 22 Mar 2026).

5. Head Adaptation, training, and implementation

PATrack’s third stage is Head Adaptation. The motivation is that even if feature-level adaptation succeeds, the prediction head still has a strong RGB-specific inductive bias. Fused multi-modal search tokens have different statistics and may contain modality fusion artifacts or background noise, so the tracker adapts the decoder interface rather than leaving the RGB-trained head untouched (Wang et al., 22 Mar 2026).

After the backbone produces fused multi-modal template and search tokens, the search tokens $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$8 are passed through a lightweight bottleneck MLP: $$B_{box} = \mathrm{Head}(F(T_\text{RGB},T_\text{X},S_\text{RGB},S_\text{X},B_{0})).$$9 The adapted search tokens are then fed to the box head. In the paper’s interpretation, HA reshapes fused search features into a form the RGB-trained head can better interpret, filters noise and background from fused multi-modal representations, and bridges the remaining mismatch between pretrained RGB decoding bias and multi-modal token distributions (Wang et al., 22 Mar 2026).

PATrack is described as a parameter-efficient fine-tuning approach initialized from RGB-pretrained OSTrack. The text frames the method as avoiding full fine-tuning and learning mainly through the inserted adapters. The implementation details reported are concise:

Item	Value
Backbone	OSTrack
Optimizer	AdamW
Weight decay	$X$0
Initial LR	$X$1
LR decay ratio	$X$2
Embedding dimension	$X$3
MDA bottleneck	$X$4

The framework supports RGB-T, RGB-D, and RGB-E, but each task is trained independently, with its own network parameters used at inference. The paper does not provide an explicit new PATrack-specific loss formula and states that training inherits the tracking objective from the base tracker setup (Wang et al., 22 Mar 2026).

This design occupies a distinct position relative to other PEFT trackers. ViPT freezes the RGB tracker and learns modal-specific prompts; BAT freezes the transformer encoders and prediction head and optimizes only a bi-directional adapter; SEATrack prioritizes cross-modal alignment of matching responses with AMG-LoRA before HMoE fusion (Zhu et al., 2023, Cao et al., 2023, Su et al., 14 Apr 2026). PATrack differs by treating task-level head adaptation as a first-class mismatch rather than limiting adaptation to the feature extractor (Wang et al., 22 Mar 2026).

6. Benchmarks, results, and significance

PATrack is evaluated on RGB+Thermal, RGB+Depth, and RGB+Event tracking tasks. The paper reports results on LasHeR, RGBT234, GTOT, DepthTrack, and VisEvent (Wang et al., 22 Mar 2026).

On LasHeR, PATrack achieves

$X$5

which is the best result in Table 1. On RGBT234, it achieves

$X$6

which is best in SR and tied-best in PR. On GTOT, it achieves

$X$7

second to GMMT. On DepthTrack, it achieves

$X$8

which is second-best behind SDSTrack. On VisEvent, it achieves

$X$9

which is the best result in Table 1 (Wang et al., 22 Mar 2026).

The paper also reports adapter-efficiency statistics. The baseline has 92.13M parameters and 56.44G FLOPs. The full PATrack configuration with MDA, CEA, and HA has 95.85M parameters and 58.32G FLOPs. The adapter-dimension study reports three settings on LasHeR: Our-tiny with 93.64M parameters, 57.45G FLOPs, and 16.95 FPS; Our-base with 95.51M parameters, 58.24G FLOPs, and 16.40 FPS; and Our-large with 102.1M parameters, 62.45G FLOPs, and 15.95 FPS. The paper identifies the base setting as the best tradeoff (Wang et al., 22 Mar 2026).

A concise summary of the principal benchmark results is as follows.

Dataset	PATrack result	Position reported
LasHeR	$H_{\text{RGB}}^0,\quad H_X^0.$0	Best
RGBT234	$H_{\text{RGB}}^0,\quad H_X^0.$1	Best SR, tied-best PR
GTOT	$H_{\text{RGB}}^0,\quad H_X^0.$2	Second
DepthTrack	$H_{\text{RGB}}^0,\quad H_X^0.$3	Second-best
VisEvent	$H_{\text{RGB}}^0,\quad H_X^0.$4	Best

The ablation evidence supports the progressive design. Baseline, MDA only, CEA only, HA only, and full MDA+CEA+HA show that the three modules are complementary, with MDA contributing the largest single improvement and the full combination performing best. The paper interprets this as evidence that modality-dependent and modality-entangled adaptation solve different but complementary problems: MDA provides good local ingredients, while CEA provides reliable cross-modal coordination, and HA aligns the fused representation with the RGB-pretrained prediction head (Wang et al., 22 Mar 2026).

Relative to adjacent work, PATrack occupies a middle ground between prompt learning, adapter tuning, and adaptive multimodal perception. ViPT offers stage-wise prompt injection into a frozen RGB tracker, BAT provides layer-wise mutual prompting with only 0.32M trainable parameters, APTrack uses equal modeling and adaptive modality interaction in a unified RGB-X model, and FlexTrack addresses temporally incomplete multimodal data with Heterogeneous Mixture-of-Experts fusion and adaptive complexity (Zhu et al., 2023, Cao et al., 2023, Hu et al., 10 Feb 2025, Tan et al., 8 Jul 2025). PATrack’s distinctive contribution is to organize adaptation explicitly as a progression from intra-modal enhancement, to inter-modal entanglement, to task-level decoding alignment (Wang et al., 22 Mar 2026).