RSTrack: Supervised Visual Tracking

• RSTrack is a single-object visual tracker that explicitly models temporal evolution via supervised target-state reasoning to enhance tracking accuracy.
• It compresses past frame features into compact spatial and channel states and integrates them with current frame data through a dual-stage temporal decoder.
• The approach addresses issues like contextual drift and weak temporal consistency while maintaining real-time speeds and state-of-the-art benchmark results.

Searching arXiv for the official “RSTrack” paper and closely related trackers with similar names. RSTrack is a single-object visual tracker introduced in "Explicit Context Reasoning with Supervision for Visual Tracking" (Zeng et al., 22 Jul 2025). It frames temporal modeling not as implicit history stacking, but as an explicitly supervised target-state reasoning problem: each past frame contributes compact target states, a temporal reasoning module predicts the current state from that history, a reconstruction module restores a richer target feature, and a temporal decoder integrates that feature with the current search frame for localization. The method is designed to address contextual association divergence, weak temporal consistency, and the lack of explicit supervision in historical context association, while maintaining real-time running speeds and state-of-the-art benchmark performance (Zeng et al., 22 Jul 2025).

1. Problem setting and conceptual motivation

RSTrack addresses the standard single-object tracking setting: given the target box in the first frame, the tracker must localize the same object in subsequent frames despite appearance changes, occlusion, distractors, scale changes, and motion (Zeng et al., 22 Jul 2025). Its starting point is a critique of mainstream temporal/context modeling in modern trackers. Many recent methods use historical features, prompts, or state tokens from previous frames, but they typically associate context by stacking historical information or propagating tokens without explicitly supervising whether the temporal association corresponds to the true evolving target state.

The paper identifies three linked deficiencies. First, contextual association divergence arises because the latent temporal association process can drift toward background or distractor information when the model is supervised only by the final tracking objective. Second, weak temporal consistency follows from simple history aggregation, which does not force the model to learn an evolution rule from past target states to the current one. Third, there is a lack of explicit supervision in historical context association, because prior methods generally supervise only final tracking outputs rather than the latent temporal reasoning itself. RSTrack is proposed as a remedy by turning temporal modeling into an explicit target-state prediction problem supervised by the true target feature of the current frame (Zeng et al., 22 Jul 2025).

A central implication is methodological rather than merely architectural. Instead of assuming that a tracker will infer temporal dynamics from concatenated history, RSTrack constructs a pipeline that compresses each frame’s target appearance into a compact state, models the evolution of those states over time, predicts the current state, reconstructs a current target feature, and then directly constrains that predicted state and feature with supervision from the true target in the current frame. This suggests a shift from implicit temporal aggregation to supervised temporal state estimation.

2. Architecture and target-state reasoning pipeline

RSTrack has five major components: a visual encoder, state compression module, state reasoning module, reconstruction module, and temporal decoder with prediction head (Zeng et al., 22 Jul 2025). At time $t$, the tracker uses the initial template frame $\mathcal{I}_z$ and the current search frame $\mathcal{I}_x$. The backbone is Fast-iTPN-B224. Template and search patches are embedded and concatenated as

$$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$

with

$$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$

They are then processed through $L$ encoder layers,

$$\mathcal{F}_{v}^{l} = \mathbb{E}^l(\mathcal{F}_{v}^{l-1}),\quad l=1,2,\dots,L,$$

yielding template feature $\mathcal{F}_z$ and current search feature $\mathcal{F}_x$.

The temporal memory is not a pool of raw historical tokens. Instead, each frame contributes a pair of compact target states, a spatial state $\mathcal{S}^t$ and a channel state $\mathcal{I}_z$0, stored in a historical sequence

$\mathcal{I}_z$1

Given this history, RSTrack appends an initialization token pair $\mathcal{I}_z$2 and predicts the current state through a Mamba-based reasoning module:

$\mathcal{I}_z$3

Here $\mathcal{I}_z$4 contains two Mamba branches, $\mathcal{I}_z$5 for spatial states and $\mathcal{I}_z$6 for channel states. The current target state is therefore reasoned from temporal evolution, not copied from the last frame.

The predicted compact state is then transformed into a richer target feature $\mathcal{I}_z$7. The paper describes reconstruction as summing $\mathcal{I}_z$8 and $\mathcal{I}_z$9, broadcasting them to match the shape of $\mathcal{I}_x$0, concatenating with $\mathcal{I}_x$1 along channels, and processing the result with multi-scale convolution and activations. This reconstructed feature is fused with the current search feature in a dual-stage cross-attention temporal decoder. In the first cross-attention stage, $\mathcal{I}_x$2 acts as query and attends to the template and predicted features; in the second stage, the resulting joint query further enhances target representation in the search feature. A standard tracking head then predicts the bounding box using a classification branch for center probability and two regression branches for width and height (Zeng et al., 22 Jul 2025).

3. Compression, supervision, and objective design

The paper’s most distinctive technical contribution is not only target-state reasoning, but the explicit supervision applied to that reasoning process (Zeng et al., 22 Jul 2025). The target feature of frame $\mathcal{I}_x$3, denoted $\mathcal{I}_x$4, is compressed into a channel state $\mathcal{I}_x$5 and a spatial state $\mathcal{I}_x$6. The channel branch is defined as

$\mathcal{I}_x$7

where $\mathcal{I}_x$8 is $\mathcal{I}_x$9 average pooling, $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$0 is Group Normalization with group size 1, $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$1 are $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$2 depthwise separable convolutions, $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$3 is global average pooling, and $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$4 is Sigmoid. The attention is computed along the channel dimension, with $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$5.

The spatial branch decomposes width and height information:

$$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$6

Here $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$7 denotes global pooling along width or height, $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$8 depthwise separable 1D convolution, and $$\mathcal{F}_{v}^{0} = [\mathcal{P}_{z};\mathcal{P}_{x}] \in \mathbb{R}^{D \times (N_z+N_x)},$$9 GroupNorm with group size 4. Conceptually, $$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$0 encodes spatial structure and $$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$1 channel semantics.

The supervision path uses true target features as anchors. During training, the tracker extracts the actual target crop from the current search region using the ground-truth box, aligns it to the template size, and obtains $$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$2. This feature is compressed into true current target states $$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$3, which supervise the predicted states. The backbone used to obtain these supervisory features is frozen in this path. State-level supervision is

$$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$4

and feature-level reconstruction supervision is

$$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$5

These combine into the temporal supervision loss

$$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$6

The full objective is

$$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$7

with

$$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$8

The paper’s interpretive claim is precise: standard tracking supervision acts on final outputs, whereas RSTrack adds supervision inside the temporal reasoning process. Its main mechanism for suppressing drift is therefore latent state supervision rather than only box-level correction.

4. Training protocol and online inference

RSTrack is implemented in Python 3.9 and PyTorch 1.13.1, and training/testing are conducted on two NVIDIA A100 (Zeng et al., 22 Jul 2025). Training uses GOT-10k, LaSOT, TrackingNet, and COCO. The sampling strategy uses 4 search images per video, 15,000 sequences per epoch, and 60,000 search images per epoch. The optimizer is AdamW. Initial learning rates are $$N_z = \frac{H_z W_z}{16^2}, \qquad N_x = \frac{H_x W_x}{16^2}, \qquad D=512.$$9 for the backbone and $L$0 for other modules. The standard schedule runs for 150 epochs, with decay beginning at epoch 120 and decay factor $L$1. For GOT-10k-only training, the schedule is 100 epochs with decay from epoch 80. Batch size is 8 for RSTrack-256 and 6 for RSTrack-384. The temporal architecture uses 3 Mamba layers in $L$2, 3 Mamba layers in $L$3, and a 3-layer temporal decoder.

A subtle training detail is that the historical state sequence is maintained using feature compression tokens only, not recursively using the model’s own predicted states. This prevents error accumulation during training and makes current-step temporal reasoning cleaner. At inference, by contrast, the historical sequence is maintained using a combination of state inference and feature compression. The paper states that using only inferred states can introduce bias, whereas using only feature compression is more computationally expensive, so the two update modes are alternated. The maximum classification score is used as a confidence signal; when confidence is low, the tracker replaces the current state with the initial value to avoid injecting incorrect states into memory, using a threshold of 0.4. The paper also introduces a reverse reasoning process from current to initial moment within a specified interval of 60, and the preferred historical sampling strategy is window sampling with window size 500 (Zeng et al., 22 Jul 2025).

The efficiency profile is explicit. For RSTrack at $L$4, the paper reports 76M parameters, 57G FLOPs, and 34.7 FPS on Tesla V100. For comparison, SeqTrack is listed with 89M parameters, 148G FLOPs, and 17.8 FPS. The paper therefore characterizes RSTrack as maintaining real-time speed while improving performance (Zeng et al., 22 Jul 2025).

5. Benchmark performance and ablation evidence

RSTrack is evaluated on LaSOT, LaSOT$L$5, GOT-10k, TrackingNet, TNL2K, and UAV123 (Zeng et al., 22 Jul 2025). The reported benchmark results are:

Benchmark	RSTrack-256	RSTrack-384
LaSOT	73.4 / 84.1 / 81.7	74.4 / 84.2 / 83.0
LaSOT$L$6	52.7 / 63.8 / 60.3	53.9 / 65.5 / 61.8
GOT-10k	76.6 / 86.4 / 76.9	78.1 / 87.1 / 78.9
TrackingNet	85.3 / 89.9 / 84.7	85.8 / 90.3 / 85.7

For LaSOT, the three metrics are AUC / P$L$7 / P. For GOT-10k they are AO / SR$L$8 / SR$L$9. On UAV123 and TNL2K, RSTrack-256 reaches UAV123 AUC 70.5 and TNL2K AUC 60.5. The paper states that the method improves particularly in difficult scenarios including fast motion, deformation, full occlusion, small fast-moving objects, and distractor-rich scenes.

The ablation studies are unusually central to the method’s interpretation. The full RSTrack-256 on LaSOT achieves 73.4 / 84.1 / 81.7. Removing all context reasoning components and retaining only the visual encoder and prediction head reduces performance to 71.1 / 81.2 / 78.2. Removing the state reasoning module while keeping the temporal decoder yields 71.8 / 82.3 / 79.8, and removing the temporal decoder in favor of cross-correlation yields 72.0 / 82.3 / 79.8. These results indicate that both target-state reasoning and decoder integration contribute materially.

The forward supervision claim is supported directly by an ablation on $$\mathcal{F}_{v}^{l} = \mathbb{E}^l(\mathcal{F}_{v}^{l-1}),\quad l=1,2,\dots,L,$$0. Removing temporal supervision lowers LaSOT performance from 73.4 / 84.1 / 81.7 to 72.4 / 82.6 / 80.2, corresponding to about +1.0 AUC for explicit reasoning supervision. Efficient state modeling is also ablated. Replacing compressed state modeling with aggregated transfer tokens gives 72.2 / 82.4 / 79.6, and replacing the proposed spatial-channel compression module with CBAM gives 72.8 / 83.1 / 80.6. Replacing Mamba with ViT plus a temporal triangular mask gives 71.9 / 82.2 / 79.3. Replacing window sampling with global sampling yields 72.9 / 83.5 / 81.2, and replacing bidirectional reasoning with unidirectional reasoning yields 73.0 / 83.5 / 81.0 (Zeng et al., 22 Jul 2025).

These ablations support the paper’s main interpretive thesis: the reported gains are not attributed to a single component in isolation, but to the combination of compressed state modeling, Mamba-based temporal evolution, temporal decoder integration, and explicit forward supervision.

6. Relation to prior tracking methods and nomenclature

RSTrack is positioned against several prior lines of work in visual tracking (Zeng et al., 22 Jul 2025). Relative to standard transformer trackers such as OSTrack, it adds frame-wise target state extraction, temporal state memory, state-space reasoning, and direct latent state supervision. Relative to context propagation methods such as ODTrack and EVPTrack, it stores a dedicated state-token pair per frame rather than relying on relatively unified or lightweight context tokens. Relative to other temporal reasoning trackers, its strongest novelty claim is the combination of explicit target-state reasoning, forward supervision using true current target states as anchors, and compression-reconstruction for efficient and semantically focused temporal modeling. The paper presents this combination as a response to contextual association divergence.

The name “RSTrack” also requires disambiguation. It refers here to the 2025 single-object visual tracker introduced in "Explicit Context Reasoning with Supervision for Visual Tracking" (Zeng et al., 22 Jul 2025). It is distinct from TR-MOT, whose official name is not “RSTrack” even though its central contribution is a Reference Search (RS) module for online multi-object tracking (Chen et al., 2022). It is also distinct from two unrelated methods named RTrack: "RTrack: Accelerating Convergence for Visual Object Tracking via Pseudo-Boxes Exploration," a single-object tracker based on TrackPoints and pseudo bounding boxes (Zeng et al., 2023), and the RGB-thermal referring multi-object tracking framework RTrack for RT-RMOT (Yu et al., 25 Feb 2026). A plausible implication is that the literature contains genuine naming ambiguity, but the official method name RSTrack belongs to the explicitly supervised visual tracking approach of (Zeng et al., 22 Jul 2025).

The paper does not provide a dedicated limitations section. Likely limitations include dependence on memory quality at inference, a heuristic inference policy based on alternating update modes and thresholding, extra architectural complexity relative to simpler one-stream trackers, possible sensitivity to the confidence threshold $$\mathcal{F}_{v}^{l} = \mathbb{E}^l(\mathcal{F}_{v}^{l-1}),\quad l=1,2,\dots,L,$$1, interval $$\mathcal{F}_{v}^{l} = \mathbb{E}^l(\mathcal{F}_{v}^{l-1}),\quad l=1,2,\dots,L,$$2, and window size $$\mathcal{F}_{v}^{l} = \mathbb{E}^l(\mathcal{F}_{v}^{l-1}),\quad l=1,2,\dots,L,$$3, and a training/inference mismatch because training uses compressed true tokens only whereas inference uses predicted or inferred states in the historical sequence. These are reasonable inferences rather than explicit claims from the authors. Even with those caveats, the method’s contribution is conceptually clear: RSTrack treats temporal consistency as a supervised latent-state prediction problem rather than as an emergent by-product of context aggregation.