STC-Encoder: Visual Object Tracking

• The paper introduces a trainable module that integrates spatial cross-correlation and temporal sequence modeling to condition a frozen Transformer backbone for accurate tracking.
• It employs a siamese convolutional subnetwork and an LSTM-based temporal model to capture local appearance correlations and recent motion history efficiently.
• The approach achieves state-of-the-art performance while preserving pre-trained semantic representations and significantly reducing training costs.

The Spatio-Temporal Condition Encoder (STC-encoder) is a trainable module for visual object tracking, designed to inject explicit spatio-temporal cues into the outputs of a frozen pre-trained Transformer backbone. In the ACTrack framework, the STC-encoder incorporates two complementary submodules—a siamese convolutional spatial processor and a temporal sequence model—whose outputs are combined additively as a conditioning term on frozen Transformer features. This approach enables efficient exploitation of both local appearance correlations and recent motion history, balancing state-of-the-art tracking accuracy with reduced training overhead by preserving pre-trained backbone capabilities and restricting finetuning to a lightweight parameter set (Han et al., 2024).

1. Architectural Integration and Workflow

The STC-encoder operates by interfacing with a frozen Transformer-based feature extractor (e.g., ViT). Its full pipeline for each frame $t$ consists of the following stages:

• Feature extraction: The target’s initial template image $Z$ and the search crop $X_t$ for frame $t$ are processed through the frozen backbone to produce patch-token feature maps $F^Z \in \mathbb{R}^{D\times H_Z \times W_Z}$ and $F^X_t \in \mathbb{R}^{D\times H_X \times W_X}$.
• Condition encoding: STC-encoder takes $F^Z$, $F^X_t$, and (optionally) the list of $K-1$ past predicted bounding boxes $B_{t-K+1:t-1}$ to generate a spatio-temporal condition tensor $\Delta F_t \in \mathbb{R}^{D\times H_X \times W_X}$.
• Additive conditioning: Conditioning is applied by $F'_t = F^X_t + \Delta F_t$.
• Box prediction: $F'_t$ is passed to a lightweight regression head to yield the updated object location $B_t = (x_t, y_t, w_t, h_t)$.

The backbone’s pre-trained weights remain frozen throughout, ensuring preservation of prior learned semantic representations and limiting optimization to the STC-encoder and a minimal detection head.

2. Additive Siamese Convolutional Subnetwork

Spatial feature integration is accomplished through a shared-weight siamese convolutional network $g$. This component processes $F^Z$ and $F^X_t$ independently through $L$ convolutional blocks (typically $L=3$), each comprising Conv2D$$(D\to D,~3\times 3,~\text{stride}=1,~\text{padding}=1)\to$$ BatchNorm $\to$ ReLU, maintaining spatial dimensions across layers.

After transformation, one obtains $f^Z = g(F^Z)$ and $f^X = g(F^X_t)$ with identical channel depth $D$. To extract local cross-image correlations, $f^Z$ is slide-correlated over $f^X$, effectively implemented as grouped convolution:

$$f^{corr}(i,x,y) = \sum_{u,v} f^X(i, x+u, y+v) \cdot f^Z(i, u, v)$$

where $(i,x,y)$ denotes channel and spatial indices. The result $f^{corr} \in \mathbb{R}^{D\times H_X \times W_X}$ is fused channelwise via a $1\times 1$ convolution to yield $$\Delta F^{spatial} \in \mathbb{R}^{D\times H_X \times W_X}$$. This design recovers fine-grained spatial localization lost during global Transformer pooling, and preserves spatial structure via stride-1 convolutions and cross-correlation.

3. Temporal Sequence Model

Temporal conditioning exploits the object’s recent motion, embedding past $K-1$ bounding boxes plus, optionally, the initial template. Each normalized bounding box $B_j = (x_j, y_j, w_j, h_j)$ is passed through a small MLP to produce $e_j \in \mathbb{R}^{d_m}$ (with $d_m = 64$), and the sequence $[e_{t-K+1}, ..., e_{t-1}]$ is processed by a single-layer LSTM with hidden size $H_h = 128$:

$$h_{t-1}, c_{t-1} = \text{LSTM}([e_{t-K+1:t-1}], (h_{t-K}, c_{t-K}))$$

The final hidden state is projected to the feature depth through $m_t = W_m h_{t-1} + b_m$, and broadcast spatially:

$$\Delta F^{temporal}_t = \mathrm{reshape}(m_t) \in \mathbb{R}^{D \times H_X \times W_X}$$

A plausible implication is that this temporal embedding enables the tracker to anticipate smooth or abrupt motion patterns, improving robustness under appearance or scale changes.

4. Additive Spatio-Temporal Conditioning and Mathematical Formulation

Spatio-temporal conditioning is applied via channelwise addition:

$$\Delta F_t = \Delta F^{spatial}_t + \Delta F^{temporal}_t$$

$F'_t = F^X_t + \Delta F_t$

Summary of core equations: $$\begin{align*} &f^Z = g_{\theta}(F^Z),\quad f^X = g_{\theta}(F^X_t),\ &f^{corr} = \text{Corr}(f^X, f^Z),\ &\Delta F^{spatial}_t = W_{1 \times 1} * f^{corr},\ &m_t = W_m h_{t-1} + b_m,\quad \Delta F^{temporal}_t = \text{reshape}(m_t),\ &\Delta F_t = \Delta F^{spatial}_t + \Delta F^{temporal}_t,\ &F'_t = F^X_t + \Delta F_t. \end{align*}$$

For the convolutional block, the operation per layer follows: $$y_{i, x, y} = \sigma \left(\sum_{c=1}^D\sum_{u=-1}^1\sum_{v=-1}^1 W_{c, i, u, v}\, x_{c, x+u, y+v} + b_i\right)$$ with $\sigma$ denoting ReLU activation.

5. Training Procedure and Optimization

The training pipeline is designed for efficiency:

• Frozen: All Transformer backbone parameters.
• Trainable: STC-encoder (siamese conv, $1\times1$ fusion, MLP$_{\text{bbox}}$, LSTM, $W_m$, $b_m$) and the final regression head (two-layer MLP mapping pooled $F'_t$ to 4D bounding box).
• Losses: $\ell_1$ loss on predicted vs. ground-truth box coordinates ($$L_{\text{bbox}} = \|B_t^{\text{pred}} - B_t^{\text{gt}}\|_1$$) plus, optionally, IoU/GIoU loss ($L_{\text{iou}}(B_{\text{pred}}, B_{\text{gt}})$). Total objective: $L = L_{\text{bbox}} + \lambda \cdot L_{\text{iou}}$.
• Optimizer: AdamW (learning rate $1 \times 10^{-4}$, weight decay $1 \times 10^{-3}$).
• Batch size: 32 sequence samples.
• Schedule: 50 epochs, with learning-rate decay at epochs 30 and 40. Only the STC-encoder and regression head are updated per step.

Hyperparameters include $D=256$, $L=3$ convolutional layers, $K=4$ for temporal window, with dropout 0.1 in MLP and LSTM input.

6. Practical Implementation and Computational Efficiency

Key architectural choices facilitate rapid training and deployment:

• The backbone is frozen, thus preserving semantic richness and allowing standard pre-trained ViT/Transformer models.
• The STC-encoder has approximately 1.5 million parameters.
• Grouped convolution implements efficient spatial cross-correlation.
• Training completes in under two days on a single GPU. The entire system forms a compact and fast visual object tracker that achieves state-of-the-art benchmark performance while substantially reducing training cost and memory requirements (Han et al., 2024).

A summary of STC-encoder input/output shapes at each stage:

Module	Input Shape	Output Shape
Transformer (search)	$X_t$ ($C\times H \times W$)	$F^X_t$ ($D\times H_X \times W_X$)
Transformer (template)	$Z$ ($C\times H \times W$)	$F^Z$ ($D\times H_Z \times W_Z$)
Siamese Conv + Correlation	$F^Z$, $F^X_t$	$\Delta F^{spatial}$ ($D\times H_X \times W_X$)
Temporal Sequence Model	Past $K-1$ boxes	$\Delta F^{temporal}$ ($D\times H_X \times W_X$)
Additive Conditioning	$F^X_t$, $\Delta F_t$	$F'_t$ ($D\times H_X \times W_X$)
Regression Head	$F'_t$ (pooled/flattened)	$B_t$ (4-dim coord)

7. End-to-End Tracker Pseudocode

The operational flow in pseudocode:

Z = crop_template(frame1, bbox1) F_Z = Transformer(Z) # frozen B_history = [bbox1] initialize LSTM hidden h0, c0 = zeros() for t in range(2, T+1): X_t = crop_search(frame_t, B_history[-1]) F_X = Transformer(X_t) # frozen # Siamese conv spatial condition fZ = SiameseConv(F_Z) fX = SiameseConv(F_X) f_corr = cross_correlation(fX, fZ) deltaF_spatial = Conv1x1(f_corr) # Temporal condition e_seq = [MLP_bbox(b) for b in B_history[-(K-1):]] h_prev, c_prev = LSTM(e_seq, (h_{t-K}, c_{t-K})) m_t = W_m @ h_prev + b_m deltaF_temporal = broadcast(m_t, H_X, W_X) # Combine and add deltaF_t = deltaF_spatial + deltaF_temporal F_prime = F_X + deltaF_t # Predict new bbox B_t = RegHead(FlattenPool(F_prime)) # Update history B_history.append(B_t)

This execution model enables efficient video object tracking with minimal computational overhead and no degradation of backbone representations, facilitating robust spatio-temporal integration for high-performance VOT applications (Han et al., 2024).