TrackNetV5: Advanced Object Tracking

• TrackNetV5 is an advanced object tracking architecture that integrates Motion Direction Decoupling (MDD) and Residual-Driven Spatio-Temporal Refinement (R-STR) to enhance the tracking of small, fast-moving objects in sports videos.
• It augments a U-Net-like backbone with explicit motion polarity encoding and factorized spatio-temporal attention, leading to significant reductions in false negatives and improved real-time performance.
• Empirical evaluation on the TrackNetV2 tennis dataset demonstrates a state-of-the-art F1-score of 0.9859 and an accuracy of 0.9733, marking considerable advances over previous TrackNet versions.

TrackNetV5 is an object tracking architecture engineered for precision tracking of small, fast-moving objects in sports video, overcoming occlusion and motion ambiguity limitations inherent to previous iterations of the TrackNet series. It introduces two novel modules—Motion Direction Decoupling (MDD) and Residual-Driven Spatio-Temporal Refinement (R-STR)—which collectively enable both explicit motion direction encoding and robust spatio-temporal context integration. Empirical evaluation on the TrackNetV2 tennis dataset demonstrates a new state-of-the-art with an F1-score of 0.9859 and an accuracy of 0.9733, representing significant improvements over predecessors while retaining real-time inference capabilities (Haonan et al., 2 Dec 2025).

1. Architecture Overview and Data Flow

TrackNetV5 augments the U-Net-like TrackNetV2 backbone with MDD at the input and R-STR at the output, following a coarse-to-fine, multi-frame reasoning paradigm.

• Input: Consecutive RGB frames $I_{t-1}$, $I_t$, and $I_{t+1}$ of spatial size $512 \times 288$.
• MDD Module: Computes motion difference maps, derives signed polarity attention maps, and interleaves these with RGB data, forming a 13-channel input tensor.
• Backbone: Processes the 13-channel input via a U-Net encoder–decoder with skip connections.
• Heatmap Draft: A $1 \times 1$ convolution produces an initial heatmap; polarity maps are fused again.
• R-STR Head: Applies dropout to the fused result during training, extracts a residual correction via a factorized Transformer, and computes the final probability heatmap.
• Loss: Weighted Binary Cross-Entropy (WBCE) against a binary Gaussian ball mask.

The data flow schema is as follows:

1. MDD: $\Delta_1 = I_t - I_{t-1}$, $\Delta_2 = I_{t+1} - I_t$;
2. Polarity decomposition and attention: $$P_1^+,P_1^-,P_2^+,P_2^- \rightarrow A_1^+,A_1^-,A_2^+,A_2^-$$;
3. Interleaving: $$X_{in} = [I_{t-1}, A_1^+, A_1^-, I_t, A_2^+, A_2^-, I_{t+1}]$$;
4. Backbone/decoder: $\rightarrow$ Draft heatmap;
5. R-STR: Draft + $A$-maps to residual correction, final heatmap $H = \sigma(\text{Draft\_MDD} + \Delta R)$.

2. Motion Direction Decoupling (MDD) Module

The MDD module addresses the directional ambiguity of TrackNetV4's absolute difference preprocessing by restoring signed motion polarity.

2.1 Polarity Decomposition

For adjacent frames, the difference map $\Delta I$ is partitioned:

• $P^+ = \max(\Delta I, 0)$,
• $P^- = \max(-\Delta I, 0)$.

2.2 Learnable Non-linear Attention Mapping

Each polarity field $x$ is mapped by

$$A = f(x; \alpha, \beta) = \frac{1}{1 + \exp\left(-k(\alpha)\cdot(|x| - m(\beta))\right)}$$

with

• $$k(\alpha) = 5.0 / (0.45 \cdot |\tanh(\alpha)| + \epsilon)$$,
• $m(\beta) = 0.6 \cdot \tanh(\beta)$,
• $\alpha, \beta$ are learned scalars; $\epsilon$ ensures stability.

2.3 Feature Interleaving

The attention maps for $t-1 \rightarrow t$ are $A_1^+, A_1^-$, and for $t \rightarrow t+1$ are $A_2^+, A_2^-$. The final input tensor is:

$$X_{in} = \text{concat}\left( I_{t-1}[3],\ A_1^+[1],\ A_1^-[1],\ I_t[3],\ A_2^+[1],\ A_2^-[1],\ I_{t+1}[3] \right)$$

yielding 13 channels.

3. Residual-Driven Spatio-Temporal Refinement (R-STR) Head

R-STR builds on a coarse heatmap by predicting a residual $\Delta R$ that corrects the backbone output, enabling fine-grained recovery from occlusions and capturing temporal dependencies.

3.1 Input Construction

The input to R-STR is the concatenation of the Draft heatmap and all four MDD attention maps, resulting in Draft_MDD.

3.2 Factorized Spatio-Temporal Self-Attention (TSATTHead)

• Patch embedding: Non-overlapping patches per frame are flattened to tokens.
• Spatial block: Within-frame token interactions via multi-head self-attention:

$$\text{Attn}^s = \mathrm{softmax}\left(\frac{Q^s (K^s)^\top}{\sqrt{d_k}}\right)V^s$$

• Temporal block: Tokens at the same patch location are sequenced temporally across three frames and passed through temporal MHSA:

$$\text{Attn}^t = \mathrm{softmax}\left(\frac{Q^t (K^t)^\top}{\sqrt{d_k}}\right)V^t$$

• Reconstruction: Tokens are rearranged and PixelShuffle upsamples back to spatial resolution, yielding $\Delta R$.

3.3 Output Formulation

• Training: $\Delta R = \text{TSATTHead}(\text{Draft}')$ (with $0.1$ dropout).
• Inference: Dropout omitted.

The final heatmap is

$H_\text{final} = \sigma(\text{Draft (or Draft}')} + \Delta R)$

4. Implementation and Training Protocol

• Backbone: U-Net style encoder-decoder with skip connections (TrackNetV2).
• Input resolution: $512 \times 288$ per frame.
• Batch size: 2.
• Optimizer: AdamW, initial learning rate $1\times 10^{-4}$, decay by $\gamma=0.1$ at epochs 20 and 25, train for 30 epochs.
• Loss: WBCE to correct foreground/background imbalance.
• Hardware: Training on NVIDIA RTX 4090; inference on NVIDIA T4.

Training and inference operations are formalized in the following pseudocode:

def TRAIN_BATCH(I_t_minus_1, I_t, I_t_plus_1, Y_gt): Δ1 = I_t - I_t_minus_1 Δ2 = I_t_plus_1 - I_t P1_plus, P1_minus = ReLU(Δ1), ReLU(-Δ1) P2_plus, P2_minus = ReLU(Δ2), ReLU(-Δ2) A1_plus, A1_minus = f(P1_plus; alpha, beta), f(P1_minus; alpha, beta) A2_plus, A2_minus = f(P2_plus; alpha, beta), f(P2_minus; alpha, beta) X_in = concat(I_t_minus_1, [A1_plus, A1_minus], I_t, [A2_plus, A2_minus], I_t_plus_1) bottleneck = V2_Backbone(X_in) Draft = Conv1x1(decoder_features) Draft_MDD = concat(Draft, A1_plus, A1_minus, A2_plus, A2_minus) Draft_prime = dropout(Draft_MDD, ρ=0.1) ΔR = TSATTHead(Draft_prime) H_pred = sigmoid(Draft_prime + ΔR) L = WBCE(H_pred, Y_gt) backprop(L) def INFER(I_t_minus_1, I_t, I_t_plus_1): # compute A-maps as above X_in = concat(…) Draft = V2_Backbone + 1x1conv Draft_MDD = concat(Draft, A-maps) ΔR = TSATTHead(Draft_MDD) return sigmoid(Draft_MDD + ΔR)

5. Experimental Evaluation and Quantitative Benchmarks

TrackNetV5 was evaluated on the TrackNetV2 public tennis dataset (1280×720), with a 70%/30% train–validation split and center error threshold of 4 pixels, reporting per-frame Precision, Recall, F1, and Accuracy.

Main comparison:

Model	Accuracy	Precision	Recall	F1
V2	0.9396	0.9919	0.9446	0.9677
V4	0.9224	0.9965	0.9225	0.9581
V5	0.9733	0.9923	0.9797	0.9859

• V5 reduces False Negatives from 1,317 (V4) to 344 (–73.9%).
• F1 gain of +2.78% over V4 at high Precision.

Efficiency metrics (inference on NVIDIA T4):

Model	FPS (×3-frame)	FLOPs (G)	Params (M)
V2	41.09	112.89	11.33
V4	40.32	112.89	11.33
V5	38.12	117.09	14.77

• TrackNetV5's increase in FLOPs over V4 is only 3.7%, maintaining over 114 FPS for three-frame multi-input multi-output (MIMO) operation.

Ablations:

• V2 + MDD: F1 = 0.9695 (FN $\downarrow$ 76 vs. V2 alone)
• V2 + R-STR: F1 = 0.9866 (highest), Precision $\downarrow$ 0.39%, FP $\uparrow$ 65
• V5 (MDD+R-STR): Achieves high Recall with competitive Precision and top F1.

6. Comparative Advantages and Mechanistic Insights

TrackNetV5 leverages the complementary strengths of MDD and R-STR to address previous model limitations:

• MDD: Recovers motion polarity (brightening/darkening), providing explicit directionality cues and reducing missed detections (yielding higher Recall and fewer False Negatives), without expanding input depth.
• R-STR: Utilizes factorized space-time self-attention for temporal coherence, efficiently correcting backbone heatmap drafts by residual prediction. This process facilitates the recovery of occluded or ambiguous targets and suppresses motion artifacts.
• Integration: The tandem of explicit motion guidance (MDD) and refined output correction (R-STR) yields robust tracking accuracy and minimal computational overhead relative to earlier versions.

7. Context, Limitations, and Impact

TrackNetV5’s enhancements—explicit encoding of motion direction and residual refinement of coarse predictions—substantially advance high-speed, small-object tracking. This methodology addresses two principal failure modes: motion direction ambiguity (TrackNetV4) and reliance on visual cues alone under occlusion (TrackNetV1–V3). The approach sets a new high-water mark for both precision and computational efficiency in its target domain, under evaluation regimes sensitive to both detection strictness (4-pixel center error) and real-time resource constraints (Haonan et al., 2 Dec 2025).

A plausible implication is that the coarse-to-fine, polarity-aware spatio-temporal modeling paradigm embodied in TrackNetV5 may generalize to other domains where temporal directionality and fine object localization are critical.