TrajSV: Trajectory-Based Sports Video Framework

• TrajSV is a trajectory-based framework that extracts player and ball movements from broadcast videos to create reusable representations.
• It integrates a Clip Representation Network and a Video Representation Network, employing triple contrastive loss without manual labels.
• Empirical results demonstrate significant improvements in HR@1, action spotting accuracy, and captioning metrics across multiple sports.

Searching arXiv for the exact TrajSV paper and a few directly related works named in the provided material. TrajSV is a trajectory-based framework for learning representations of sports broadcast videos, designed specifically for soccer, basketball, and volleyball. It addresses three persistent problems in sports video analytics: data unavailability, lack of a general trajectory-based video framework, and dependence on extensive supervision. The framework comprises three components—data preprocessing, Clip Representation Network (CRNet), and Video Representation Network (VRNet)—and is trained with a triple contrastive loss in an unsupervised manner. Its target applications are sports video retrieval, action spotting, and video captioning on broadcast video data rather than on dedicated tracking-system outputs (Wang et al., 15 Aug 2025).

1. Problem formulation and design rationale

TrajSV is motivated by the observation that many prior sports analytics systems, such as play2vec and Chalkboard, rely on high-precision tracking data captured by professional optical tracking systems or GPS. Those systems are expensive and typically deployed only in elite competitions, producing datasets that are limited in scale and scope. Broadcast videos, by contrast, are ubiquitous but do not directly expose trajectories. TrajSV is designed to recover and exploit player and ball trajectories from such videos, thereby constructing reusable representations from widely available audiovisual material rather than from specialized sensing pipelines (Wang et al., 15 Aug 2025).

The framework also responds to the lack of a unified trajectory-based representation model for sports videos. Existing sports video models for retrieval, spotting, and captioning often operate on short, task-specific clips, use raw visual features such as ResNet or SlowFast embeddings, or rely on simple temporal pooling. TrajSV instead treats trajectories as first-class inputs for both clip-level and video-level representation learning. This suggests a representation-learning perspective in which player and ball motion encode task-agnostic semantics that can later be reused across downstream tasks.

A further design goal is to reduce dependence on manual labels. Most sports video methods are supervised and require timestamped events, action labels, or dense captions. TrajSV is trained without manual labels through structural perturbations of videos and trajectories. This suggests that the framework is intended not merely as a specialized retrieval model, but as an unsupervised pretraining method for sports video understanding more broadly.

2. Preprocessing and trajectory extraction from broadcast video

TrajSV begins with a data preprocessing pipeline that segments raw broadcast videos into sports-relevant clips, calibrates the camera, and performs multi-object tracking to recover player and ball trajectories in field coordinates (Wang et al., 15 Aug 2025).

Video segmentation is based on a camera-type classifier. Each frame is classified into camera types such as main center, left, right, and close-up using a 1D CNN over ResNet features reduced to 512 dimensions by PCA. Consecutive frames with the same type form a clip, clips shorter than $0.4$ seconds are discarded, and sports-action clips correspond to main-camera views. This stage removes replays, studio shots, and crowd shots, preserving segments likely to contain tactical and event information.

Camera calibration is performed to convert image coordinates into field coordinates. The framework uses TVCalib with field markings detected by a DeepLabV3-ResNet segmentation model. Camera and lens distortion parameters are optimized by minimizing reprojection error between detected pixel coordinates of field markings and known world coordinates of a standard field model. For soccer, field coordinates are standardized to

$$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$

The same mapping is used for basketball and volleyball for consistency of spatial encoding.

Trajectory extraction uses FairMOT with a DLA-34 backbone, trained or fine-tuned on SoccerNet-Tracking. The detection head follows “Objects as Points,” learning heatmaps of object centers, offsets, and bounding-box sizes, while a re-identification head links detections across frames. To improve ball tracking, the loss weight for small boxes below $500$ pixels is set to $10$. The resulting bounding-box centers are mapped to field coordinates, and objects tracked outside the field are filtered out. For each tracked object, a trajectory is represented as

$$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$

where $(x_t,y_t)$ are real-world field coordinates at frame $t$.

3. Clip Representation Network (CRNet)

CRNet learns a representation $\mathbf{c}$ for each clip by combining trajectory and visual information. Its trajectory branch converts motion into a sequence of segment matrices, and its visual branch provides clip embeddings from X-CLIP. The outputs are concatenated into a unified clip representation (Wang et al., 15 Aug 2025).

Each clip is divided into non-overlapping time segments, for example $1$ second each, with a fixed number $m$ of segments such as $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$0. For each segment $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$1, the field is partitioned into a grid of fixed cell size, for example $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$2, and a binary segment matrix $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$3 is constructed. A cell is set to $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$4 if any trajectory passes through it within the segment and $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$5 otherwise. This produces a sequence $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$6 that encodes coarse spatial occupancy over time.

To reduce redundancy, TrajSV tokenizes segment matrices following play2vec. A global dictionary of previously seen matrices is maintained; for each $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$7, Jaccard similarity is computed against existing matrices, and a new token is introduced only when similarity falls below $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$8. Each segment is then represented by a segment embedding $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$9 and a positional embedding $500$0, yielding

$500$1

CRNet applies a standard Transformer encoder to $500$2. Multi-head self-attention is written as

$500$3

with

$500$4

The attention output is passed through a position-wise feed-forward network,

$500$5

where $500$6 is ReLU, yielding the trajectory-based clip embedding $500$7. In the implementation, the Transformer encoder has $500$8 layers and $500$9 heads, with $10$0.

The visual branch uses X-CLIP pretrained on Kinetics-600 and outputs a visual embedding $10$1. CRNet concatenates trajectory and visual embeddings: $10$2 This clip representation is then used directly for clip-level tasks and as input to VRNet for video-level aggregation.

4. Video Representation Network (VRNet) and unsupervised objective

VRNet aggregates the sequence of clip embeddings $10$3 into a single video-level representation $10$4. It adopts Set Transformer-style components, specifically the Multihead Attention Block (MAB) and Multihead Self-attention Block (MSB), to model dependencies among clips and to produce content-aware global pooling (Wang et al., 15 Aug 2025).

The MAB is defined as

$10$5

and the MSB is the special case

$10$6

The encoder receives $10$7 and applies two MSB layers: $10$8 The decoder uses a learned seed vector $10$9 and computes

$$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$0

In the reported implementation, clip embeddings of dimension $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$1 are mapped through an encoder of dimensions $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$2, and the final video embedding lies in $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$3.

Training is unsupervised and uses a triple contrastive loss. For each original video $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$4, two corrupted variants are constructed. The intra-clip video $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$5 replaces some trajectories within clips using trajectories sampled from other videos, controlled by a noise rate $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$6. The inter-clip video $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$7 replaces some clips using clips sampled from other videos, again controlled by $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$8. Given embeddings $$T = \bigl( (x_1, y_1), (x_2, y_2), \dots, (x_{|T|}, y_{|T|}) \bigr),$$9, $(x_t,y_t)$0, and $(x_t,y_t)$1, a symmetric InfoNCE loss between variants $(x_t,y_t)$2 and $(x_t,y_t)$3 is

$(x_t,y_t)$4

where

$(x_t,y_t)$5

The final objective is

$(x_t,y_t)$6

with $(x_t,y_t)$7, $(x_t,y_t)$8, and temperature $(x_t,y_t)$9. This objective encourages robustness to trajectory corruption within clips, robustness to clip-level temporal corruption, and mutual information retention between the two corrupted variants.

5. Datasets, training configuration, and downstream tasks

TrajSV is evaluated on three broadcast video datasets spanning soccer, basketball, and volleyball. The experimental protocol emphasizes representation transfer across retrieval, spotting, and captioning tasks rather than end-to-end supervised optimization for a single objective (Wang et al., 15 Aug 2025).

The YouTube dataset contains $t$0 soccer videos from sports channels, with durations from $t$1 to $t$2 seconds and no manual annotations; it is used for retrieval. SoccerNet-v2 contains $t$3 full broadcast games from European leagues and provides $t$4 action classes with single timestamps for action spotting, as well as $t$5 temporally localized textual comments for video captioning. SportsMOT contains $t$6 videos from soccer, basketball, and volleyball and is used for retrieval experiments and qualitative analysis. Although the sports differ in physical geometry, trajectory coordinate ranges are standardized to soccer-field dimensions for all sports.

The framework is trained as follows. The preprocessing pipeline uses a field discretization cell size of $t$7 meters and time segment length of $t$8 second. For YouTube, $t$9, meaning $\mathbf{c}$0 segments per clip and $\mathbf{c}$1 clips per video. CRNet uses $\mathbf{c}$2-dimensional segment embeddings, a $\mathbf{c}$3-layer $\mathbf{c}$4-head Transformer, and X-CLIP as visual backbone. VRNet outputs a $\mathbf{c}$5-dimensional video embedding. The noise rate $\mathbf{c}$6 for corruption is sampled uniformly in $\mathbf{c}$7. Optimization uses SGD with learning rate $\mathbf{c}$8, momentum $\mathbf{c}$9, and $1$0 epochs with early stopping patience $1$1. Batch sizes are $1$2 for YouTube and SoccerNet, and $1$3 for SportsMOT. Training is reported on a single NVIDIA V100 32GB GPU with PyTorch 1.8.

Three downstream applications are emphasized. Sports video retrieval uses a corrupted video $1$4 as query and retrieves its original counterpart using video embeddings $1$5, evaluated by HR@1 and MRR. Action spotting uses SoccerNet labels and evaluates Avg-mAP across $1$6 action classes over tolerances from $1$7 to $1$8 seconds. Video captioning on SoccerNet-Caption uses both dense captioning and single-anchored dense captioning settings, with metrics including mAP@$1$9, BLEU@4, METEOR, ROUGE-L, CIDEr, and SODA$m$0.

6. Empirical results, deployment, and significance

TrajSV achieves state-of-the-art performance in sports video retrieval, large gains in several action spotting categories, and improvements in video captioning. These gains are reported across both soccer-only and multi-sport settings, indicating that the learned representations are not confined to a single sport domain (Wang et al., 15 Aug 2025).

For sports video retrieval at noise rate $m$1, the YouTube dataset results are:

• ResNet (MLP): HR@1 $m$2, MRR $m$3
• X-CLIP (MLP): HR@1 $m$4, MRR $m$5
• TrajSV: HR@1 $m$6, MRR $m$7

On SoccerNet at the same noise rate, the best baseline X-CLIP MLP achieves HR@1 $m$8, MRR $m$9, whereas TrajSV achieves HR@1 $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$00, MRR $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$01. On SportsMOT, TrajSV reaches HR@1 $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$02, MRR $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$03, compared with ResNet (MLP) at HR@1 $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$04, MRR $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$05, and X-CLIP (MLP) at HR@1 $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$06, MRR $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$07. These results are summarized in the paper as nearly $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$08 relative improvement in HR@1 in some settings.

For action spotting on SoccerNet, TrajSV clip embeddings are concatenated with Baidu-AS features. Overall Avg-mAP increases from $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$09 for Baidu-AS and $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$10 for Baidu-AS + play2vec to $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$11 for Baidu-AS + TrajSV. More specifically, TrajSV reaches state-of-the-art performance in $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$12 of $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$13 action categories, including improvements of $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$14 for Yellow$$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$15Red cards, $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$16 for Substitution, and $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$17 for Indirect free-kick.

For video captioning on SoccerNet-Caption, TrajSV improves commentary spotting and caption quality when integrated with Baidu-VC. For example, mAP@30 increases from $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$18 to $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$19, mAP@60 from $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$20 to $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$21, and CIDEr from $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$22 to $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$23. The paper also reports up to approximately $$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$24 relative improvements in some captioning metrics, especially those tied to temporal anchoring.

TrajSV has also been deployed in an industrial sports video search engine at Huawei. Offline, the system computes clip embeddings and video embeddings for database videos and builds an approximate nearest neighbor index using HNSW, with compression or quantization to reduce memory footprint. Online, a query video is embedded, ANN search retrieves top-$$x \in [-52.5, 52.5]\ \text{meters}, \qquad y \in [-34, 34]\ \text{meters}.$$25 candidates, and an optional ranking layer such as GBRank refines results using similarity scores and click-through statistics. The deployed use cases include sports video retrieval, action spotting through plugin clip features, and video captioning through integration with Baidu-VC. A plausible implication is that the framework is intended as a precomputed representation layer rather than as a monolithic end-task network.

7. Position in the literature, limitations, and outlook

TrajSV sits at the intersection of trajectory-based sports analytics, sports video understanding, and contrastive representation learning. Relative to play2vec, Chalkboard, LearnRank, and SPORTS-TRAJ, it differs by extracting trajectories from broadcast videos rather than assuming access to professional tracking systems, and by learning task-agnostic clip and video representations rather than optimizing a single analytic task (Wang et al., 15 Aug 2025).

Relative to sports video models such as CALF, NetVLAD++, LRRCN, Baidu-AS, Baidu-VC, and VCMG, TrajSV contributes trajectory-enhanced embeddings learned without labels. Relative to general contrastive video representation learning methods such as CLIP, X-CLIP, and VideoMAE, its contrastive structure is purely video-structural: it contrasts original videos with trajectory-corrupted and clip-corrupted views, and also contrasts the two corrupted views with each other.

The paper notes several limitations. Representation quality depends on the quality of multi-object tracking and camera calibration, so errors in preprocessing propagate into segment matrices and downstream embeddings. The framework is soccer-centric in several design choices, especially field calibration and action taxonomy, although SportsMOT experiments suggest cross-sport adaptability. The preprocessing pipeline is also computationally nontrivial, taking several times the video duration. These constraints suggest that the representation gains partly depend on the maturity of the geometric preprocessing stack.

Future directions proposed in the paper include better trajectory modeling through continuous state-space models such as Mamba or probabilistic trajectory representations, joint trajectory prediction or imputation with representation learning, richer multimodal tasks involving audio, text, and knowledge graphs, and online adaptation from new games without retraining from scratch. This suggests that TrajSV may be interpreted as a first-stage representation framework that can later be extended toward broader multimodal sports understanding rather than as a closed architecture.