SportsMoments: Fine-Grained Sports Video Retrieval

Updated 5 July 2026

SportsMoments is a benchmark for fine-grained video-to-video moment retrieval in sports, featuring 770K query-target pairs over 29 action classes in soccer and cricket.
It challenges models to perform cross-video temporal-semantic alignment on full-length match videos using dual-stage sequence alignment with soft-DTW.
The benchmark rigorously evaluates retrieval performance via metrics like mIoU and R@1, demonstrating significant gains with the proposed MATR model.

SportsMoments is a benchmark for fine-grained video-to-video moment retrieval in sports, introduced to localize, within a target match video, the start and end times of a moment that semantically matches a query clip. It was proposed because existing resources such as Sports-1M and ActivityNet-VRL do not adequately capture fine-grained sports moments such as “Cover Drive,” “Ducking a Bouncer,” “Goal Kick,” and “Penalty.” Built from 176.6 hours of complete match footage drawn from 80 full-length YouTube videos in soccer and cricket, SportsMoments contains about 770K query-target pairs over 29 action classes and evaluates retrieval under disjoint action classes across train, validation, and test splits (Kumar et al., 21 Aug 2025). In adjacent literature, the phrase “sports moments” is also used more broadly for meaningful segments surfaced from tracking data, completion instants of actions, active-play intervals, highlight shots, and momentum shifts within matches (Metulini, 2016, Heidarivincheh et al., 2018, Facchinetti et al., 2019, Díaz-Juan et al., 1 Sep 2025, Goyal et al., 2020).

1. Concept and scope

SportsMoments is defined through the task of video-to-video moment retrieval: given a query video $V_q$ and a target video $V_t$ , the system must retrieve the temporally localized moment in $V_t$ that matches the query. The benchmark is domain-specific, using only soccer and cricket, and it is explicitly motivated by the claim that prior datasets are either too broad or too coarse for realistic sports retrieval. In particular, the benchmark is positioned against Sports-1M, which is described as primarily for broad video classification, and ActivityNet-VRL, whose sports coverage includes coarse categories such as “Playing Polo” rather than fine-grained within-sport events (Kumar et al., 21 Aug 2025).

The benchmark’s design makes the retrieval problem difficult in the way sports footage is difficult. The target videos are full-length match videos, not short trimmed clips, so the model must search through long, repetitive, context-heavy broadcasts in which many moments are visually similar. The retrieval objective is therefore not category recognition alone; it is cross-video temporal-semantic alignment under variation in speed, viewpoint, and surrounding context. The paper emphasizes that this is especially natural for sports, where a user may have an example clip but may not know the precise vocabulary needed to describe a highly specific event (Kumar et al., 21 Aug 2025).

2. Corpus, annotation, and split design

The dataset is constructed from 176.6 hours of complete match footage covering 80 full-length videos from YouTube. It contains 29 action classes in total: 13 soccer actions and 16 cricket actions. Representative examples named in the paper include “Cover Drive,” “Ducking a Bouncer,” “Goal Kick,” and “Penalty.” The annotation unit is a temporal segment in a full match, and two annotators with strong knowledge of these sports marked the start and end timestamps for the specified actions (Kumar et al., 21 Aug 2025).

Statistic	Value
Match footage	176.6 hours
Full-length videos	80
Sports	Soccer, cricket
Action classes	29
Soccer classes	13
Cricket classes	16
Query-target pairs	$\sim 770$ K

The split structure is central to the benchmark. The training split contains approximately 750K pairs across 16 classes; the validation split contains 10K pairs across 4 classes; and the test split contains 10K pairs across 9 classes. The paper states that action classes are disjoint across train, validation, and test, and that each split includes both cricket and soccer actions, producing a “well-rounded distribution.” This makes SportsMoments an unseen-class retrieval benchmark rather than a same-class memorization test. The paper does not report SportsMoments-specific summary statistics for average clip duration, median clip duration, or minimum/maximum clip duration (Kumar et al., 21 Aug 2025).

3. Retrieval formulation and the MATR model

SportsMoments supports a retrieval problem in which the query is itself a video segment. The paper represents the target video by $M$ uniformly sampled frames and the query video by $N$ uniformly sampled frames, with frames sampled every 2 seconds. The proposed model, MATR (Moment Alignment TRansformer), begins from frozen CLIP ViT-B/32 features and learns a query-conditioned representation of the target video through an encoder, a dual-stage alignment mechanism, and a decoder with temporal prediction heads (Kumar et al., 21 Aug 2025).

The input embeddings are written as

$\mathrm{E}_t \in \mathbb{R}^{M \times d}, \qquad \mathrm{E}_q \in \mathbb{R}^{N \times d},$

and concatenated as

$\mathrm{E}_c = [\mathrm{E}_t;\mathrm{E}_q] \in \mathbb{R}^{(M+N)\times d}.$

A Transformer encoder produces fused query-conditioned target and query representations, after which the model performs dual-stage sequence alignment using soft-DTW: first on the pre-fusion representations, then again on the post-fusion representations. The paper’s ablations show that post-fusion alignment contributes more than pre-fusion alignment, but that the two together are strongest.

After alignment, a Transformer decoder refines the aligned target subsequence. Two prediction heads then operate on the final representation: a foreground/background classification head and a boundary prediction head. The overall training loss combines foreground classification, boundary prediction, and the two alignment losses: $\mathcal{L} = \frac{1}{S} \sum_{i=1}^{S} \left( \lambda_{fg}\mathcal{L}_{fg} + \lambda_{\text{seg}}\mathcal{L}_{\text{seg}} + \lambda_{\text{align}}^{\text{pre}}\mathcal{L}_{\text{align}}^{\text{pre}} + \lambda_{\text{align}}^{\text{post}}\mathcal{L}_{\text{align}}^{\text{post}} \right).$

The model is further strengthened by a task-specific self-supervised pretraining stage. A random clip is sampled from within a target video and used as a query, and the model is trained to localize that clip back inside the source video. The augmentations used in pretraining are reversing frames, adding Gaussian noise, slowing down the action, and speeding up the action. For SportsMoments, the implementation details reported are: hidden dimension 1024, 4 encoder layers, 4 decoder layers, Transformer dropout 0.1, linear projection dropout 0.5, $l=10$ learnable decoder queries, AdamW with learning rate $V_t$ 0, weight decay $V_t$ 1, 40 training epochs, batch size 40, and all $V_t$ 2 weights set to 1 (Kumar et al., 21 Aug 2025).

4. Evaluation protocol and empirical results

SportsMoments is evaluated with standard video-to-video moment retrieval metrics: mIoU and R@1, IoU=0.5. The strongest implemented baseline reported on SportsMoments is UniVTG Finetuned+V, while the proposed model is reported in zero-shot, trained, and fine-tuned forms. The paper’s main claim is that MATR is substantially stronger than all adapted baselines, with especially large gains on this fine-grained sports benchmark (Kumar et al., 21 Aug 2025).

Method	mIoU	R@1
UniVTG Finetuned+V	43.6	41.8
MATR Zero-shot	31.8	30.7
MATR Trained+V	56.2	52.7
MATR Finetuned+V	59.2	56.5

Relative to UniVTG Finetuned+V, MATR Finetuned+V improves by +15.6 mIoU and +14.7 R@1 according to the table values. The abstract reports a 14.7% gain in R@1 and a 14.4% gain in mIoU over strong baselines, while the detailed discussion states +15.6 mIoU and +14.7 R@1 over UniVTG; the table-based comparison is therefore the more specific one.

The ablations explain where these gains come from. Without either alignment stage, performance is 49.1 mIoU and 46.6 R@1; with both pre-fusion and post-fusion alignment, it rises to 56.2 mIoU and 52.7 R@1. Replacing no alignment with soft-DTW raises performance from 52.1 mIoU / 48.6 R@1 to 59.2 mIoU / 56.5 R@1. Self-supervised pretraining with augmentation further improves over no pretraining, from 56.2 mIoU / 52.7 R@1 to 59.2 mIoU / 56.5 R@1. The paper also shows that predicting directly from alignment matrices is much weaker than using the decoder plus dedicated prediction heads: the best alignment-only variant reaches 36.4 mIoU / 33.5 R@1, far below the full model (Kumar et al., 21 Aug 2025).

In adjacent sports analytics literature, “sports moments” also denotes meaningful segments identified from movement or event data rather than video-to-video retrieval. One line of work uses interactive motion charts to surface candidate moments from player-tracking trajectories. In a basketball case study, gvisMotionChart from the googleVis R package is used to animate player trajectories, inspect offensive and defensive spacing, and isolate short windows as candidate “moments.” The same study then validates visual impressions with team-level metrics such as average inter-player distance and Voronoi area, reporting Defense: 28.47 $V_t$ 3, 5.68 m and Attack: 42.59 $V_t$ 4, 7.25 m (Metulini, 2016).

A second line treats moments as action-completion instants. “Action Completion: A Temporal Model for Moment Detection” formulates the task as locating the first frame at which an action’s goal is achieved, or declaring the sequence incomplete. Its best classification-pre / regression-post model detects the completion moment within 30 frames (1 second) in 89% of tested sequences, within 15 frames (0.5 second) in 74%, and at the exact frame in 30.4% of cases (Heidarivincheh et al., 2018).

In sensor-based basketball analytics, “moments” may instead denote active game periods. A rule-based method removes inactive periods using three criteria—wrong number of players on court, free-throw situations, and sustained low movement of all five on-court players—and then labels active moments as offensive, defensive, or transition. On a case study from the Italian Basketball Cup Final Eight 2017, the tuned parameters are reported as $V_t$ 5 km/h, $V_t$ 6, with best AUC = 0.8329 for active/inactive classification (Facchinetti et al., 2019).

In soccer broadcast understanding, moments become summary segments rather than isolated events. SoccerHigh introduces a benchmark of 237 matches from the Spanish, French, and Italian leagues, aligned to official highlight reels. Its baseline, using VideoMAEv2 ViT-Giant features and a 2-layer Transformer encoder, achieves a test F1 = 0.3956, while the length-constrained F1 Score@T is 0.3883 (Díaz-Juan et al., 1 Sep 2025).

Statistical work on tennis and football uses the language of moments to describe momentum or change points rather than retrieval targets. In Grand Slam tennis, generalized linear mixed-effect models applied to data from 2014 through 2019 find strong evidence of carryover effects at the set, game, and point level, including the result that winning the previous set increases the odds of winning the current set by 56.6% for AO men, 114.5% for AO women, 108.6% for FO men, 182.1% for FO women, 65.5% for WB men, 187.2% for WB women, 79.0% for USO men, and 42.8% for USO women (Goyal et al., 2020). In football, a copula-based multivariate hidden Markov model fitted to 34 Bundesliga matches of Borussia Dortmund in 2017/18 infers latent states interpretable as low control, balanced play, and high control, thereby modeling momentum shifts as changes in latent match-control regimes (2002.01193).

6. Limitations, reuse, and position in the literature

SportsMoments is deliberately specialized. Its experimental domain is limited to soccer and cricket, and its source videos are full-length match videos from YouTube. The paper does not provide SportsMoments-specific descriptive statistics for average query duration or target clip duration, nor does it spell out the exact combinatorics used to generate the approximately 770K query-target pairs. These omissions do not change the benchmark’s scale, but they do constrain reproducibility at the level of fine-grained data generation (Kumar et al., 21 Aug 2025).

At the same time, later work indicates that the benchmark is reusable. A subsequent paper on Generalized Moment Retrieval builds Soccer-GMR partly from Sportsmoments and reports, on 100 randomly sampled clips, a mean boundary deviation < 2s after human verification. That later benchmark also generalizes the retrieval setting from “exactly one matching moment” to a setting in which a query may correspond to multiple or no moments, a direction that is especially natural for sports search interfaces built around repeated event types such as corners, saves, offsides, or fouls (Ding et al., 4 May 2026).

Taken together, the literature positions SportsMoments as a benchmark for fine-grained cross-video temporal-semantic alignment in sports. It is distinct from tracking backbones, movement visualization tools, or momentum models, but it interfaces naturally with them: tracking and pose estimation can provide lower-level temporal structure, while completion detection, summarization, and momentum analysis define adjacent notions of what counts as an important sports moment. Within that landscape, SportsMoments contributes a concrete retrieval problem: given one sports event as a video example, find its counterpart in another full match video (Kumar et al., 21 Aug 2025).