MatchTime: Temporal Alignment for Soccer Commentary

• MatchTime is a multi-modal temporal alignment framework that corrects video–text misalignment for soccer commentary generation.
• It integrates coarse audio-based summarization with fine-grained CLIP contrastive alignment to precisely synchronize commentary and game events.
• Enhanced alignment in MatchTime leads to significant improvements in BLEU, METEOR, and CIDEr scores, underpinning the success of the MatchVoice model.

MatchTime is a multi-modal temporal alignment framework and an aligned training dataset for automatic soccer game commentary generation, introduced to correct the pervasive video–text misalignment in SoccerNet-Caption and used to train the commentary model MatchVoice (Rao et al., 2024). In the original formulation, MatchTime combines coarse alignment from broadcast audio and LLM summarization with fine-grained CLIP-based contrastive alignment, and it is evaluated together with the manually realigned benchmark SN-Caption-test-align and the MatchVoice generator (Rao et al., 2024). Subsequent work reuses the same alignment and generation paradigm for short soccer highlights in the GOAL dataset, treating MatchTime as the central prior system for temporally aligned soccer commentary beyond full-match streams (Ravuru, 11 Aug 2025).

1. Problem setting and motivation

MatchTime was proposed for automatic soccer game commentary generation: given a soccer broadcast video, the commentary model should generate a textual commentary for each clip, with the task formalized as

$$\mathcal{V} = \{\mathbf{V}_1, \mathbf{V}_2, \dots, \mathbf{V}_T\}, \qquad \hat{\mathbf{C}}_i = \Psi(\mathbf{V}_i; \Theta_2).$$

The target is not generic captioning, but professional soccer commentary that captures shots, fouls, build-ups, and tactical context with soccer terminology (Rao et al., 2024).

The framework is motivated by the observation that SoccerNet-Caption pairs broadcast video with text live commentaries scraped from websites, and that these timestamps exhibit substantial temporal delay relative to the visual event. In the manually re-annotated test material used to study this problem, offsets range from $-108$ to $152$ seconds, with average offset $13.85$ seconds and mean absolute offset $16.63$ seconds. Only 26.29% of samples fall within 10 seconds of the correct frame, 60.21% within 30 seconds, 74.96% within 45 seconds, and 85.03% within 60 seconds (Rao et al., 2024).

This misalignment affects both training and evaluation. A commentary about a goal may be paired with frames showing a replay or a different phase of play, which corrupts the supervision signal. MatchTime addresses this by introducing a manually aligned benchmark, an automatic large-scale alignment pipeline, and a new commentary model trained on the corrected data. A later highlight-oriented study preserves this basic diagnosis, arguing that temporal alignment remains central even when the unit of analysis shifts from full matches to 10–15 second highlight clips (Ravuru, 11 Aug 2025).

2. Benchmark, dataset, and corpus structure

MatchTime is closely tied to three distinct artifacts: the manually aligned benchmark SN-Caption-test-align, the automatically aligned MatchTime training corpus, and the MatchVoice generation model.

Artifact	Function	Scale
SN-Caption-test-align	Manually aligned benchmark and alignment supervision	49 matches, 3,267 pairs
MatchTime	Automatically aligned training and validation dataset	422 matches, 29,476 pairs
MatchVoice	Commentary generation model trained on aligned pairs	Trained on MatchTime

SN-Caption-test-align was created by manually realigning 49 matches from the SoccerNet-Caption test set. The annotation procedure used 20 football fans and instructed them to watch the entire video, adjust each commentary timestamp to match the moments when events occur, place timestamps 1 second earlier for continuous actions such as shots, passes, and fouls, and mark the first occurrence rather than a replay when replay footage was present. The resulting benchmark contains 3,267 video clip–text pairs (Rao et al., 2024).

This benchmark serves two roles. First, it is the evaluation set for commentary generation. Second, it provides supervision for the temporal alignment model: 45 videos comprising 2,975 pairs are used to train the alignment module, and 4 videos comprising 292 samples are held out as unseen alignment test data (Rao et al., 2024).

The MatchTime dataset itself is obtained by automatically aligning all SoccerNet-Caption training and validation matches. It contains 422 matches in total, with 373 training matches and 49 validation matches. The aligned corpus contains 29,476 video–text pairs, split into 26,058 training pairs and 3,418 validation pairs (Rao et al., 2024). In this organization, the manually aligned 49-match benchmark remains conceptually separate from the automatically aligned training and validation data.

A later extension transfers the MatchTime paradigm to the GOAL dataset, using short highlight clips with associated textual descriptions or commentary. In that setting, the downloaded corpus consists of 358 training videos, 86 validation videos, and 192 test videos, while approximately 500 additional videos could not be downloaded because of regional issues and were excluded (Ravuru, 11 Aug 2025). This change in corpus structure is accompanied by a change in temporal granularity: full-match, streaming commentary in MatchTime versus short, self-contained highlight clips in GOAL.

3. Multi-modal temporal alignment pipeline

The MatchTime alignment pipeline is explicitly two-stage: coarse alignment followed by fine-grained alignment (Rao et al., 2024).

In the coarse stage, WhisperX is applied to match audio to obtain narration text with corresponding timestamp intervals. Because the resulting speech transcripts are fragmented and colloquial, the video is divided into 10-second segments and LLaMA-3 is used to summarize the key events in each segment into timestamped event descriptions. These descriptions are then used to align the original SoccerNet-Caption textual commentaries to approximate time intervals, producing coarse corrected timestamps (Rao et al., 2024). This stage is robust but approximate, and matches without audio skip it.

In the fine-grained stage, MatchTime trains a contrastive temporal alignment model on the manually annotated benchmark. The model uses pre-trained CLIP ViT-B/32 encoders for text and vision, followed by trainable MLPs $f(\cdot)$ and $g(\cdot)$ that map commentary and key-frame features into a shared $d=512$ dimensional space: $$\text{C}, \text{V} = f(\Phi_{\text{CLIP-T}}(\mathcal{C})), \hspace{3pt} g(\Phi_{\text{CLIP-V}}(\mathcal{V})),$$ with $\text{C} \in \mathbb{R}^{k \times d}$ and $-108$0. The affinity matrix is computed by cosine similarity,

$-108$1

and training uses an InfoNCE-like contrastive loss

$-108$2

where $-108$3 if commentary $-108$4 corresponds to frame $-108$5, and $-108$6 otherwise (Rao et al., 2024).

Training samples frames at 1 FPS within a 2-minute window around each ground-truth timestamp, uses the 45 manually annotated videos with 2,975 pairs, and optimizes with AdamW at learning rate $-108$7 for 50 epochs. At inference time, because coarse preprocessing has already narrowed the search region and soccer videos contain replays, the fine-grained model searches from 45 seconds before to 30 seconds after the current coarse timestamp, chooses

$-108$8

and updates the commentary timestamp by setting $-108$9 (Rao et al., 2024).

The empirical effect of this pipeline is substantial. On 4 unseen annotated matches comprising 292 samples, the full pipeline reduces average absolute offset from $152$0 seconds to $152$1 seconds, increases $152$2 from 35.32% to 80.73%, and increases $152$3 from 88.07% to 98.17% (Rao et al., 2024). The paper describes this as evidence that better temporal alignment directly improves downstream commentary generation.

4. MatchVoice generation model

MatchVoice is the commentary generation model trained on MatchTime-aligned segment–commentary pairs. It has three principal components: a frozen visual encoder $152$4, a Perceiver-like temporal aggregator $152$5, and a projection-plus-decoder stack $152$6 based on LLaMA-3 (Rao et al., 2024).

Given a video segment $152$7, the visual encoder produces frame-level features

$152$8

the temporal aggregator produces temporally aware features

$152$9

and an MLP projection maps $13.85$0 into prefix tokens for the decoder, which then generates commentary: $13.85$1 Training uses standard negative log-likelihood,

$13.85$2

with aligned visual context as conditioning input (Rao et al., 2024).

The temporal aggregator is described as a Perceiver-like architecture with two transformer decoder layers and fixed-length learnable query tokens. The projection layer outputs 768-dimensional prefixes to match LLaMA-3. The visual encoder remains frozen, and the model can use C3D, ResNet, Baidu soccer embeddings, CLIP, or InternVideo as framewise visual features. All encoders are framewise at 2 FPS except InternVideo and Baidu, which use 1 FPS. InternVideo internally aggregates 8 frames per second into 1 feature (Rao et al., 2024).

Window size is a central hyperparameter in MatchVoice training. For the original full-match setting, the model is trained on temporal windows of 10, 30, 45, or 60 seconds around the aligned timestamp, and the default best-performing choice is 30 seconds (Rao et al., 2024). In the later highlight setting, the same architecture is reused but the window sizes are shortened to 3, 5, 10, or 15 seconds to reflect the shorter GOAL clips (Ravuru, 11 Aug 2025).

The architectural significance of MatchVoice lies in its modularity. The original paper presents it as a commentary-specific video–LLM trained on curated aligned data (Rao et al., 2024). The later highlight study does not introduce a new generator; instead, it preserves the MatchVoice design and alters the data distribution, windowing strategy, fine-tuning regimen, and evaluation regime (Ravuru, 11 Aug 2025). This suggests that MatchTime’s distinctive contribution is as much about temporal supervision quality as about decoder design.

5. Empirical results and ablation findings

MatchTime is evaluated both as an alignment method and as a training corpus for commentary generation. Commentary generation is measured on SN-Caption-test-align with BLEU-1, BLEU-4, METEOR, ROUGE-L, CIDEr, and GPT-score in the original work (Rao et al., 2024). The highlight extension uses BLEU-1, BLEU-4, METEOR, ROUGE-L, CIDEr, and sBERT (Ravuru, 11 Aug 2025).

For the SN-Caption baseline with Baidu features, training on original SoccerNet-Caption yields BLEU-1 29.61, BLEU-4 6.83, METEOR 25.38, ROUGE-L 25.28, CIDEr 20.61, and GPT-score 6.72. Training the same baseline on MatchTime yields BLEU-1 29.74, BLEU-4 7.31, METEOR 26.40, ROUGE-L 26.19, CIDEr 23.74, and GPT-score 6.84 (Rao et al., 2024). These numbers isolate the effect of alignment quality independently of architectural change.

For MatchVoice with Baidu features, training on original SoccerNet-Caption yields BLEU-1 30.32, BLEU-4 8.45, METEOR 25.25, ROUGE-L 29.40, CIDEr 33.84, and GPT-score 7.07. Training on MatchTime yields BLEU-1 31.42, BLEU-4 8.92, METEOR 26.12, ROUGE-L 29.66, CIDEr 38.42, and GPT-score 7.08 (Rao et al., 2024). Across alternative encoders, the original paper reports consistent CIDEr gains when training on MatchTime rather than unaligned SoccerNet-Caption.

Ablation studies further identify the interaction between temporal alignment and window size. Without alignment, a 30-second window gives the strongest overall MatchVoice results, while excessively short or long windows degrade performance. With alignment, a 10-second window already becomes highly informative, reaching CIDEr 40.51, but the final selected configuration remains a 30-second window with BLEU-1 31.42, BLEU-4 8.92, METEOR 26.12, ROUGE-L 29.66, and CIDEr 38.42 (Rao et al., 2024). The paper explicitly interprets this as evidence that temporal alignment is especially crucial for shorter windows, where there is little slack for timestamp error.

A second ablation isolates alignment stages. With MatchVoice plus Baidu features and a 30-second window, the original data yields BLEU-1 30.32, BLEU-4 8.45, METEOR 25.25, ROUGE-L 29.40, and CIDEr 33.84. Coarse-only alignment yields BLEU-1 30.52, BLEU-4 8.90, METEOR 25.73, ROUGE-L 28.18, and CIDEr 37.53. Fine-only alignment yields BLEU-1 30.55, BLEU-4 8.81, METEOR 26.03, ROUGE-L 29.40, and CIDEr 36.13. Coarse plus fine yields BLEU-1 31.42, BLEU-4 8.92, METEOR 26.12, ROUGE-L 29.66, and CIDEr 38.42 (Rao et al., 2024). The full pipeline therefore produces the largest improvement.

6. Highlight adaptation, limitations, and future directions

The 2025 highlight-focused study extends MatchVoice to short soccer highlight clips in GOAL while preserving the core MatchTime alignment pipeline and model structure (Ravuru, 11 Aug 2025). It reimplements coarse ASR plus LLM alignment, fine-grained CLIP contrastive alignment, and the MatchVoice generator, but changes the task from full 90+ minute games with dense commentary to 10–15 second highlight clips. It also modifies windowing and fine-tuning under resource-constrained training, including fine-tuning on only 100 GOAL videos because of hardware limits (Ravuru, 11 Aug 2025).

This work also reports a reproduction of MatchTime under different hardware. The original MatchTime used A100 GPUs with batch size 64 and 16 workers, whereas the reproduction uses a single NVIDIA L40 GPU with batch size 32 and 8 workers. In the reproduced SN-Caption setting, the Baidu Aligned variant obtains BLEU-1 29.643, BLEU-4 8.227, METEOR 26.491, ROUGE-L 26.316, CIDEr 34.452, and sBERT 68.981, which the authors describe as close to the original results (Ravuru, 11 Aug 2025).

For GOAL zero-shot transfer, window size again proves decisive. With MatchVoice trained on MatchTime/SoccerNet-Caption-aligned data and evaluated zero-shot on GOAL, a 3-second window yields BLEU-1 6.832, BLEU-4 0.056, METEOR 14.932, ROUGE-L 8.235, CIDEr 1.079, and sBERT 33.736; a 5-second window yields BLEU-1 6.905, BLEU-4 0.107, METEOR 14.889, ROUGE-L 8.312, CIDEr 1.130, and sBERT 33.868; a 10-second window yields BLEU-1 7.120, BLEU-4 0.095, METEOR 15.157, ROUGE-L 8.449, CIDEr 1.264, and sBERT 34.299; and a 15-second window yields BLEU-1 7.024, BLEU-4 0.091, METEOR 15.105, ROUGE-L 8.493, CIDEr 1.232, and sBERT 34.010 (Ravuru, 11 Aug 2025). The study therefore selects a 10-second default for GOAL because it gives the best overall zero-shot metrics.

Fine-tuning on GOAL improves highlight commentary metrics. Before fine-tuning, GOAL test performance is BLEU-1 7.120, BLEU-4 0.095, METEOR 15.157, ROUGE-L 8.449, CIDEr 1.264, and sBERT 34.299. After fine-tuning, it becomes BLEU-1 8.24, BLEU-4 0.401, METEOR 16.138, ROUGE-L 8.92, CIDEr 2.18, and sBERT 38.76 (Ravuru, 11 Aug 2025). The same fine-tuning reduces SN-Caption lexical metrics, with BLEU-1 decreasing from 27.433 to 20.12 and CIDEr from 24.965 to 21.23, while METEOR and sBERT remain relatively high at 23.89 and 63.421 respectively (Ravuru, 11 Aug 2025). This suggests a domain shift from full-match broadcast-style commentary toward highlight-style descriptions.

The highlight study also compares MatchVoice with other video–language baselines. On GOAL, zero-shot Video-ChatGPT yields BLEU-1 1.484, BLEU-4 0.017, ROUGE-L 4.166, CIDEr 0.005, and sBERT 28.380, while zero-shot plus post-processing yields BLEU-1 2.260, BLEU-4 0.062, ROUGE-L 5.935, CIDEr 0.214, and sBERT 31.842 (Ravuru, 11 Aug 2025). On SN-Caption, UniSoccer or MatchVision yields BLEU-1 21.65, BLEU-4 3.27, ROUGE-L 21.02, CIDEr 17.79, and sBERT 12.90 in zero-shot mode, and BLEU-1 27.49, BLEU-4 6.96, ROUGE-L 24.50, CIDEr 23.33, and sBERT 30.81 when fine-tuned on MatchTime (Ravuru, 11 Aug 2025). The paper interprets these comparisons as evidence that domain-specific alignment and generative training remain advantageous for soccer commentary.

Several limitations are explicit across the MatchTime line of work. The original paper notes that current commentary remains anonymous and cannot accurately describe player information on the field, and that MatchVoice may struggle to distinguish highly similar actions such as corner kicks and free kicks because of frozen generic visual and language encoders (Rao et al., 2024). The highlight study adds that dataset captions may remain misaligned, GOAL often anonymizes entities as $13.85$3 and $13.85$4, highlight fine-tuning is performed on only around 100 videos, and broader video–language techniques such as large multimodal transformers, stronger video-language pretraining, better temporal localization modules, temporal proposal networks, and retrieval-augmented generation may be necessary for further progress (Ravuru, 11 Aug 2025).

Reproducibility resources are available but distributed across projects. The MatchTime paper provides a project page at https://haoningwu3639.github.io/MatchTime/, while the highlight adaptation releases code at https://github.com/chidaksh/SoccerCommentary (Rao et al., 2024, Ravuru, 11 Aug 2025). The original paper does not specify licensing or exact download instructions in the main text.