TennisVL: Expert Tennis Video Benchmark

• TennisVL is a large-scale tennis video benchmark featuring 202 Grand Slam matches segmented into 40,523 rally clips with expert analytical commentary.
• It integrates multiple modalities including video, ASR transcripts, shot sequences, scoreboard data, and object trajectories to enable comprehensive tactical inference.
• Paired with TennisExpert, the dataset supports tasks like commentary generation and momentum analysis, evaluated with metrics such as BLEU-4, ROUGE-L, CIDEr, and an LLM-based expert score.

Searching arXiv for TennisVL/TennisExpert and closely related tennis video understanding papers. TennisVL is a large-scale tennis video benchmark introduced for expert-level, analytical commentary and fine-grained tactical understanding in broadcast tennis video (Liu et al., 11 Mar 2026). It comprises 202 broadcast singles matches from Grand Slam tournaments played between 2019–2025, totaling 471.9 hours of untrimmed video at $1280\times720$ and $25$–$30$ FPS, and segmented into 40,523 rally-level clips with mean duration $7.68$ s. Unlike earlier commentary datasets oriented toward descriptive play-by-play narration, TennisVL emphasizes expert analytical commentary that captures tactical reasoning, player decisions, and match momentum. The benchmark is paired with TennisExpert, a multimodal framework built on Qwen3-VL-8B that combines video semantic parsing with hierarchical memory for commentary generation and related tennis-understanding tasks (Liu et al., 11 Mar 2026).

1. Dataset definition and scope

TennisVL is described as the first large-scale tennis video benchmark designed expressly for expert-level, analytical commentary and fine-grained tactical understanding (Liu et al., 11 Mar 2026). Its source material consists of 202 broadcast singles matches drawn from the Australian Open, Roland Garros, Wimbledon, and the US Open. Rally-level segmentation produces 40,523 clips, each corresponding to one uninterrupted point. Across the corpus, expert commentaries average 31.4 words per rally, the data cover 94 unique players across ATP and WTA, and the annotated shot inventory contains 162,503 total shots.

The dataset is notable not merely for scale but for its target output. Earlier tennis work often concentrated on ball localization, stroke recognition, or commentary retrieval in narrower settings. TrackNet, for example, addressed high-speed tennis-ball tracking from broadcast video using a heatmap-based detector rather than rally-level language understanding (Huang et al., 2019). TennisVid2Text aligned five Olympic singles matches comprising 710 “tennis-points” with professional text commentary and retrieved a single best sentence from a commentary corpus, but it operated in a much smaller and more formulaic setting (Sukhwani et al., 2015). TennisVL therefore marks a shift from domain-specific captioning toward analytical match understanding.

2. Annotation schema and corpus organization

Each TennisVL rally clip is annotated with five primary components: "commentary", "audio_transcript", "shot_sequence", "scoreboard", and "object_trajectories" (Liu et al., 11 Mar 2026). The commentary field is a concise, expert-level natural language analysis emphasizing tactics, momentum, and performance evaluation. The audio transcript is a WhisperX-aligned ASR transcription of the broadcast audio. The shot sequence is a structured list of fine-grained events $(c_i,\tau_i)$, where $c_i$ includes shot type, technique, direction, outcome, or bounce type. The scoreboard stores player A and B set/game/point scores together with server identity. The object-trajectory field provides 2D image coordinates of players and ball at each event timestamp, projected via homography into real-world court coordinates.

The benchmark is split at the match level to ensure zero overlap between training and test data: 182 matches yielding 35,687 clips for training and 20 matches yielding 4,836 clips for testing (Liu et al., 11 Mar 2026). Tournament coverage is approximately balanced in the sense that each of the four Grand Slams contributes roughly 20–30% of the clips.

This organization gives TennisVL a multi-view structure uncommon in earlier tennis resources. Prior work on tennis commentary aligned video points with commentary text and modeled phrase sequences, but did not combine rally-level video, ASR, event structure, scoreboard state, and homography-based object trajectories in a single benchmark (Sukhwani et al., 2015). A plausible implication is that TennisVL is designed not only for generation tasks but also for controlled studies of how symbolic match context interacts with visual evidence.

3. Supported tasks and evaluation protocol

TennisVL defines three benchmark tasks: commentary generation, tactical inference, and momentum analysis (Liu et al., 11 Mar 2026). Commentary generation requires production of expert-level text given video and structure. Tactical inference requires prediction of the next shot type or court positioning from the preceding shot sequence and positions. Momentum analysis targets shifts in rally momentum, including streaks of points or psychological turning points.

For commentary generation, the benchmark reports BLEU-4, METEOR, ROUGE-L, and CIDEr. The paper gives the standard forms of BLEU-4, ROUGE-L, and CIDEr. In particular,

$$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$

and

$$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$

CIDEr is defined through TF-IDF-weighted $n$-gram vector similarity between candidate and reference commentaries (Liu et al., 11 Mar 2026).

To supplement standard NLG metrics, TennisVL introduces a five-criterion LLM-based “Expert Score” ranging from $0$ to $25$0: $25$1 where each component lies in $25$2 and measures Accuracy, Coherence, Excitement, Professionalism, and Pacing (Liu et al., 11 Mar 2026). The introduction of this metric reflects the benchmark’s focus on tactical depth and factual accuracy rather than lexical overlap alone. This suggests that TennisVL treats analytical adequacy as a first-class evaluation target, not as a by-product of caption fluency.

4. Position within prior tennis video research

TennisVL emerges from a lineage of tennis-specific computer vision systems that addressed narrower subproblems. TrackNet developed a VGG-16–based encoder with a DeconvNet-style decoder to localize a high-speed, tiny tennis ball from broadcast video, operating on $25$3 inputs and achieving $25$4 precision, $25$5 recall, and $25$6 F1-measure on the 2017 Universiade setting when trained with additional videos (Huang et al., 2019). Its core problem was ball trajectory estimation rather than rally semantics or expert commentary.

TennisVid2Text addressed fine-grained textual description for lawn tennis by combining court and player detection, dense trajectory features, phrase-level $25$7-SVMs, MRF smoothing, and LSI-based retrieval over a 2,689-line Tennis-Text commentary corpus (Sukhwani et al., 2015). Its final BLEU-4 was $25$8, compared with $25$9 for the next-best baseline. However, its setting was small-scale and relied on retrieving a single human-written sentence, not on expert analytical modeling of match context across a large modern benchmark.

A more recent study on multimodal LLMs evaluated VideoLLaMA2-7B-Base on FineTennis, a dataset of 9,716 rally clips annotated at frame level with shot timing and 56 composite event classes (Teo, 24 Jun 2025). That work reported an overall single-shot classification accuracy of $30$0 and rally identification edit scores ranging from $30$1 to $30$2 under default and event-count-assisted prompting, while structured prompt enrichment with court corners and ball coordinates reached $30$3. The authors concluded that off-the-shelf VideoLLaMA2 was strong at textual reasoning but weak at extracting fine-grained visual cues needed to classify strokes and count events. In relation to TennisVL, this supports the view that expert tennis understanding benefits from explicit structured metadata rather than raw video alone.

A common misconception is to treat all tennis-commentary datasets as equivalent. The record given for TennisVL directly distinguishes it from earlier resources by marking it as analytic, whereas prior listed datasets either omit analytic commentary or focus on descriptive, non-analytic settings (Liu et al., 11 Mar 2026).

5. TennisExpert reference framework

The benchmark is introduced together with TennisExpert, a multimodal tennis understanding framework that combines a real-time video semantic parser with a memory-augmented MLLM based on Qwen3-VL-8B (Liu et al., 11 Mar 2026). For each rally $30$4, the semantic parser extracts metadata $30$5. The scoreboard component is

$30$6

obtained by prompt-guided OCR, where $30$7 denotes set/game/point scores and $30$8 denotes server identity. The event sequence is

$30$9

generated by a dense temporal detector with Edit Score $7.68$0. The object detections are

$7.68$1

which are tracked and homography-projected to court coordinates.

The memory mechanism is hierarchical. Short-term memory is a FIFO buffer over the previous $7.68$2 rallies,

$7.68$3

used to capture immediate momentum. Long-term memory is updated via

$7.68$4

where $7.68$5 consolidates per-player cumulative statistics such as aces, unforced errors, and first-serve percentage (Liu et al., 11 Mar 2026).

At generation time, TennisExpert encodes $7.68$6 into visual tokens $7.68$7, serializes $7.68$8 into symbolic tokens $7.68$9, and prompts Qwen3-VL-8B to generate commentary $(c_i,\tau_i)$0 by maximizing

$(c_i,\tau_i)$1

TennisExpert is therefore not identical to TennisVL: the former is the model family proposed for the latter benchmark. That distinction is central to interpreting reported results.

6. Empirical performance, ablations, and significance

On TennisVL commentary generation, TennisExpert outperforms reported zero-shot baselines and proprietary systems (Liu et al., 11 Mar 2026). Qwen3-VL-8B in zero-shot mode obtains BLEU-4 $(c_i,\tau_i)$2, ROUGE-L $(c_i,\tau_i)$3, CIDEr $(c_i,\tau_i)$4, and total Expert Score $(c_i,\tau_i)$5. Gemini-3-Pro zero-shot reaches BLEU-4 $(c_i,\tau_i)$6, ROUGE-L $(c_i,\tau_i)$7, CIDEr $(c_i,\tau_i)$8, and Expert Score $(c_i,\tau_i)$9. GPT-5.2 zero-shot records BLEU-4 $c_i$0, ROUGE-L $c_i$1, CIDEr $c_i$2, and Expert Score $c_i$3. TennisExpert achieves BLEU-4 $c_i$4, ROUGE-L $c_i$5, CIDEr $c_i$6, and Expert Score $c_i$7, with component scores Accuracy $c_i$8, Coherence $c_i$9, Excitement $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$0, Professionalism $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$1, and Pacing $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$2.

The ablation study isolates the role of structure and memory. With video only, the Qwen3-VL-8B backbone achieves CIDEr $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$3, Accuracy $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$4, Professionalism $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$5, and total score $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$6. Adding metadata $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$7 raises these to CIDEr $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$8, Accuracy $$\mathrm{BLEU}_4 = \mathrm{BP}\;\exp\Bigl(\tfrac{1}{4}\sum_{n=1}^{4}\log p_n\Bigr),$$9, Professionalism $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$0, and total $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$1. Adding short-term memory $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$2 produces CIDEr $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$3 and total $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$4. Adding both short- and long-term memory yields the full TennisExpert result of CIDEr $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$5 and total $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$6 (Liu et al., 11 Mar 2026).

These ablations directly counter a second common misconception: that expert tennis commentary can be recovered from visual frames alone. Within the reported setup, visual input without structured metadata or memory is markedly weaker than the full system. This aligns with the 2025 MLLM study showing that prompt-level fusion of structured outputs from classical vision modules, including court geometry and ball tracks, substantially improves sequence understanding in tennis video (Teo, 24 Jun 2025). A plausible implication is that TennisVL’s design intentionally rewards systems that integrate symbolic match state, event structure, and temporal context.

The benchmark’s broader significance lies in moving tennis video understanding from surface-level captioning toward professional-grade analysis (Liu et al., 11 Mar 2026). Its combination of rally video, ASR, structured events, scoreboard state, and object trajectories establishes a testbed for commentary generation, tactical inference, and momentum analysis under realistic broadcast conditions. The accompanying TennisExpert system further indicates that such analysis can be performed in real time, with reported latency below $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$7 s and approximately $$\mathrm{ROUGE\!-\!L} = \frac{(1+\beta^2)\,R_L\,P_L}{R_L + \beta^2P_L}, \quad R_L = \frac{\mathrm{LCS}(C,R)}{|R|},\; P_L = \frac{\mathrm{LCS}(C,R)}{|C|}.$$8 GB VRAM. Within the development arc from TrackNet’s ball tracking (Huang et al., 2019) through TennisVid2Text’s retrieval-based descriptions (Sukhwani et al., 2015) and recent MLLM rally-sequencing studies (Teo, 24 Jun 2025), TennisVL occupies the role of a benchmark for analytical, match-aware tennis understanding rather than isolated perception or narrow captioning.