TennisExpert: Multimodal Tennis Video Analysis

• TennisExpert is a domain-specific framework that generates expert commentary by transforming rally videos into structured metadata and match memory.
• It leverages the TennisVL benchmark and a multimodal parser to capture tactical reasoning, player decisions, and match momentum from tennis broadcasts.
• Its hierarchical memory design compresses both short- and long-term match history, enabling efficient real-time deployment with significant performance gains.

TennisExpert denotes domain-specific systems for analytical understanding of tennis video. In an earlier usage, it referred to a retrieval-based commentary generator for broadcast lawn-tennis points; in the later formulation, it denotes a multimodal tennis understanding framework that couples the TennisVL benchmark with a video semantic parser and hierarchical memory built on Qwen3-VL-8B, with the stated goal of producing commentary that captures tactical reasoning, player decisions, and match momentum rather than descriptive play-by-play narration (Sukhwani et al., 2015, Liu et al., 11 Mar 2026).

1. Conceptual scope and problem definition

The 2026 formulation of TennisExpert is motivated by two stated bottlenecks in automatic tennis understanding: the lack of large-scale benchmarks with fine-grained annotations and expert-level commentary, and the difficulty of building accurate yet efficient multimodal systems suitable for real-time deployment (Liu et al., 11 Mar 2026). In this setting, expert tennis commentary is treated as a qualitatively distinct target. It is delivered between points, and it is expected to explain tactical intent, momentum shifts, pressure patterns, and match-level context.

Formally, TennisExpert operates over a match sequence $\mathcal{S}=\{V_1,\dots,V_N\}$ of rally clips. For the $t$-th rally, the system transforms raw video into structured metadata $M_t$, maintains a memory state $\mathcal{H}_t$ summarizing prior rallies, and generates commentary through

$C_t = \text{MLLM}(V_t, M_t, \mathcal{H}_t).$

This design replaces long raw-video histories with compact symbolic state, so that immediate rally evidence and accumulated match history can both condition generation (Liu et al., 11 Mar 2026).

The 2015 system bearing the same name addressed a narrower but closely related problem: given a broadcast video segment of a tennis point, it produced a commentary sentence by combining court and player analysis, action phrase recognition, temporal smoothing with an MRF, and retrieval from a corpus of human-written tennis commentary (Sukhwani et al., 2015). The later system therefore preserves the emphasis on expert-like tennis description while shifting from retrieval over point-level cues to structured multimodal generation over rally-level semantics and memory.

2. TennisVL benchmark and annotation pipeline

TennisVL is the principal dataset contribution associated with TennisExpert. It contains 202 full broadcast singles matches from 2019–2025, totaling 471.9 hours of video and 40,523 rally-level clips (Liu et al., 11 Mar 2026). The matches are high-resolution broadcasts at 1280×720 and 25–30 FPS, spanning the Australian Open, Roland Garros, Wimbledon, and the US Open. The dataset covers 94 unique players and 162,503 total shots; the average clip is 7.68 seconds long and the average reference commentary is 31.42 words.

Each sample is organized as a rally rather than a continuous segment, reflecting the observation that tennis commentary is naturally produced after each rally during the pause before the next point. Each JSON entry stores match metadata, scoreboard state, aligned audio transcript, structured shot-by-shot sequence, and final commentary (Liu et al., 11 Mar 2026). Construction is semi-automated. Match videos are collected from YouTube, rallies are segmented using a lightweight audio model that detects racquet-ball impacts, and clips are filtered to retain standard broadcast top-down court views with visible scoreboards. WhisperX provides the transcript, TennisAbstract supplies shot-by-shot descriptions, and Gemini 3 Pro is prompted to synthesize expert commentary from tournament information, score state, shot sequence, and aligned transcript. Verification then proceeds through an automated sanity check on critical entities and a human review of 2,000 sampled pairs, yielding an acceptance rate above 95%.

The benchmark is split by match rather than by clip: 182 matches, corresponding to 35,687 clips, are used for training, and 20 disjoint matches, corresponding to 4,836 clips, are used for testing (Liu et al., 11 Mar 2026). This avoids leakage from within-match temporal dependence.

TennisVL is explicitly positioned against earlier tennis-video resources that were smaller, sparsely labeled, or centered on descriptive narration. A related benchmark, TennisTV, also targets rally-level and stroke-level understanding, but its purpose is diagnostic: it evaluates whether current multimodal LLMs understand tennis rallies, and reports that even the strongest evaluated systems remain only modestly above random on many tasks (Bao et al., 19 Sep 2025). This suggests that TennisExpert addresses a setting in which benchmark construction and model design are both constrained by unusually dense short-horizon temporal structure.

3. Semantic parsing and hierarchical memory

The TennisExpert architecture is modular. Its core idea is to parse each rally into structured symbolic state and then augment generation with hierarchical memory, rather than relying on raw-video context growth (Liu et al., 11 Mar 2026). The semantic parser extracts

$M_t=\{s_t,e_t,o_t\},$

where $s_t$ is scoreboard state, $e_t$ is the event sequence, and $o_t$ is object and court geometry.

The scoreboard state is represented as

$s_t=(\rho_t^{(A)},\rho_t^{(B)}, v_t),$

where each player’s score includes set, game, and point values, and $t$0 denotes the server. Because tennis scoreboards vary across tournaments, the system uses prompt-guided OCR/MLLM extraction with tournament-specific templates (Liu et al., 11 Mar 2026).

The event module defines fine-grained events as

$t$1

with $t$2 denoting a detailed event class and $t$3 its timestamp. The taxonomy includes hit events with shot types, techniques, directions, and outcomes, together with ball-bounce events, and extends the F$t$4Set taxonomy with near- and far-court bounce categories. The detector combines local spatiotemporal features with a GRU and reaches an Edit Score of 81.2 on its test set (Liu et al., 11 Mar 2026).

The object module tracks players, the ball, and court keypoints. For key event timestamps $t$5, it samples

$t$6

where each $t$7 is an image-space center. Court corners are used to estimate a homography so that player positions and bounce locations can be projected into real court coordinates (Liu et al., 11 Mar 2026). This supports geometry-aware reasoning about shot placement and court coverage.

Memory is divided into short-term and long-term components. Short-term memory is a FIFO buffer

$t$8

with implementation choice $t$9 (Liu et al., 11 Mar 2026). Long-term memory is a persistent match summary,

$M_t$0

updated deterministically by

$M_t$1

when an old rally leaves short-term memory. The cumulative statistics include aces, double faults, first-serve percentage, return points won, winners, and unforced errors (Liu et al., 11 Mar 2026). The paper characterizes this as symbolic compression of history for reasoning over hours of match dynamics without overflowing context.

4. Multimodal generation, training procedure, and deployment profile

After parsing, the structured metadata $M_t$2 and memory state $M_t$3 are serialized into textual tokens $M_t$4, while the rally video is encoded into visual tokens $M_t$5 by Qwen3-VL’s vision encoder (Liu et al., 11 Mar 2026). Commentary generation is modeled autoregressively as

$M_t$6

where $M_t$7.

The implementation uses Qwen3-VL-8B-Instruct. Fine-tuning affects only the LLM; the vision encoder and multimodal projector are frozen (Liu et al., 11 Mar 2026). Training uses supervised fine-tuning for 3 epochs, learning rate $M_t$8, cosine scheduling, 10% warmup, BF16 precision, batch size 1 per device with 2 gradient accumulation steps, on 4 NVIDIA H200 GPUs. Video clips are pre-encoded once per step and cached across overlapping windows, and loss is computed independently per window with masking so that each token is supervised exactly once. The training recipe also includes a distributed stability mechanism that equalizes sliding-window counts across ranks by subsampling to the global minimum and broadcasting a shared random seed (Liu et al., 11 Mar 2026).

Inference is organized for online use. As rallies arrive, the semantic parser extracts $M_t$9, memories are updated, and the current rally is generated conditioned on structured metadata and history; afterwards, $\mathcal{H}_t$0 is inserted into short-term memory for the next point (Liu et al., 11 Mar 2026). The parser runs at up to 40 FPS on a single GPU. Relative to dense raw-video baselines whose context grows as $\mathcal{H}_t$1, TennisExpert keeps token complexity roughly constant, around 14k tokens, uses about 20 GB VRAM, and achieves less than 2 seconds latency (Liu et al., 11 Mar 2026). This places efficiency alongside analytical quality as a central design criterion.

5. Evaluation methodology, empirical performance, and ablations

Evaluation combines standard captioning metrics—BLEU-4, METEOR, ROUGE-L, and CIDEr—with an expert LLM score from 0 to 100 (Liu et al., 11 Mar 2026). The latter is decomposed into five equally weighted categories: Accuracy, Coherence, Excitement, Professionalism, and Pacing. Accuracy measures factual alignment to metadata; Coherence measures narrative flow; Excitement measures tone fit; Professionalism measures tactical insight and domain terminology; Pacing measures length and detail relative to rally complexity.

The reported quantitative results place TennisExpert above both open-weight and proprietary baselines on this evaluation. The strongest proprietary baseline is Gemini-3-Pro at 59.89 total, followed by GPT-5.2 at 56.59, Qwen3.5-Plus at 52.55, Gemini-3-Flash at 52.50, and Claude-4.5-Sonnet at 50.75 (Liu et al., 11 Mar 2026). TennisExpert reaches 88.05 total, with category scores of 15.79 Accuracy, 19.29 Coherence, 16.84 Excitement, 16.73 Professionalism, and 19.40 Pacing. On traditional metrics, it also reports CIDEr 43.71, compared with 10.11 for the strongest baseline.

The ablation studies isolate the contribution of structured metadata and memory. In zero-shot mode with Qwen3-VL-8B, raw video alone yields CIDEr 3.89 and Total 37.68; adding structured metadata $\mathcal{H}_t$2 raises Total to 61.53 (Liu et al., 11 Mar 2026). After supervised fine-tuning, $\mathcal{H}_t$3 alone reaches 42.74, while $\mathcal{H}_t$4 reaches 73.74. Adding short-term memory increases CIDEr from 24.15 to 41.92 and Total from 73.74 to 85.97. Adding long-term memory produces the final best result, 43.71 CIDEr and 88.05 Total, with an additional +1.79 CIDEr and +2.08 Total over short-term memory alone (Liu et al., 11 Mar 2026). The paper interprets these gains as evidence that both recent rally chains and cumulative match history are important for expert commentary.

Qualitative examples emphasize that the model does more than name shots. It explains point construction and tactical intent, references score states such as “down 0–2” or “game point at deuce,” and uses domain-specific terms including “swinging volley” and “dictating geometry” (Liu et al., 11 Mar 2026). This suggests that the system’s target genre is not generic captioning but analyst-style post-rally interpretation.

6. Historical lineage, neighboring tasks, and stated limitations

The earlier TennisExpert of 2015 already established several themes that remain visible in the later framework. It treated the tennis point as the unit of analysis, exploited the upper/lower player structure of broadcast tennis, learned 76 tennis-specific action phrases, smoothed phrase predictions with an MRF, and retrieved expert-written commentary by semantic matching with LSI rather than generating text from scratch (Sukhwani et al., 2015). Its evaluation reported BLEU 0.235 with LSI, compared with 0.198 for naive lexical matching, and human scores of 3.21/5 for semantics and 3.9/5 for structure and readability (Sukhwani et al., 2015). The 2026 framework generalizes this agenda to rally-level multimodal generation with explicit score, event, geometry, and match-memory state.

The broader research landscape around TennisExpert includes several adjacent tasks. TennisTV formalizes rally-level and stroke-level benchmark evaluation for multimodal LLMs and reports that GPT-4.1 reaches 45.0 average, while the random baseline is 27.9, leading to the conclusion that current models do not yet reliably understand tennis rallies (Bao et al., 19 Sep 2025). A pose-based biomechanics pipeline built from MediaPipe Pose and a compact transformer reports 83.7% stroke-type accuracy under a random 80/20 split, but much weaker transfer for shot direction and posture quality, indicating that stroke recognition is easier than tactical intention or nuanced form assessment (Hazarika, 14 Jun 2026). This suggests that TennisExpert occupies a higher semantic layer: it depends on event parsing and geometry, but its principal output is match analysis rather than stroke classification alone.

The paper is explicit about limitations. Because the system depends on a multi-stage parser, errors in OCR, event spotting, or detection and tracking can propagate into commentary (Liu et al., 11 Mar 2026). The long-term memory is symbolic and cumulative, which may oversimplify nuanced game patterns or miss subtler psychological and strategic developments. Commentary quality still depends on a prompt- and LLM-based annotation pipeline, with some reliance on Gemini-generated references and on the correctness of automatic sanity checks. Raw videos are not redistributed for copyright reasons; the benchmark provides YouTube URLs with annotations instead (Liu et al., 11 Mar 2026).

Within these constraints, TennisExpert can be understood as a structured and memory-augmented approach to expert-level tennis video understanding. Its distinguishing properties are the combination of rally-level benchmark design, explicit semantic parsing of score and event state, hierarchical memory for momentum and match context, and a deployment profile intended for online use (Liu et al., 11 Mar 2026).