---
title: 'TennisExpert: Multimodal Tennis Video Analysis'
url: https://www.emergentmind.com/topics/tennisexpert
type: topic
---

# TennisExpert: Multimodal Tennis Video Analysis

TennisExpert denotes domain-specific systems for analytical understanding of tennis video. In an earlier usage, it referred to a retrieval-based commentary generator for broadcast lawn-tennis points; in the later formulation, it denotes a multimodal tennis understanding framework that couples the TennisVL benchmark with a video semantic parser and hierarchical memory built on Qwen3-VL-8B, with the stated goal of producing commentary that captures tactical reasoning, player decisions, and match momentum rather than descriptive play-by-play narration [1511.08522][2603.13397].

## 1. Conceptual scope and problem definition

The 2026 formulation of TennisExpert is motivated by two stated bottlenecks in automatic tennis understanding: the lack of large-scale benchmarks with fine-grained annotations and expert-level commentary, and the difficulty of building accurate yet efficient multimodal systems suitable for real-time deployment [2603.13397]. In this setting, expert tennis commentary is treated as a qualitatively distinct target. It is delivered between points, and it is expected to explain tactical intent, momentum shifts, pressure patterns, and match-level context.

Formally, TennisExpert operates over a match sequence $\mathcal{S}=\{V_1,\dots,V_N\}$ of rally clips. For the $t$-th rally, the system transforms raw video into structured metadata $M_t$, maintains a memory state $\mathcal{H}_t$ summarizing prior rallies, and generates commentary through
$$
C_t = \text{MLLM}(V_t, M_t, \mathcal{H}_t).
$$
This design replaces long raw-video histories with compact symbolic state, so that immediate rally evidence and accumulated match history can both condition generation [2603.13397].

The 2015 system bearing the same name addressed a narrower but closely related problem: given a broadcast video segment of a tennis point, it produced a commentary sentence by combining court and player analysis, action phrase recognition, temporal smoothing with an MRF, and retrieval from a corpus of human-written tennis commentary [1511.08522]. The later system therefore preserves the emphasis on expert-like tennis description while shifting from retrieval over point-level cues to structured multimodal generation over rally-level semantics and memory.

## 2. TennisVL benchmark and annotation pipeline

TennisVL is the principal dataset contribution associated with TennisExpert. It contains 202 full broadcast singles matches from 2019–2025, totaling 471.9 hours of video and 40,523 rally-level clips [2603.13397]. The matches are high-resolution broadcasts at 1280×720 and 25–30 FPS, spanning the Australian Open, Roland Garros, Wimbledon, and the US Open. The dataset covers 94 unique players and 162,503 total shots; the average clip is 7.68 seconds long and the average reference commentary is 31.42 words.

Each sample is organized as a rally rather than a continuous segment, reflecting the observation that tennis commentary is naturally produced after each rally during the pause before the next point. Each JSON entry stores match metadata, scoreboard state, aligned audio transcript, structured shot-by-shot sequence, and final commentary [2603.13397]. Construction is semi-automated. Match videos are collected from YouTube, rallies are segmented using a lightweight audio model that detects racquet-ball impacts, and clips are filtered to retain standard broadcast top-down court views with visible scoreboards. WhisperX provides the transcript, TennisAbstract supplies shot-by-shot descriptions, and Gemini 3 Pro is prompted to synthesize expert commentary from tournament information, score state, shot sequence, and aligned transcript. Verification then proceeds through an automated sanity check on critical entities and a human review of 2,000 sampled pairs, yielding an acceptance rate above 95%.

The benchmark is split by match rather than by clip: 182 matches, corresponding to 35,687 clips, are used for training, and 20 disjoint matches, corresponding to 4,836 clips, are used for testing [2603.13397]. This avoids leakage from within-match temporal dependence.

TennisVL is explicitly positioned against earlier tennis-video resources that were smaller, sparsely labeled, or centered on descriptive narration. A related benchmark, TennisTV, also targets rally-level and stroke-level understanding, but its purpose is diagnostic: it evaluates whether current multimodal large language models understand tennis rallies, and reports that even the strongest evaluated systems remain only modestly above random on many tasks [2509.15602]. This suggests that TennisExpert addresses a setting in which benchmark construction and model design are both constrained by unusually dense short-horizon temporal structure.

## 3. Semantic parsing and hierarchical memory

The TennisExpert architecture is modular. Its core idea is to parse each rally into structured symbolic state and then augment generation with hierarchical memory, rather than relying on raw-video context growth [2603.13397]. The semantic parser extracts
$$
M_t=\{s_t,e_t,o_t\},
$$
where $s_t$ is scoreboard state, $e_t$ is the event sequence, and $o_t$ is object and court geometry.

The scoreboard state is represented as
$$
s_t=(\rho_t^{(A)},\rho_t^{(B)}, v_t),
$$
where each player’s score includes set, game, and point values, and $v_t$ denotes the server. Because tennis scoreboards vary across tournaments, the system uses prompt-guided OCR/MLLM extraction with tournament-specific templates [2603.13397].

The event module defines fine-grained events as
$$
e_t=\{(c_i,\tau_i)\}_{i=1}^{N_e},
$$
with $c_i$ denoting a detailed event class and $\tau_i$ its timestamp. The taxonomy includes hit events with shot types, techniques, directions, and outcomes, together with ball-bounce events, and extends the F\(^3\)Set taxonomy with near- and far-court bounce categories. The detector combines local spatiotemporal features with a GRU and reaches an Edit Score of 81.2 on its test set [2603.13397].

The object module tracks players, the ball, and court keypoints. For key event timestamps $\tau_i$, it samples
$$
o_t^{(\tau_i)}=\{\mathbf{b}_t^{(A)}, \mathbf{b}_t^{(B)}, \mathbf{b}_t^{(\text{ball})}\},
$$
where each $\mathbf{b}\in \mathbb{R}^2$ is an image-space center. Court corners are used to estimate a homography so that player positions and bounce locations can be projected into real court coordinates [2603.13397]. This supports geometry-aware reasoning about shot placement and court coverage.

Memory is divided into short-term and long-term components. Short-term memory is a FIFO buffer
$$
\mathcal{S}_t=\{(V_i,M_i,C_i)\mid i=t-K,\dots,t-1\},
$$
with implementation choice $K=4$ [2603.13397]. Long-term memory is a persistent match summary,
$$
\mathcal{L}_t=\{\Omega_t^{(P_1)},\Omega_t^{(P_2)}\},
$$
updated deterministically by
$$
\mathcal{L}_t = \Phi(\mathcal{L}_{t-1}, M_{t-K}),
$$
when an old rally leaves short-term memory. The cumulative statistics include aces, double faults, first-serve percentage, return points won, winners, and unforced errors [2603.13397]. The paper characterizes this as symbolic compression of history for reasoning over hours of match dynamics without overflowing context.

## 4. Multimodal generation, training procedure, and deployment profile

After parsing, the structured metadata $M_t$ and memory state $\mathcal{H}_t$ are serialized into textual tokens $\mathbf{Z}_s^{(t)}$, while the rally video is encoded into visual tokens $\mathbf{Z}_v^{(t)}$ by Qwen3-VL’s vision encoder [2603.13397]. Commentary generation is modeled autoregressively as
$$
\log p(C_t \mid V_t, M_t, \mathcal{H}_t) = \sum_{j=1}^{L} \log p_{\theta}(w_j \mid w_{<j}, \mathbf{Z}_v^{(t)}, \mathbf{Z}_s^{(t)}),
$$
where $C_t=\{w_1,\dots,w_L\}$.

The implementation uses Qwen3-VL-8B-Instruct. Fine-tuning affects only the language model; the vision encoder and multimodal projector are frozen [2603.13397]. Training uses supervised fine-tuning for 3 epochs, learning rate $1\times10^{-5}$, cosine scheduling, 10% warmup, BF16 precision, batch size 1 per device with 2 gradient accumulation steps, on 4 NVIDIA H200 GPUs. Video clips are pre-encoded once per step and cached across overlapping windows, and loss is computed independently per window with masking so that each token is supervised exactly once. The training recipe also includes a distributed stability mechanism that equalizes sliding-window counts across ranks by subsampling to the global minimum and broadcasting a shared random seed [2603.13397].

Inference is organized for online use. As rallies arrive, the semantic parser extracts $M_t$, memories are updated, and the current rally is generated conditioned on structured metadata and history; afterwards, $(V_t,M_t,C_t)$ is inserted into short-term memory for the next point [2603.13397]. The parser runs at up to 40 FPS on a single GPU. Relative to dense raw-video baselines whose context grows as $\mathcal{O}(T)$, TennisExpert keeps token complexity roughly constant, around 14k tokens, uses about 20 GB VRAM, and achieves less than 2 seconds latency [2603.13397]. This places efficiency alongside analytical quality as a central design criterion.

## 5. Evaluation methodology, empirical performance, and ablations

Evaluation combines standard captioning metrics—BLEU-4, METEOR, ROUGE-L, and CIDEr—with an expert LLM score from 0 to 100 [2603.13397]. The latter is decomposed into five equally weighted categories: Accuracy, Coherence, Excitement, Professionalism, and Pacing. Accuracy measures factual alignment to metadata; Coherence measures narrative flow; Excitement measures tone fit; Professionalism measures tactical insight and domain terminology; Pacing measures length and detail relative to rally complexity.

The reported quantitative results place TennisExpert above both open-weight and proprietary baselines on this evaluation. The strongest proprietary baseline is Gemini-3-Pro at 59.89 total, followed by GPT-5.2 at 56.59, Qwen3.5-Plus at 52.55, Gemini-3-Flash at 52.50, and Claude-4.5-Sonnet at 50.75 [2603.13397]. TennisExpert reaches 88.05 total, with category scores of 15.79 Accuracy, 19.29 Coherence, 16.84 Excitement, 16.73 Professionalism, and 19.40 Pacing. On traditional metrics, it also reports CIDEr 43.71, compared with 10.11 for the strongest baseline.

The ablation studies isolate the contribution of structured metadata and memory. In zero-shot mode with Qwen3-VL-8B, raw video alone yields CIDEr 3.89 and Total 37.68; adding structured metadata $V_t + M_t$ raises Total to 61.53 [2603.13397]. After supervised fine-tuning, $V_t$ alone reaches 42.74, while $V_t + M_t$ reaches 73.74. Adding short-term memory increases CIDEr from 24.15 to 41.92 and Total from 73.74 to 85.97. Adding long-term memory produces the final best result, 43.71 CIDEr and 88.05 Total, with an additional +1.79 CIDEr and +2.08 Total over short-term memory alone [2603.13397]. The paper interprets these gains as evidence that both recent rally chains and cumulative match history are important for expert commentary.

Qualitative examples emphasize that the model does more than name shots. It explains point construction and tactical intent, references score states such as “down 0–2” or “game point at deuce,” and uses domain-specific terms including “swinging volley” and “dictating geometry” [2603.13397]. This suggests that the system’s target genre is not generic captioning but analyst-style post-rally interpretation.

## 6. Historical lineage, neighboring tasks, and stated limitations

The earlier TennisExpert of 2015 already established several themes that remain visible in the later framework. It treated the tennis point as the unit of analysis, exploited the upper/lower player structure of broadcast tennis, learned 76 tennis-specific action phrases, smoothed phrase predictions with an MRF, and retrieved expert-written commentary by semantic matching with LSI rather than generating text from scratch [1511.08522]. Its evaluation reported BLEU 0.235 with LSI, compared with 0.198 for naive lexical matching, and human scores of 3.21/5 for semantics and 3.9/5 for structure and readability [1511.08522]. The 2026 framework generalizes this agenda to rally-level multimodal generation with explicit score, event, geometry, and match-memory state.

The broader research landscape around TennisExpert includes several adjacent tasks. TennisTV formalizes rally-level and stroke-level benchmark evaluation for multimodal large language models and reports that GPT-4.1 reaches 45.0 average, while the random baseline is 27.9, leading to the conclusion that current models do not yet reliably understand tennis rallies [2509.15602]. A pose-based biomechanics pipeline built from MediaPipe Pose and a compact transformer reports 83.7% stroke-type accuracy under a random 80/20 split, but much weaker transfer for shot direction and posture quality, indicating that stroke recognition is easier than tactical intention or nuanced form assessment [2606.15992]. This suggests that TennisExpert occupies a higher semantic layer: it depends on event parsing and geometry, but its principal output is match analysis rather than stroke classification alone.

The paper is explicit about limitations. Because the system depends on a multi-stage parser, errors in OCR, event spotting, or detection and tracking can propagate into commentary [2603.13397]. The long-term memory is symbolic and cumulative, which may oversimplify nuanced game patterns or miss subtler psychological and strategic developments. Commentary quality still depends on a prompt- and LLM-based annotation pipeline, with some reliance on Gemini-generated references and on the correctness of automatic sanity checks. Raw videos are not redistributed for copyright reasons; the benchmark provides YouTube URLs with annotations instead [2603.13397].

Within these constraints, TennisExpert can be understood as a structured and memory-augmented approach to expert-level tennis video understanding. Its distinguishing properties are the combination of rally-level benchmark design, explicit semantic parsing of score and event state, hierarchical memory for momentum and match context, and a deployment profile intended for online use [2603.13397].

Source: https://www.emergentmind.com/topics/tennisexpert