---
title: 'TennisTV: Benchmark for Tennis Rally Analysis'
url: https://www.emergentmind.com/topics/tennistv
type: topic
---

# TennisTV: Benchmark for Tennis Rally Analysis

TennisTV is a benchmark for tennis video understanding introduced in the 2025 paper “TennisTV: Do Multimodal Large Language Models Understand Tennis Rallies?” [2509.15602]. It was designed to systematically evaluate multimodal large language models (MLLMs) on a domain in which rally clips are short yet information-dense, and in which general video-understanding competence does not reliably transfer. TennisTV models each rally as a temporal-ordered sequence of consecutive stroke events, covers 8 tasks at rally and stroke levels, includes 2,500 human-verified questions, and evaluates 16 representative MLLMs. Its central finding is that current MLLMs exhibit substantial shortcomings on tennis rally understanding, especially where fine-grained temporal grounding is required [2509.15602].

## 1. Research problem and benchmark scope

TennisTV addresses a specific failure mode of contemporary MLLMs: strong performance on general video benchmarks does not imply robust understanding of fast, high-frequency sports. The benchmark treats tennis as a temporally structured, multi-agent environment in which a rally must be parsed not merely as a scene, but as an ordered sequence of strokes, movements, directions, and outcomes. This design shifts evaluation away from coarse captioning and toward event-level reasoning [2509.15602].

The benchmark is described as the first and most comprehensive benchmark for tennis video understanding. Its emphasis is not on isolated image recognition, nor on scoreboard extraction, but on rally semantics across multiple granularities. This places it in a distinct niche within tennis video analysis: earlier systems concentrated on score-based indexing of broadcast tennis videos [1801.01430], fine-grained textual description generation from tennis video [1511.08522], or tennis-specific analytical commentary generation [2603.13397], whereas TennisTV is explicitly an evaluation framework for MLLM reasoning over rally structure [2509.15602].

A plausible implication is that TennisTV functions as an intermediate layer between low-level sports vision and high-level multimodal reasoning. Rather than replacing tracking, OCR, or commentary systems, it exposes whether MLLMs can convert temporally sparse video evidence into structured tennis understanding.

## 2. Rally representation and annotation pipeline

The benchmark formalizes a rally as a time-ordered sequence of stroke events. A stroke event is treated as the atomic unit and comprises movement initiation, shot action choice, directional execution, and the ball’s landing point [2509.15602]. This representation is consequential: it embeds tennis-specific temporal order directly into the benchmark rather than leaving event structure implicit.

TennisTV is built from F3Set and uses a three-stage annotation pipeline. The first stage performs aggregation and filtering: strokes are aggregated by rally, structured rally statistics are built, and samples are filtered for quality, including court visibility. The second stage performs motion extraction: lightweight detectors localize players on frame pairs, and movement direction is computed from positional shift exceeding a threshold $\delta$. The movement criterion is expressed as
$$
\Delta x = |x_2 - x_1|,\ \Delta y = |y_2 - y_1|,\ \text{movement detected if }\Delta x > \delta \text{ or }\Delta y > \delta.
$$
The third stage performs QA generation: GPT-4o and templates generate diverse questions from the structured data, and all 2,500 questions are manually validated [2509.15602].

The benchmark contains approximately 1,298 tennis rally videos and 2,500 human-verified questions [2509.15602]. The human-verification step is methodologically important because the benchmark depends on structured QA as the evaluation interface. This suggests that TennisTV is not merely an automatically generated question set; it is a curated testbed intended to support controlled comparative analysis across heterogeneous MLLMs.

## 3. Task taxonomy and evaluation targets

TennisTV partitions evaluation into stroke-level and rally-sequence tasks. This division reflects the fact that some errors arise from sub-event perception, while others arise from aggregation across an entire rally [2509.15602].

| Level | Task | Description |
|---|---|---|
| Stroke-level | AR | Recognize the type of stroke |
| Stroke-level | MG | Pinpoint direction/player movement per stroke |
| Stroke-level | HD | Identify the ball’s direction after a stroke |
| Stroke-level | HO | Determine what happened after a stroke |
| Rally-sequence | TI | Recognize prominent technique across rally |
| Rally-sequence | RC | Count the number of strokes in the rally |
| Rally-sequence | TP | Deduce technical tendency |
| Rally-sequence | PW | Determine who won the rally |

Most tasks use multiple-choice questions, while Point Winner uses true/false judgement statements [2509.15602]. The benchmark also defines aggregate metrics: AvgSE averages the four stroke-event tasks, AvgRS averages the four rally-sequence tasks, and Avg is the overall mean across all 8 tasks [2509.15602].

This taxonomy is more granular than earlier tennis-language systems. “TennisVid2Text” focused on rich commentary-like description generation from broadcast clips [1511.08522], and “TennisExpert” emphasized expert analytical commentary conditioned on structured tennis metadata [2603.13397]. TennisTV instead decomposes understanding into evaluable subproblems, allowing researchers to localize failure modes such as action recognition, movement grounding, temporal counting, or tactical preference inference [2509.15602].

## 4. Model coverage and empirical results

TennisTV evaluates 16 representative MLLMs: 3 closed-source models and 13 open-source models. The closed-source set comprises Gemini2.5 Pro, GPT-4.1, and GPT-4o. The open-source set includes both “non-thinking” and “thinking” models, including Qwen2.5VL-7B and VersaVid-R1 [2509.15602]. Closed-source models were evaluated at 1 FPS for budget reasons, while open-source models were evaluated with 8–128 frames depending on the setting [2509.15602].

The headline result is that all evaluated models remain only moderately above chance on this benchmark. The random baseline is 27.9 overall. Among closed-source systems, GPT-4.1 achieves the best overall score at 45.0, followed by GPT-4o at 44.3 and Gemini2.5 Pro at 43.2. Among open-source systems, VersaVid-R1 reaches 39.8 and Qwen2.5VL-7B reaches 39.4 [2509.15602].

| Model | Setting | Avg |
|---|---|---|
| Random | – | 27.9 |
| GPT-4.1 | 1 FPS | 45.0 |
| GPT-4o | 1 FPS | 44.3 |
| Gemini2.5 Pro | 1 FPS | 43.2 |
| VersaVid-R1 | 32 Frames | 39.8 |
| Qwen2.5VL-7B | 32 Frames | 39.4 |

Task-wise leadership is fragmented. Gemini2.5 Pro records the best Movement Grounding score at 41.7 and the best AvgSE at 42.0. GPT-4.1 leads Technique Preference at 52.0 and has the best overall average at 45.0. GPT-4o achieves the best Point Winner accuracy at 56.3 and the best AvgRS at 48.1. Among open-source models, VersaVid-R1 attains the best Action Recognition score at 52.3, the best Rally Count score at 27.0, and the best Technique Identification score at 47.8 [2509.15602].

A common misconception is that general-purpose MLLMs already understand tennis rallies adequately if given video input. TennisTV contradicts this view quantitatively: even the best models remain far from saturation, and the gap between overall performance and domain expectations is substantial [2509.15602].

## 5. Sampling density, reasoning, and temporal grounding

Two findings in TennisTV are especially methodologically significant. First, frame-sampling density should be tailored and balanced across tasks. Second, improving temporal grounding is essential for stronger reasoning [2509.15602]. These conclusions emerge from explicit ablation rather than conjecture.

For Qwen2.5VL-7B, a sampling-density study compares 1 FPS, 16 frames, 32 frames, and 64 frames. The best overall result is obtained at 32 frames, with AvgSE 39.3, AvgRS 39.5, and Avg 39.4. Performance for stroke-level tasks improves with denser sampling, but rally-level tasks saturate and can regress with additional redundancy: at 64 frames, AvgRS falls to 38.5 and overall Avg to 38.6 [2509.15602]. The benchmark therefore rejects the naive assumption that more frames monotonically improve performance.

TennisTV also analyzes reasoning-augmented open-source models against non-reasoning counterparts. On knowledge-light tasks such as Movement Grounding, Hit Direction, and Rally Count, reasoning-augmented models consistently outperform non-reasoning models, with the most pronounced improvements on Rally Count and Hit Direction. However, the gain on motion-oriented tasks such as Movement Grounding is more modest [2509.15602]. This pattern suggests that chain-of-thought-style reasoning assists aggregation and discrete inference, but does not solve the underlying perception problem of temporally localizing sub-second sports events.

The paper further reports strong positive correlations among Action Recognition, Technique Identification, and Technique Preference [2509.15602]. This implies that low-level action parsing, mid-level technique recognition, and higher-level tactical tendency inference are not independent competencies; they form a logical progression in which failures propagate upward.

## 6. Position within tennis video understanding research

TennisTV belongs to a broader lineage of domain-specific tennis video analysis, but its role is diagnostic rather than directly generative. “SmartTennisTV” proposed score based indexing of tennis videos through rally segmentation, score recognition, and refinement using the tennis scoring system [1801.01430]. “TennisVid2Text” generated semantically richer descriptions from lawn tennis broadcast videos by combining fine-grained action analysis with retrieval from a large human-written commentary corpus [1511.08522]. “TennisExpert” later introduced TennisVL and a memory-augmented multimodal system for analytical commentary generation, emphasizing tactical reasoning, player decisions, and match momentum [2603.13397].

Within MLLM-focused work, the benchmark complements a separate study on multimodal LLMs for tennis video analysis that found raw MLLMs lag domain-specific systems and improve substantially when structured player, ball, and court information is supplied as text prompts [2507.02904]. TennisTV makes a related but distinct contribution: it provides a standardized testbed showing where contemporary MLLMs fail when asked to reason directly over rally videos [2509.15602].

This suggests that future progress in tennis video intelligence will likely depend on hybridization. TennisTV indicates that benchmark performance is constrained by temporal grounding; related work indicates that structured detectors and domain metadata can materially improve MLLM outputs [2507.02904]. In that sense, TennisTV can be read as both a benchmark and an argument for tighter integration between sports-specific perception modules and multimodal reasoning layers.

## 7. Interpretation, limitations, and significance

The main significance of TennisTV is that it exposes a mismatch between broad video-language competence and domain-specific temporal sports understanding. Tennis rallies are short, rapidly alternating, and tactically dense. TennisTV operationalizes these properties through stroke-level and rally-level tasks, then shows that contemporary MLLMs remain only moderately above chance, with persistent weaknesses in temporal grounding [2509.15602].

Its findings also temper two simplistic narratives. The first is that larger or more general MLLMs will automatically solve sports understanding. TennisTV reports substantial shortcomings even for strong closed-source systems [2509.15602]. The second is that increased sampling or added reasoning alone is sufficient. The reported ablations show instead that sampling must be task-dependent and that reasoning gains remain limited when temporal localization is weak [2509.15602].

For the research community, TennisTV provides a benchmark that disentangles perception, temporal aggregation, and tactical inference in a single domain. For sports-video research more broadly, a plausible implication is that its design principles may generalize to other high-temporal-density settings in which events are brief, sequential, and semantically interdependent. Within tennis specifically, it establishes a quantitative yardstick for evaluating whether future MLLMs genuinely understand rallies rather than merely recognize tennis as a scene [2509.15602].

Source: https://www.emergentmind.com/topics/tennistv