TennisTV: Benchmark for Tennis Rally Analysis

• TennisTV is a benchmark defining tennis rallies as temporally ordered stroke sequences, enabling fine-grained evaluation of MLLM understanding.
• It employs a three-stage annotation pipeline and tailored frame-sampling, highlighting the importance of temporal grounding for accurate event-level inference.
• Empirical results reveal that even top-performing models fall short on precise rally analysis, underscoring the need for hybrid perception and reasoning approaches.

TennisTV is a benchmark for tennis video understanding introduced in the 2025 paper “TennisTV: Do Multimodal LLMs Understand Tennis Rallies?” (Bao et al., 19 Sep 2025). It was designed to systematically evaluate multimodal LLMs (MLLMs) on a domain in which rally clips are short yet information-dense, and in which general video-understanding competence does not reliably transfer. TennisTV models each rally as a temporal-ordered sequence of consecutive stroke events, covers 8 tasks at rally and stroke levels, includes 2,500 human-verified questions, and evaluates 16 representative MLLMs. Its central finding is that current MLLMs exhibit substantial shortcomings on tennis rally understanding, especially where fine-grained temporal grounding is required (Bao et al., 19 Sep 2025).

1. Research problem and benchmark scope

TennisTV addresses a specific failure mode of contemporary MLLMs: strong performance on general video benchmarks does not imply robust understanding of fast, high-frequency sports. The benchmark treats tennis as a temporally structured, multi-agent environment in which a rally must be parsed not merely as a scene, but as an ordered sequence of strokes, movements, directions, and outcomes. This design shifts evaluation away from coarse captioning and toward event-level reasoning (Bao et al., 19 Sep 2025).

The benchmark is described as the first and most comprehensive benchmark for tennis video understanding. Its emphasis is not on isolated image recognition, nor on scoreboard extraction, but on rally semantics across multiple granularities. This places it in a distinct niche within tennis video analysis: earlier systems concentrated on score-based indexing of broadcast tennis videos (Ghosh et al., 2018), fine-grained textual description generation from tennis video (Sukhwani et al., 2015), or tennis-specific analytical commentary generation (Liu et al., 11 Mar 2026), whereas TennisTV is explicitly an evaluation framework for MLLM reasoning over rally structure (Bao et al., 19 Sep 2025).

A plausible implication is that TennisTV functions as an intermediate layer between low-level sports vision and high-level multimodal reasoning. Rather than replacing tracking, OCR, or commentary systems, it exposes whether MLLMs can convert temporally sparse video evidence into structured tennis understanding.

2. Rally representation and annotation pipeline

The benchmark formalizes a rally as a time-ordered sequence of stroke events. A stroke event is treated as the atomic unit and comprises movement initiation, shot action choice, directional execution, and the ball’s landing point (Bao et al., 19 Sep 2025). This representation is consequential: it embeds tennis-specific temporal order directly into the benchmark rather than leaving event structure implicit.

TennisTV is built from F3Set and uses a three-stage annotation pipeline. The first stage performs aggregation and filtering: strokes are aggregated by rally, structured rally statistics are built, and samples are filtered for quality, including court visibility. The second stage performs motion extraction: lightweight detectors localize players on frame pairs, and movement direction is computed from positional shift exceeding a threshold $\delta$. The movement criterion is expressed as

$$\Delta x = |x_2 - x_1|,\ \Delta y = |y_2 - y_1|,\ \text{movement detected if }\Delta x > \delta \text{ or }\Delta y > \delta.$$

The third stage performs QA generation: GPT-4o and templates generate diverse questions from the structured data, and all 2,500 questions are manually validated (Bao et al., 19 Sep 2025).

The benchmark contains approximately 1,298 tennis rally videos and 2,500 human-verified questions (Bao et al., 19 Sep 2025). The human-verification step is methodologically important because the benchmark depends on structured QA as the evaluation interface. This suggests that TennisTV is not merely an automatically generated question set; it is a curated testbed intended to support controlled comparative analysis across heterogeneous MLLMs.

3. Task taxonomy and evaluation targets

TennisTV partitions evaluation into stroke-level and rally-sequence tasks. This division reflects the fact that some errors arise from sub-event perception, while others arise from aggregation across an entire rally (Bao et al., 19 Sep 2025).

Level	Task	Description
Stroke-level	AR	Recognize the type of stroke
Stroke-level	MG	Pinpoint direction/player movement per stroke
Stroke-level	HD	Identify the ball’s direction after a stroke
Stroke-level	HO	Determine what happened after a stroke
Rally-sequence	TI	Recognize prominent technique across rally
Rally-sequence	RC	Count the number of strokes in the rally
Rally-sequence	TP	Deduce technical tendency
Rally-sequence	PW	Determine who won the rally

Most tasks use multiple-choice questions, while Point Winner uses true/false judgement statements (Bao et al., 19 Sep 2025). The benchmark also defines aggregate metrics: AvgSE averages the four stroke-event tasks, AvgRS averages the four rally-sequence tasks, and Avg is the overall mean across all 8 tasks (Bao et al., 19 Sep 2025).

This taxonomy is more granular than earlier tennis-language systems. “TennisVid2Text” focused on rich commentary-like description generation from broadcast clips (Sukhwani et al., 2015), and “TennisExpert” emphasized expert analytical commentary conditioned on structured tennis metadata (Liu et al., 11 Mar 2026). TennisTV instead decomposes understanding into evaluable subproblems, allowing researchers to localize failure modes such as action recognition, movement grounding, temporal counting, or tactical preference inference (Bao et al., 19 Sep 2025).

4. Model coverage and empirical results

TennisTV evaluates 16 representative MLLMs: 3 closed-source models and 13 open-source models. The closed-source set comprises Gemini2.5 Pro, GPT-4.1, and GPT-4o. The open-source set includes both “non-thinking” and “thinking” models, including Qwen2.5VL-7B and VersaVid-R1 (Bao et al., 19 Sep 2025). Closed-source models were evaluated at 1 FPS for budget reasons, while open-source models were evaluated with 8–128 frames depending on the setting (Bao et al., 19 Sep 2025).

The headline result is that all evaluated models remain only moderately above chance on this benchmark. The random baseline is 27.9 overall. Among closed-source systems, GPT-4.1 achieves the best overall score at 45.0, followed by GPT-4o at 44.3 and Gemini2.5 Pro at 43.2. Among open-source systems, VersaVid-R1 reaches 39.8 and Qwen2.5VL-7B reaches 39.4 (Bao et al., 19 Sep 2025).

Model	Setting	Avg
Random	–	27.9
GPT-4.1	1 FPS	45.0
GPT-4o	1 FPS	44.3
Gemini2.5 Pro	1 FPS	43.2
VersaVid-R1	32 Frames	39.8
Qwen2.5VL-7B	32 Frames	39.4

Task-wise leadership is fragmented. Gemini2.5 Pro records the best Movement Grounding score at 41.7 and the best AvgSE at 42.0. GPT-4.1 leads Technique Preference at 52.0 and has the best overall average at 45.0. GPT-4o achieves the best Point Winner accuracy at 56.3 and the best AvgRS at 48.1. Among open-source models, VersaVid-R1 attains the best Action Recognition score at 52.3, the best Rally Count score at 27.0, and the best Technique Identification score at 47.8 (Bao et al., 19 Sep 2025).

A common misconception is that general-purpose MLLMs already understand tennis rallies adequately if given video input. TennisTV contradicts this view quantitatively: even the best models remain far from saturation, and the gap between overall performance and domain expectations is substantial (Bao et al., 19 Sep 2025).

5. Sampling density, reasoning, and temporal grounding

Two findings in TennisTV are especially methodologically significant. First, frame-sampling density should be tailored and balanced across tasks. Second, improving temporal grounding is essential for stronger reasoning (Bao et al., 19 Sep 2025). These conclusions emerge from explicit ablation rather than conjecture.

For Qwen2.5VL-7B, a sampling-density study compares 1 FPS, 16 frames, 32 frames, and 64 frames. The best overall result is obtained at 32 frames, with AvgSE 39.3, AvgRS 39.5, and Avg 39.4. Performance for stroke-level tasks improves with denser sampling, but rally-level tasks saturate and can regress with additional redundancy: at 64 frames, AvgRS falls to 38.5 and overall Avg to 38.6 (Bao et al., 19 Sep 2025). The benchmark therefore rejects the naive assumption that more frames monotonically improve performance.

TennisTV also analyzes reasoning-augmented open-source models against non-reasoning counterparts. On knowledge-light tasks such as Movement Grounding, Hit Direction, and Rally Count, reasoning-augmented models consistently outperform non-reasoning models, with the most pronounced improvements on Rally Count and Hit Direction. However, the gain on motion-oriented tasks such as Movement Grounding is more modest (Bao et al., 19 Sep 2025). This pattern suggests that chain-of-thought-style reasoning assists aggregation and discrete inference, but does not solve the underlying perception problem of temporally localizing sub-second sports events.

The paper further reports strong positive correlations among Action Recognition, Technique Identification, and Technique Preference (Bao et al., 19 Sep 2025). This implies that low-level action parsing, mid-level technique recognition, and higher-level tactical tendency inference are not independent competencies; they form a logical progression in which failures propagate upward.

6. Position within tennis video understanding research

TennisTV belongs to a broader lineage of domain-specific tennis video analysis, but its role is diagnostic rather than directly generative. “SmartTennisTV” proposed score based indexing of tennis videos through rally segmentation, score recognition, and refinement using the tennis scoring system (Ghosh et al., 2018). “TennisVid2Text” generated semantically richer descriptions from lawn tennis broadcast videos by combining fine-grained action analysis with retrieval from a large human-written commentary corpus (Sukhwani et al., 2015). “TennisExpert” later introduced TennisVL and a memory-augmented multimodal system for analytical commentary generation, emphasizing tactical reasoning, player decisions, and match momentum (Liu et al., 11 Mar 2026).

Within MLLM-focused work, the benchmark complements a separate study on multimodal LLMs for tennis video analysis that found raw MLLMs lag domain-specific systems and improve substantially when structured player, ball, and court information is supplied as text prompts (Teo, 24 Jun 2025). TennisTV makes a related but distinct contribution: it provides a standardized testbed showing where contemporary MLLMs fail when asked to reason directly over rally videos (Bao et al., 19 Sep 2025).

This suggests that future progress in tennis video intelligence will likely depend on hybridization. TennisTV indicates that benchmark performance is constrained by temporal grounding; related work indicates that structured detectors and domain metadata can materially improve MLLM outputs (Teo, 24 Jun 2025). In that sense, TennisTV can be read as both a benchmark and an argument for tighter integration between sports-specific perception modules and multimodal reasoning layers.

7. Interpretation, limitations, and significance

The main significance of TennisTV is that it exposes a mismatch between broad video-language competence and domain-specific temporal sports understanding. Tennis rallies are short, rapidly alternating, and tactically dense. TennisTV operationalizes these properties through stroke-level and rally-level tasks, then shows that contemporary MLLMs remain only moderately above chance, with persistent weaknesses in temporal grounding (Bao et al., 19 Sep 2025).

Its findings also temper two simplistic narratives. The first is that larger or more general MLLMs will automatically solve sports understanding. TennisTV reports substantial shortcomings even for strong closed-source systems (Bao et al., 19 Sep 2025). The second is that increased sampling or added reasoning alone is sufficient. The reported ablations show instead that sampling must be task-dependent and that reasoning gains remain limited when temporal localization is weak (Bao et al., 19 Sep 2025).

For the research community, TennisTV provides a benchmark that disentangles perception, temporal aggregation, and tactical inference in a single domain. For sports-video research more broadly, a plausible implication is that its design principles may generalize to other high-temporal-density settings in which events are brief, sequential, and semantically interdependent. Within tennis specifically, it establishes a quantitative yardstick for evaluating whether future MLLMs genuinely understand rallies rather than merely recognize tennis as a scene (Bao et al., 19 Sep 2025).