CourtSI-Bench: Sports Spatial Reasoning Benchmark

• CourtSI-Bench is a curated benchmark that tests vision-language models on 3D spatial reasoning in dynamic sports scenes with precise court-anchored metric evaluation.
• It leverages a semi-automatic reconstruction engine combining court calibration, ball tracking, and player mesh recovery to generate human-verified 3D spatial data.
• The benchmark exposes limitations in existing models, while fine-tuning on CourtSI data significantly boosts metric accuracy and supports cross-sport transfer.

CourtSI-Bench is the high-quality evaluation benchmark introduced alongside CourtSI in “Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports” to test whether vision-LLMs can perform metrically grounded, human-centric spatial reasoning in sports scenes, rather than only broad semantic understanding or static-scene spatial reasoning. Within the CourtSI framework, CourtSI is the large-scale training dataset, CourtSI-Bench is the held-out benchmark for rigorous evaluation, and CourtSI-Ext is a small cross-sport extension for transfer to an unseen but related sport. CourtSI-Bench contains 3,686 QA pairs sampled from 1,988 images across 382 distinct scenes, with no scene overlap with CourtSI (Yang et al., 10 Mar 2026).

1. Role within the CourtSI framework

CourtSI-Bench is positioned as necessary because existing spatial intelligence benchmarks largely focus on static indoor scenes, rigid objects, or non-sports settings, whereas sports combine articulated human bodies, fast-moving small objects, court-anchored geometry, and severe perspective ambiguity. The benchmark is therefore designed to expose failures in 3D localization, distance estimation, and relational reasoning that may remain hidden on conventional benchmarks. It evaluates sports scenes from badminton, tennis, and table tennis, using metrically anchored court geometry rather than purely appearance-based heuristics (Yang et al., 10 Mar 2026).

Its role differs sharply from that of the full CourtSI dataset. CourtSI contains 1,008,941 QA pairs generated from 52,481 images spanning 1,057 unique scenes and serves as the training resource. CourtSI-Bench is intended purely for evaluation and is subject to stricter human verification, filtering of ambiguous items, and balancing across sports and task categories. In the appendix, the final 3,686 QA pairs are reported as having been selected from 4,356 raw samples; any pair flagged by either of two annotators was removed, and the remaining questions were resampled to maintain balanced task and sport coverage.

This design implies a benchmark philosophy centered on diagnostic validity rather than scale alone. CourtSI-Bench is not simply a smaller sample of CourtSI, but a curated test instrument built to isolate failure modes in spatial intelligence under dynamic, human-centric, metrically constrained conditions.

2. Reconstruction engine and metric grounding

CourtSI-Bench is built on a semi-automatic reconstruction data engine that places sports scenes in a world-grounded coordinate system anchored to court geometry. Broadcast-view images are drawn from RacketVision, a dataset of 1,672 professional net-sports clips covering badminton, tennis, and table tennis. Frames with extreme viewing angles are filtered out first. The pipeline then applies three core components: court annotation, ball annotation, and player mesh recovery (Yang et al., 10 Mar 2026).

For court calibration, annotators label six 2D keypoints: four ground corner points and two height points on the net. Because court geometry is standardized, those points have known 3D coordinates in metric space. Camera calibration is then solved as a Perspective-n-Point problem, estimating intrinsics and extrinsics and defining a unified court-anchored world coordinate system. The appendix specifies a right-handed world frame in which the origin is the far court corner from the camera’s perspective, the $x$-axis runs along court length toward the camera, the $y$-axis runs along court width positively toward the camera, and $z$ is perpendicular to the court plane. For static broadcast views, one calibrated frame can be reused for adjacent frames. For dynamic views, DepthAnythingV3 is used only to transfer court keypoints geometrically between neighboring frames, after which camera parameters are re-estimated independently because direct parameter propagation was found unstable.

Ball reconstruction is also semi-automatic. Since the ball is tiny and monocular metric depth is unreliable, annotators click both the ball’s 2D image location and its projected ground position along an assistive projection line. Given camera parameters, a pixel $\mathbf{p}$ defines a 3D ray in world coordinates:

$$\mathbf{X}(\lambda) = -\mathbf{R}^T \mathbf{t} + \lambda \mathbf{R}^T \mathbf{K}^{-1} \mathbf{p}, \quad \lambda > 0,$$

and the intersection with the court plane is obtained by solving

$Z(\lambda) = 0.$

This permits indirect depth annotation and analytical recovery of the 3D ball position. For airborne ball motion in video, the authors fit approximate constant-acceleration trajectories to reduce annotation effort, reverting to per-frame annotation if quality is insufficient. For table tennis, they use a dedicated 2D-to-3D lifting method from prior work.

Player reconstruction uses PromptHMR to estimate SMPL-X human meshes in camera coordinates. SAM3 with a “player” prompt is used for player tracking and bounding boxes, with manual refinement when needed. Because recovered meshes often contain incorrect depth, annotators correct depth via the lowest mesh vertex, after which the mesh is re-aligned using the similarity transform

$X' = sX + (1 - s) C,$

where $C$ is the camera center and $s$ is determined by the depth correction. This avoids mesh-scale distortion that would arise from naive translation.

The paper validates this engine using a purpose-built multi-view dataset with synchronized views from professional matches and triangulated 3D references. Reported errors are $2.2\%$ for $y$0, $y$1 for $y$2, ball localization errors of $y$3 cm, $y$4 cm, and $y$5 cm in $y$6, player pelvis error of $y$7 cm, and human MPJPE of $y$8 cm. These errors also define later evaluation thresholds. This suggests that CourtSI-Bench’s metric tasks are not merely synthetic labels, but measurements backed by an explicit reconstruction-accuracy analysis.

3. Taxonomy, question formats, and benchmark composition

CourtSI-Bench follows the same underlying taxonomy as CourtSI. QA pairs are generated automatically from reconstructed 3D sports states using 94 templates. In semantic terms, the benchmark covers four top-level categories: spatial counting, distance measurement, localization, and relational reasoning. The appendix further notes 13 primary categories that expand into 20 distinct types via templating (Yang et al., 10 Mar 2026).

The core evaluated subcategories are distributed as follows:

Category	Subcategory	Count
Distance Measurement	Camera-Object	277
Distance Measurement	Height	229
Distance Measurement	Object-Line	317
Distance Measurement	Object-Object	663
Spatial Counting	Ball	28
Spatial Counting	Player	34
Localization	Object	368
Relational Reasoning	Ball-Zone	255
Relational Reasoning	Ball-Player	297
Relational Reasoning	Camera-Player	248
Relational Reasoning	Player-Zone	82
Relational Reasoning	Player-Player	393
Relational Reasoning	Player-Line	495

These questions are posed over the ball, players, and court, and require both camera-centric and world-centric reasoning, as well as both allocentric and egocentric reference frames. The benchmark is explicitly human-centric: player questions may refer to body parts, left/right from the player’s anatomical perspective, pelvis-based inter-player distance, and other mesh-grounded relations. Outputs include floats in meters, integer counts, 3D coordinates, and multiple-choice answers.

A standardized pre-prompt defines court-side conventions and distinguishes camera-view left/right from anatomical left/right. A post-prompt constrains output formats, such as a single float in meters or coordinates in “$y$9” format. For localization, one representative template defines the coordinate system as having origin $z$0 at the intersection of the far baseline and the left doubles sideline, or at the top-left corner of the table surface, with the $z$1-axis extending toward the camera along the sideline, the $z$2-axis extending along the far baseline or endline to the right, and the $z$3-axis vertical.

Sports are relatively balanced overall across badminton, tennis, and table tennis, though Player-Zone excludes table tennis because players do not stand on the table surface. This composition indicates that CourtSI-Bench targets not only numerical spatial estimation but also reference-frame control, body-grounded semantics, and court-relative geometry.

4. Evaluation protocol and metrics

CourtSI-Bench evaluates single-image reasoning rather than video-sequence understanding. Each model receives one question paired with one image. The image is annotated with bounding boxes and accompanying instructions to disambiguate players. Human evaluators are given the same image and question, plus the court size of each sport as a reference. The benchmark therefore measures spatial reasoning from a single sports frame under explicit linguistic instructions (Yang et al., 10 Mar 2026).

The main metric, following VSI, is Accuracy by exact matching. For numerical distance-measurement tasks, the paper defines Threshold Mean Relative Accuracy (T-MRA):

$z$4

where $z$5 is the ground truth, $z$6 is the prediction, $z$7, and $z$8. The threshold $z$9 is derived from reconstruction-error analysis.

Localization is evaluated differently because the output is a 3D coordinate. Instead of T-MRA, the benchmark uses binary accuracy with a smooth threshold of $\mathbf{p}$0 cm: if the 3D localization error exceeds $\mathbf{p}$1 cm the prediction receives $\mathbf{p}$2, otherwise $\mathbf{p}$3. The paper notes that $\mathbf{p}$4 cm is larger than the combined 3D threshold $\mathbf{p}$5.

Prompting is tightly controlled. Each query contains a standardized pre-prompt, the question template, and a post-prompt specifying answer format only. Some proprietary models, especially Gemini-3-Pro and Claude-Sonnet4.5, often ignore output constraints and generate extended explanations. The paper therefore also reports “---parsed” scores, in which a separate Qwen3-8B LLM extracts the final answer from the raw response using a dedicated parsing prompt. This procedural choice is itself part of the benchmark methodology, because format compliance materially affects measured performance.

5. Model study and empirical findings

The benchmark study evaluates 25 VLMs before adding the authors’ fine-tuned variant. The proprietary set includes GPT-5.2, Gemini-3-Pro, Seed1.8, Claude-Sonnet4.5, Grok4, and Qwen3-Max. The open-source general models include Qwen3-VL, InternVL3.5, Kimi-VL, and LLaVA-OneVision variants. A separate group contains open-source spatial-intelligence models trained on prior spatial reasoning datasets, including SpaceR-7B, VST-7B-SFT, VST-7B-RL, SpatialLadder, SenseNova-SI-8B, and Cambrain-S-7B (Yang et al., 10 Mar 2026).

The central empirical result is that sports-specific spatial intelligence remains difficult. Human performance on a uniform 5% subset is best overall at 73.6, but remains imperfect on metric-sensitive tasks: 64.4 on Camera-Object distance, 67.8 on Object-Line, 70.0 on Object-Object, and 11.9 on localization. The strongest proprietary system after parsing is Gemini-3-Pro---parsed at 64.6 overall, still below humans. GPT-5.2 reaches 53.7, Seed1.8 reaches 52.7, and Claude-Sonnet4.5 improves from 5.0 raw to 49.1 parsed, showing that output-format failures materially affect benchmark outcomes.

Among open-source general models, Qwen3-VL-235B-A22B is strongest at 47.2 overall, followed by LLaVA-OneVision-72B at 42.0 and InternVL3.5-241B-A28B at 40.0. Distance measurement is especially weak: many models are near zero on Camera-Object or Object-Line even under T-MRA. Localization is near-zero for almost all models. Counting is much easier, particularly player counting, where many models reach 100, while ball counting is more variable.

A major claim of the paper is that existing spatial-intelligence tuning does not transfer well to sports. CourtSI-Bench provides the evidence. SpaceR-7B scores 32.8 overall versus base Qwen2.5-VL-7B at 37.0; SpatialLadder scores 34.7 versus base Qwen2.5-VL-3B at 35.0; SenseNova-SI-8B scores 31.5 versus base InternVL3-8B at 27.8; Cambrain-S-7B reaches only 25.5. This supports the interpretation that strong performance on indoor, static, or object-centric spatial benchmarks does not automatically generalize to court sports.

The benchmark is also the main target of the paper’s adaptation experiment. Qwen3-VL-8B is fine-tuned on CourtSI for one epoch with global batch size 2048 and learning rate $\mathbf{p}$6 using LLaMA Factory. The resulting model, denoted Ours$\mathbf{p}$7, improves from 37.7 to 61.2 overall on CourtSI-Bench, a gain of 23.5 percentage points. The gains are especially large on difficult metric subtasks: Camera-Object distance 3.1 $\mathbf{p}$8 60.2, Height 49.3 $\mathbf{p}$9 94.2, Object-Object 27.1 $$\mathbf{X}(\lambda) = -\mathbf{R}^T \mathbf{t} + \lambda \mathbf{R}^T \mathbf{K}^{-1} \mathbf{p}, \quad \lambda > 0,$$0 68.4, Ball counting 39.3 $$\mathbf{X}(\lambda) = -\mathbf{R}^T \mathbf{t} + \lambda \mathbf{R}^T \mathbf{K}^{-1} \mathbf{p}, \quad \lambda > 0,$$1 92.9, Localization 0.0 $$\mathbf{X}(\lambda) = -\mathbf{R}^T \mathbf{t} + \lambda \mathbf{R}^T \mathbf{K}^{-1} \mathbf{p}, \quad \lambda > 0,$$2 7.9, Player-Zone 30.5 $$\mathbf{X}(\lambda) = -\mathbf{R}^T \mathbf{t} + \lambda \mathbf{R}^T \mathbf{K}^{-1} \mathbf{p}, \quad \lambda > 0,$$3 85.4, and Player-Line 50.1 $$\mathbf{X}(\lambda) = -\mathbf{R}^T \mathbf{t} + \lambda \mathbf{R}^T \mathbf{K}^{-1} \mathbf{p}, \quad \lambda > 0,$$4 68.5. Player counting changes only from 97.1 to 100 because it is already near ceiling.

These results establish CourtSI-Bench as both a failure-analysis instrument and a transfer target: it reveals severe weaknesses in current VLMs, yet also shows that supervision from CourtSI substantially improves domain-specific spatial competence.

6. Error modes, transfer, downstream use, and limitations

The paper’s error analysis clarifies what CourtSI-Bench measures. Figure 1 reports that models often produce reasonable, human-like reasoning structures, can identify the tiny ball, can handle some ego-to-allocentric conversions, and can use general knowledge of court structure. Yet they still fail on accurate 3D localization and fine-grained relational reasoning. Typical errors include misjudging object-to-court relations, mishandling unusual player orientations under allocentric perspective, and making incorrect metric estimates from a single 2D view. Perspective ambiguity is emphasized as a core difficulty: performance degrades as the ratio of 3D distance to 2D projected distance increases, especially for object-object and object-line tasks (Yang et al., 10 Mar 2026).

CourtSI-Ext extends the benchmark logic to cross-sport transfer. It follows the same taxonomy and evaluation process but is built on pickleball, an unseen but related sport. CourtSI-Ext contains 215 QA pairs from 111 images across 35 scenes, collected from YouTube at 1080p and 25 fps. The fine-tuned Qwen3-VL-8B improves from 38.2 to 51.4 overall, a gain of 13.2 points, with scores such as 70.0 on Camera-Object distance, 83.1 on Height, 62.7 on Object-Object, and 83.3 on Ball counting. Localization does not improve despite lower average error, and the gain is smaller than on CourtSI-Bench. A plausible implication is that CourtSI-Bench probes an in-domain competence that is transferable but not yet fully abstracted across sports.

The benchmark also supports a downstream application study in spatial-aware commentary generation. The authors extract reliable spatial relations, particularly from distance-measurement cases, and insert them into commentary prompts. On 100 generated commentaries evaluated by three volunteers, fine-tuning on CourtSI improves spatial awareness while preserving overall linguistic quality. This suggests that CourtSI-Bench is not only an evaluation set but also an instrument for demonstrating that metrically grounded supervision transfers to sports-language generation.

Several limitations are stated or implicit. CourtSI-Bench is built from broadcast-view net sports only, so viewpoint diversity is intentionally restricted. The source data come from publicly available YouTube videos in RacketVision, and benchmark coverage inherits that dataset’s biases. Some question types are inherently ambiguous because players have non-negligible physical width, particularly for left/right relations; the authors mitigate this by removing flagged cases during human review and using sport-specific thresholds. Human performance itself is imperfect, especially on metric tasks, indicating that some items are genuinely difficult even with court geometry references. CourtSI-Ext is also small and is presented only as an initial cross-sport study.

CourtSI-Bench therefore occupies a specific position in the VLM evaluation landscape: it is a rigorously filtered, human-verified benchmark of sports spatial intelligence built on semi-automatic 3D reconstruction and court-anchored metric reasoning. Its defining contribution is to show that dynamic, articulated, human-centric sports scenes expose limitations in current VLM spatial reasoning that remain under-measured by existing benchmarks, while also providing a concrete target on which domain-specific supervision yields large gains.