CourtSI: Spatio-Temporal & Spatial Intelligence

• CourtSI is a dual-use term referring to a tensor-based spatio-temporal system for basketball analytics and a 3D vision-language dataset for net sports.
• It employs advanced tensor decomposition methods like PARAFAC with diagnostics such as CORCONDIA for optimal rank selection and prototype interpretation.
• CourtSI enables efficient retrieval, indexing, and spatial reasoning by converting court geometry and event data into actionable latent representations.

CourtSI is a name used in sports-computing research for two distinct constructs. In basketball analytics, it denotes a Court Spatio-Temporal Information system built on the tHoops framework, in which a 3-mode tensor over entities, court locations, and time bins is decomposed into interpretable prototype patterns for comparison, retrieval, and indexing (Papalexakis et al., 2017). In vision-language modeling, CourtSI denotes Court Spatial Intelligence, a large-scale, metric-grounded dataset and benchmark for spatial reasoning in badminton, tennis, and table tennis, together with CourtSI-Bench and CourtSI-Ext for evaluation and transfer studies (Yang et al., 10 Mar 2026).

1. CourtSI as a tensorized spatio-temporal representation for basketball

In the tHoops line of work, CourtSI begins with a collection of events such as shots taken, possessions, or offensive formations. These events are indexed by entities $e\in\{1,\dots,E\}$, court locations $\ell\in\{1,\dots,L\}$, and time bins $t\in\{1,\dots,T\}$, where entities may be players, teams, or individual possessions; locations may be discrete court-zones or grid cells; and time bins may be shot-clock seconds, quarter indices, or minute-bins of the game clock. From these counts one forms a nonnegative tensor $X\in\mathbb{R}^{E\times L\times T}$ with

$$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$

This construction makes the unit of analysis explicit and preserves spatial and temporal structure that would be lost in a purely spatial shot-chart representation (Papalexakis et al., 2017).

CourtSI then applies a PARAFAC, or CP, decomposition to obtain a low-rank approximation

$$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$

with factor vectors $a_f\in\mathbb{R}^E$, $b_f\in\mathbb{R}^L$, and $c_f\in\mathbb{R}^T$, or equivalently factor matrices $A\in\mathbb{R}^{E\times F}$, $\ell\in\{1,\dots,L\}$0, and $\ell\in\{1,\dots,L\}$1. For count data, the formulation minimizes a KL-divergence objective under nonnegativity constraints,

$\ell\in\{1,\dots,L\}$2

Each column triple $\ell\in\{1,\dots,L\}$3 is interpreted as a prototype spatio-temporal pattern: $\ell\in\{1,\dots,L\}$4 captures the spatial signature, $\ell\in\{1,\dots,L\}$5 captures the temporal signature, and $\ell\in\{1,\dots,L\}$6 captures soft membership weights over entities. In this sense, CourtSI operationalizes basketball tendencies as mixtures of latent offensive or shot-selection patterns rather than as heuristic summaries.

2. Rank selection, prototype interpretation, and latent coordinates

A central technical issue in the basketball CourtSI system is selection of the number of components $\ell\in\{1,\dots,L\}$7. The framework uses two complementary procedures. For $\ell\in\{1,\dots,L\}$8, it uses the CORCONDIA diagnostic, where values in $\ell\in\{1,\dots,L\}$9 closer to $t\in\{1,\dots,T\}$0 indicate that the data admit a good trilinear model at rank $t\in\{1,\dots,T\}$1. An efficient sparse algorithm due to Papalexakis and Sidiropoulos (2015) is used to avoid forming huge Kronecker products, and $t\in\{1,\dots,T\}$2 is increased until CORCONDIA drops significantly or until the min-dimension is reached. Beyond that regime, the framework uses a silhouette-based heuristic that compares clustering quality in raw space and latent space (Papalexakis et al., 2017).

The silhouette procedure is defined explicitly. For each entity $t\in\{1,\dots,T\}$3, the raw feature vector is formed by flattening $t\in\{1,\dots,T\}$4; these $t\in\{1,\dots,T\}$5 vectors are clustered into $t\in\{1,\dots,T\}$6 groups to compute

$t\in\{1,\dots,T\}$7

For a candidate rank $t\in\{1,\dots,T\}$8, the latent feature of entity $t\in\{1,\dots,T\}$9 is the $X\in\mathbb{R}^{E\times L\times T}$0-th row of $X\in\mathbb{R}^{E\times L\times T}$1,

$X\in\mathbb{R}^{E\times L\times T}$2

and the set $X\in\mathbb{R}^{E\times L\times T}$3 is clustered into $X\in\mathbb{R}^{E\times L\times T}$4 groups to compute

$X\in\mathbb{R}^{E\times L\times T}$5

The selected rank is the smallest $X\in\mathbb{R}^{E\times L\times T}$6 such that $X\in\mathbb{R}^{E\times L\times T}$7 and $X\in\mathbb{R}^{E\times L\times T}$8, where $X\in\mathbb{R}^{E\times L\times T}$9 is a small convergence threshold, for example $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$0. The framework further notes that a “strong structure” is often indicated by $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$1 as a rule of thumb.

Once $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$2, $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$3, and $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$4 are obtained, each component becomes directly interpretable as a prototype. The spatial factor can be normalized as

$$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$5

and the temporal factor as

$$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$6

These normalized signatures specify where and when pattern $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$7 tends to occur. The paper notes that $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$8 might highlight “corner-3 zones” while $$X(e,\ell,t)=\text{number of events by entity }e\text{ at location }\ell\text{ during time bin }t.$$9 might peak in the “last 5 seconds.” This makes CourtSI not merely a factorization scheme, but a prototype-based descriptive language for basketball behavior.

3. CourtSI indexing, retrieval, and downstream basketball operations

CourtSI represents each entity $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$0 by the latent vector $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$1. Equivalently, the $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$2-slice of the original tensor is approximated by

$$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$3

so that $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$4 quantifies how strongly entity $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$5 uses pattern $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$6. This latent embedding is the basis for indexing. The system precomputes and stores the factor matrices $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$7, $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$8, and $$X \approx \hat X = \sum_{f=1}^F a_f \circ b_f \circ c_f,$$9; optionally normalizes $a_f\in\mathbb{R}^E$0 and $a_f\in\mathbb{R}^E$1 into $a_f\in\mathbb{R}^E$2; and stores the length-$a_f\in\mathbb{R}^E$3 vector $a_f\in\mathbb{R}^E$4 for each entity or possession (Papalexakis et al., 2017).

Querying proceeds by spatio-temporal template. A user specifies a query $a_f\in\mathbb{R}^E$5 over locations $a_f\in\mathbb{R}^E$6 time, such as “corner 3’s” plus “last 5s.” The system projects $a_f\in\mathbb{R}^E$7 onto the learned prototypes by computing similarity scores

$a_f\in\mathbb{R}^E$8

It retains those patterns for which $a_f\in\mathbb{R}^E$9 exceeds a threshold $b_f\in\mathbb{R}^L$0, and retrieves entities whose coordinate $b_f\in\mathbb{R}^L$1 is large for those patterns. Because $b_f\in\mathbb{R}^L$2 and each lookup is $b_f\in\mathbb{R}^L$3, retrieval is $b_f\in\mathbb{R}^L$4 rather than $b_f\in\mathbb{R}^L$5. The development workflow is correspondingly explicit: define bins for court-zones and time; build $b_f\in\mathbb{R}^L$6 with counts or rates; run nonnegative PARAFAC with an initial $b_f\in\mathbb{R}^L$7; use CORCONDIA and the silhouette heuristic to choose $b_f\in\mathbb{R}^L$8; extract and normalize the factors; set each embedding $b_f\in\mathbb{R}^L$9; and implement the query interface.

The stated downstream uses are broad within basketball operations. CourtSI can visualize team and player tendencies in $c_f\in\mathbb{R}^T$0-dimensional latent space, compare entities by Euclidean or cosine distance, efficiently retrieve film clips or event sequences matching a query in sublinear time, and drive similarity search, clustering, outlier detection, and synthetic data generation by sampling the learned prototypes. A common misconception is to reduce this CourtSI usage to a visualization layer; in the tHoops formulation it is an end-to-end indexing and retrieval system grounded in tensor decomposition.

4. CourtSI as a large-scale VLM dataset for court-grounded spatial intelligence

In the 2026 literature, CourtSI refers to a different object: a dataset and benchmark designed to benchmark and advance the spatial reasoning capabilities of vision-LLMs in dynamic, human-centric sports scenes, specifically badminton, tennis, and table tennis. Its stated objectives are to provide a metrically grounded testbed for fine-grained spatial tasks, to exploit well-defined court geometry as a stable metric anchor for recovering accurate 3D scene states from monocular broadcast images, and to enable scalable automatic QA generation together with a rigorously verified evaluation subset. The motivation is that existing spatial benchmarks focus on static, rigid objects or indoor scenes, whereas sports introduce non-rigid human motion, articulated body poses, rapid object dynamics, and strict real-world metric constraints (Yang et al., 10 Mar 2026).

CourtSI covers four task families, with each QA pair derived from 3D coordinates $c_f\in\mathbb{R}^T$1 of players and ball in a unified world frame anchored to court corners. Spatial Counting asks for the number of visible entities, such as the number of players on court or whether the shuttlecock is visible. Distance Measurement computes Euclidean distances between camera, objects, lines, or court plane, using

$c_f\in\mathbb{R}^T$2

in meters. Its subtasks include camera–object distance, height above the court surface, perpendicular distance from the ball to a service line, and distance between the two players’ pelvis joints. Localization returns the absolute 3D coordinate $c_f\in\mathbb{R}^T$3 of a specified object in a right-handed court frame. Relational Reasoning covers ball-zone, ball-player, camera-player, player-zone, player-player, and player-line questions, including egocentric and allocentric left-right and front-back relations.

The scale of the dataset and benchmark is as follows:

Split	QA pairs	Images / scenes
CourtSI	1,008,941	52,481 images / 1,057 distinct scenes
CourtSI-Bench	3,686	1,988 images / 382 distinct scenes
CourtSI-Ext	215	evaluation set on Pickleball

The training set contains over 1 million QA pairs generated from 52,481 images and 1,057 distinct scenes, with approximate per-sport distribution of 31.0% badminton, 25.2% tennis, and 43.7% table tennis. Its reported per-category counts are Distance Measurement (4 subtasks): 408 (769)* QA pairs; Spatial Counting (2 subtasks): 45,892; Localization: 101,698; and Relational Reasoning (6 subtasks): 497,712. All distances and coordinates are in SI units. The benchmark contains 3,686 QA pairs from 1,988 images and 382 distinct scenes, with sport distribution 30.5% badminton, 25.4% tennis, and 44.1% table tennis. Its per-category counts are 277 Distance–Cam-Obj, 229 Distance–Height, 317 Distance–Obj-Line, 663 Distance–Obj-Obj, 28 Spatial Counting–Ball, 34 Spatial Counting–Player, 368 Localization, 255 Relational–Ball-Zone, 297 Relational–Ball-Player, 248 Relational–Cam-Player, 82 Relational–Player-Zone, 393 Relational–Player-Player, and 495 Relational–Player-Line. This distribution indicates that CourtSI-Bench is not limited to coarse recognition; it emphasizes metrically specified geometric reasoning.

5. Scene reconstruction, metric anchoring, and evaluation protocol

CourtSI’s data engine is semi-automatic and is centered on explicit 3D scene reconstruction. The pipeline begins with court annotation: annotators manually label six 2D keypoints, namely four court corners and two net-height points, and then solve a Perspective-n-Point problem to obtain camera intrinsics $c_f\in\mathbb{R}^T$4 and extrinsics $c_f\in\mathbb{R}^T$5 in a metric world coordinate system with court plane $c_f\in\mathbb{R}^T$6. Ball annotation then requires annotators to click the 2D pixel $c_f\in\mathbb{R}^T$7 of the ball and its ground projection on $c_f\in\mathbb{R}^T$8, after which the system solves

$c_f\in\mathbb{R}^T$9

with $A\in\mathbb{R}^{E\times F}$0 chosen so that $A\in\mathbb{R}^{E\times F}$1, in order to recover 3D ball position. Player mesh recovery uses PromptHMR to estimate SMPL-X human meshes in camera coordinates, given player bounding boxes obtained from SAM3 plus manual refinement and camera intrinsics; annotators then label the lowest mesh vertex’s real height above court, compute depth correction, and apply a similarity transform about the camera center $A\in\mathbb{R}^{E\times F}$2, $A\in\mathbb{R}^{E\times F}$3, with $A\in\mathbb{R}^{E\times F}$4 defined as corrected depth over predicted depth (Yang et al., 10 Mar 2026).

Court geometry supplies the metric anchors. The paper gives badminton court length $A\in\mathbb{R}^{E\times F}$5, width $A\in\mathbb{R}^{E\times F}$6, and net height $A\in\mathbb{R}^{E\times F}$7 as examples of fixed real-world dimensions used to define 3D keypoints for PnP. The reported consequence is centimeter-level accuracy, with focal length error $A\in\mathbb{R}^{E\times F}$8 and ball/player localization errors $A\in\mathbb{R}^{E\times F}$9 cm. This is significant because the downstream questions ask for metric distances and coordinates rather than purely ordinal spatial relations.

CourtSI-Bench is curated by sampling 3,686 QA pairs over all categories and sports with no scene overlap with CourtSI training. Human verification is performed by two annotators who independently inspect rendered 3D scene visualizations together with the QA; any QA flagged by either annotator is removed, and the final benchmark retained 3,686 of 4,356 candidate QA. The evaluation protocol uses a single image, bounding boxes, a question, and standardized pre- and post-prompts. Metrics are Exact-match Accuracy for multiple-choice questions and counting, T-MRA for distance tasks, and binary 30 cm threshold accuracy for localization. On a 5% subset, human performance is reported as 73.6% overall, breaking down as Distance Measurement $\ell\in\{1,\dots,L\}$00, Counting $\ell\in\{1,\dots,L\}$01, Localization $\ell\in\{1,\dots,L\}$02, and Relational Reasoning $\ell\in\{1,\dots,L\}$03. The low localization score under the benchmark’s protocol helps clarify that CourtSI’s localization task is unusually strict: it asks for absolute 3D coordinates under a binary threshold rather than approximate scene description.

6. Empirical findings, adaptation, and terminological overlap

CourtSI-Bench is used to evaluate 25 proprietary and open-source VLMs. The reported top proprietary systems are GPT-5.2 at 53.7% overall and Gemini-3-Pro at a parsed 64.6% overall, while the open-source general VLM Qwen3-VL-235B-A22B reaches 47.2% overall. Specialized spatial VLMs including SpaceR, VST, SpatialLadder, SenseNova-SI, and Cambrain-S show minimal gains over their bases, specifically less than 2 percentage points. The benchmark therefore exposes a remaining human–AI performance gap of approximately 10 percentage points overall and more than 25 percentage points in distance measurement (Yang et al., 10 Mar 2026).

The paper further reports fine-tuning Qwen3-VL-8B on the full CourtSI training set using 1 epoch, global batch 2048, and learning rate $\ell\in\{1,\dots,L\}$04 in LLaMA Factory. On CourtSI-Bench, overall performance rises from 37.7% to 61.2%, a gain of 23.5 percentage points. Category-specific improvements are also reported: Distance–Cam-Obj from 3.1% to 60.2%, Distance–Height from 49.3% to 94.2%, Distance–Obj-Line from 21.3% to 47.6%, Distance–Obj-Obj from 27.1% to 68.4%, Counting–Ball from 39.3% to 92.9%, Counting–Player from 97.1% to 100.0%, Localization from 0.0% to 7.9%, and average Relational performance from 56.9% to 65.1%. On CourtSI-Ext, an unseen Pickleball evaluation set of 215 QA pairs, the same model improves from 38.2% to 51.4%, showing a reported transfer of spatial reasoning to a similar but unseen net sport.

CourtSI is also used for spatial-aware commentary generation. Given a distance measurement such as “Ball is 4.2 m from player A,” the model generates live-style sports commentary that embeds the numerical fact. A user study on 100 sampled cases, with 3 volunteers rating base versus tuned outputs on linguistic quality and spatial awareness, finds that the tuned model significantly outperforms the base in spatial awareness, with vote share increasing by approximately 50 percentage points, while linguistic quality remains competitive with no significant degradation. A representative example contrasts the base sentence “The player returns the ball quickly.” with the tuned sentence “With the shuttlecock still hovering 3.8 m from the baseline, she lunges forward and delivers a fierce cross-court shot.”

A recurrent source of confusion is terminological overlap. In the 2017 basketball analytics framework, CourtSI denotes a tensor-based Court Spatio-Temporal Information system for indexing and retrieving basketball behaviors (Papalexakis et al., 2017). In the 2026 VLM benchmark, CourtSI denotes Court Spatial Intelligence, a dataset and evaluation suite for metric 3D reasoning in net sports (Yang et al., 10 Mar 2026). The two uses are methodologically distinct—one is a latent-factor representation over event tensors, the other a reconstructed 3D QA benchmark over images—yet both are organized around court-grounded spatial structure. This suggests a broader family resemblance in which “CourtSI” names systems that convert court geometry and event structure into searchable or learnable spatial representations.