RacketVision: Multi-Sport Vision Dataset

• RacketVision is a multi-sport benchmark offering large-scale, fine-grained annotations for ball tracking, racket pose estimation, and trajectory forecasting.
• It employs robust multi-modal fusion techniques, notably a critical CrossAttention mechanism that enhances feature integration.
• The dataset sets a new standard for sports analytics by enabling detailed analysis across table tennis, tennis, and badminton environments.

RacketVision is a novel dataset and benchmark for advancing computer vision in sports analytics, covering table tennis, tennis, and badminton. It is designed to tackle three interconnected tasks: fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting. The dataset is the first to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. Its baseline study also isolates a central methodological result for multi-modal fusion: while naively concatenating racket pose features degrades performance, a CrossAttention mechanism is essential to unlock their value (Dong et al., 21 Nov 2025).

1. Dataset scope and annotation design

RacketVision is a large-scale, multi-sport video benchmark designed to advance unified analysis of fast-moving balls and rackets. The sports covered are Table Tennis, Tennis, and Badminton. Its scale and diversity are specified as follows: #Games = 942 professional matches; #Clips = 1 672, each 5–10 s long with total duration ≈ 12 755 s; #Frames = 435 179 at 1080 × 1920 resolution; and environments consisting of broadcast videos spanning indoor/outdoor courts and multiple camera angles (Dong et al., 21 Nov 2025).

The annotation types are likewise explicit. Ball positions are annotated as 2D pixel coordinates $(x_t,y_t)$ plus a visibility flag $v_t\in\{0,1\}$. Racket pose is annotated per frame as a bounding box $(x_{\min},y_{\min},x_{\max},y_{\max})$ plus five 2D keypoints $\{(x_i,y_i)\}_{i=1}^5$ capturing racket topology. This annotation design couples small-object localization with articulated object structure, which is the principal formal distinction of the benchmark (Dong et al., 21 Nov 2025).

The per-sport statistics further indicate the distribution of supervision. TableTennis comprises 50 games, 780 clips, 170\,027 frames, duration 3\,878 s, 19\,495 ball annotations, and 6\,648 racket annotations. Tennis comprises 431 games, 431 clips, 150\,399 frames, duration 4\,285 s, 21\,544 ball annotations, and 7\,395 racket annotations. Badminton comprises 461 games, 461 clips, 114\,753 frames, duration 4\,592 s, 23\,003 ball annotations, and 10\,578 racket annotations. In total, the benchmark contains $N_{\rm frames}=435\,179$, $N_{\rm B}=64\,042$, and $N_{\rm R}=24\,621$ (Dong et al., 21 Nov 2025).

2. Tasks, losses, and evaluation protocol

RacketVision defines three interconnected tasks—ball tracking, racket pose estimation, and trajectory forecasting—which form a pipeline from low-level perception to high-level prediction. For ball tracking, the problem is: given a frame $I_t$, or a short video $\{I_{t-N},\dots,I_t\},N=5$, predict $(\hat x_t,\hat y_t)$ and visibility $v_t\in\{0,1\}$0. The loss is

$v_t\in\{0,1\}$1

and the reported metrics are Precision, Recall at $v_t\in\{0,1\}$2, Mean Distance Error (MDE), and mAP@50 (Dong et al., 21 Nov 2025).

For racket pose estimation, the task is: given $v_t\in\{0,1\}$3, predict the bounding box and keypoints $v_t\in\{0,1\}$4. The loss is

$v_t\in\{0,1\}$5

with evaluation by [email protected], MPJPE, mOKS, NME, and Detection mAP@50. The formulation makes the racket an articulated object rather than a coarse detection target, which is consequential for downstream conditioning and interaction analysis (Dong et al., 21 Nov 2025).

For ball trajectory forecasting, the problem is: given history of ball coordinates $v_t\in\{0,1\}$6, and optionally racket poses $v_t\in\{0,1\}$7, predict future $v_t\in\{0,1\}$8. The loss is

$v_t\in\{0,1\}$9

and the metrics are ADE and FDE. Two settings are standardized: Short with $(x_{\min},y_{\min},x_{\max},y_{\max})$0, and Long with $(x_{\min},y_{\min},x_{\max},y_{\max})$1. Splits are by game: ≈ 70% games for training, 10% for validation, and 20% for testing—each sport independently and a combined multi-sport split for cross-sport experiments. All evaluations use held-out clips and report average metrics across sports (Dong et al., 21 Nov 2025).

3. Baseline architectures and the fusion result

The baseline stack spans all three tasks. Ball tracking backbones are RTMDet, YOLO11, WASB, and TrackNetV3. Racket pose estimation uses RTMPose with RTMDet for initial detection. Trajectory prediction backbones are a 2-layer LSTM and a 4-layer self-attention Transformer. For multimodal fusion, the benchmark formalizes CrossAttention with ball embeddings $(x_{\min},y_{\min},x_{\max},y_{\max})$2 and racket embeddings $(x_{\min},y_{\min},x_{\max},y_{\max})$3:

$(x_{\min},y_{\min},x_{\max},y_{\max})$4

$(x_{\min},y_{\min},x_{\max},y_{\max})$5

The compared settings are Ball-only, Ball+Racket via Concatenation, and Ball+Racket via CrossAttention (Dong et al., 21 Nov 2025).

For ball tracking, the best reported configuration is MS-TrackNetV3 with #F=4 and background modeling. Its results are: TableTennis, Prec. 0.924, Rec. 0.762, MDE 3.41, mAP 71.1; Tennis, Prec. 0.945, Rec. 0.880, MDE 1.96, mAP 81.9; Badminton, Prec. 0.915, Rec. 0.865, MDE 1.54, mAP 83.1. The benchmark further states that multi-sport training raises mAP by +19.2% in Tennis and +14.6% in Badminton versus single-sport, that background modeling cuts MDE by ≈ 54–61%, and that multi-frame input $(x_{\min},y_{\min},x_{\max},y_{\max})$6 improves recall and mAP but may slightly jitter final localization (Dong et al., 21 Nov 2025).

For racket pose estimation, MS RTMPose reports: TableTennis, [email protected] 81.8, MPJPE 9.71, mOKS 0.498, NME 0.254, mAP 78.4; Tennis, 89.6, 5.34, 0.630, 0.223, 79.4; Badminton, 88.5, 5.00, 0.668, 0.235, 75.5. Multi-sport training boosts [email protected] by +6–6.4% over SS. Structural keypoints—top, bottom, handle—exceed 92% PCK, whereas side keypoints lag at 65–80% due to occlusion and blur (Dong et al., 21 Nov 2025).

The trajectory forecasting experiments supply the benchmark’s main caution against naive multimodal integration. In the short setting $(x_{\min},y_{\min},x_{\max},y_{\max})$7, the LSTM on TableTennis gives Ball-only $(x_{\min},y_{\min},x_{\max},y_{\max})$8 in ADE/FDE, Ball+Concat $(x_{\min},y_{\min},x_{\max},y_{\max})$9, and Ball+CrossAttn $\{(x_i,y_i)\}_{i=1}^5$0. In the long setting $\{(x_i,y_i)\}_{i=1}^5$1, the same model gives Ball-only $\{(x_i,y_i)\}_{i=1}^5$2, Ball+Concat $\{(x_i,y_i)\}_{i=1}^5$3, and Ball+CrossAttn $\{(x_i,y_i)\}_{i=1}^5$4. The stated interpretation is that naive concatenation degrades performance because racket features are noisy when the ball is in flight, whereas CrossAttention picks out “hit” frames and yields consistent improvement versus ball-only; the overall ADE/FDE gains remain modest because many segments lack racket contact (Dong et al., 21 Nov 2025).

4. Technique recognition and tactical analysis

A related line of work positions RacketVision beyond ball-and-racket kinematics and into stroke semantics. In “ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis,” the technical report describes ViSTec as a two-stage action-perception system designed to recognize fine-grained stroke techniques in broadcast racket-sport videos and inject high-level tactical knowledge. It comprises a video-encoding backbone based on VideoMAE adapted to extremely short “strokes,” a temporal segmentation (“seg”) module that detects instantaneous stroke events, and a two-part classification stage (“cls” + “grh”) that predicts a stroke’s technique from visual features, then refines it by consulting a learnable tactical graph (He et al., 2024).

The graph component explicitly models strategic knowledge in stroke sequences. Its node set is $\{(x_i,y_i)\}_{i=1}^5$5, with directed edges connecting every node to every other node except no incoming edges into “null.” At inference, the previous stroke label selects a transition vector $\{(x_i,y_i)\}_{i=1}^5$6, both logits and graph scores are min–max normalized into $\{(x_i,y_i)\}_{i=1}^5$7, and the combined posterior is

$\{(x_i,y_i)\}_{i=1}^5$8

Adaptive updates depend on whether the predicted label matches the ground truth and on an entropy-based uncertainty term

$\{(x_i,y_i)\}_{i=1}^5$9

The total loss per video is $N_{\rm frames}=435\,179$0, with $N_{\rm frames}=435\,179$1, and no additional regularizers were introduced beyond standard weight decay (He et al., 2024).

The associated dataset is constructed from broadcast videos of World Table Tennis: 18 televised games and 4000 rally clips, automatically segmented via scoreboard change. Stroke timestamps are labeled at frame level by experts. Technique labels are $N_{\rm frames}=435\,179$2, assigned per stroke by professional table-tennis coaches. Sliding windows of 3 consecutive technique labels form basic tactical units for analysis and initialize graph edge frequencies. On this setup, Full ViSTec reports $N_{\rm frames}=435\,179$3 on F1@10,25,50; $N_{\rm frames}=435\,179$4 Acc.; and $N_{\rm frames}=435\,179$5 Edit, with gains of +2.5% F1@50 and +6.4% Acc over the best baseline, and inference speed of 39.3 fps on a single A100 GPU. Case studies with experts from the Chinese national table tennis team include t-SNE clustering of the 1280-d stroke features and a tactical analysis in which $N_{\rm frames}=435\,179$6 achieved $N_{\rm frames}=435\,179$7 win-rate, while $N_{\rm frames}=435\,179$8 drops win-rate to $N_{\rm frames}=435\,179$9. This suggests an extension from RacketVision’s benchmarked perception tasks to technical profiling, tactical pattern discovery, and strategy mining (He et al., 2024).

5. Ball sensing, blur, impact, and 3D tracking

Several neighboring studies address failure modes that are central in racket-sport perception. “BlurBall: Joint Ball and Motion Blur Estimation for Table Tennis Ball Tracking” starts from the observation that existing labeling conventions mark the ball at the leading edge of the blur streak. BlurBall instead places the ball center at the midpoint of the straight-line blur streak and annotates two additional blur attributes per frame: orientation $N_{\rm B}=64\,042$0 and half-length $N_{\rm B}=64\,042$1. The benchmark released in that work contains 64 119 frames total, with 51 423 training frames and 12 696 test frames, mean inter-frame displacement ≈ 19 px, blur ratio ≈ 62%, and a half-length distribution concentrated around 5 px with maximum observed $N_{\rm B}=64\,042$2 px. The proposed model uses HRNet with 1–3 sequential frames stacked along channels and inserts Squeeze-and-Excitation blocks after each stage. Its reported results include, for 1-step + SE at $N_{\rm B}=64\,042$3, F1 $N_{\rm B}=64\,042$4, Acc $N_{\rm B}=64\,042$5, AP $N_{\rm B}=64\,042$6, blur length MAE $N_{\rm B}=64\,042$7 px, and angle MAE $N_{\rm B}=64\,042$8; tuning $N_{\rm B}=64\,042$9 gives F1 $N_{\rm R}=24\,621$0, Acc $N_{\rm R}=24\,621$1, AP $N_{\rm R}=24\,621$2, blur length MAE $N_{\rm R}=24\,621$3 px, and angle MAE $N_{\rm R}=24\,621$4. The paper also reports that position-only trajectory fitting yields MAE $N_{\rm R}=24\,621$5 px, whereas position+blur yields $N_{\rm R}=24\,621$6 px, described as nearly 40 % reduction in average error and lower variance (Gossard et al., 22 Sep 2025).

“Locating Tennis Ball Impact on the Racket in Real Time Using an Event Camera” addresses a different sensing regime. Its input is a continuous stream of events $N_{\rm R}=24\,621$7, which are accumulated into overlapping event packets and processed in three stages: time-range detection of a swing, impact-timing detection via PATS, and contour extraction of ball and racket. PATS is defined through positive-polarity structure before a reference time and negative-polarity structure after that time:

$N_{\rm R}=24\,621$8

with a focal-time weighting $N_{\rm R}=24\,621$9 used to emphasize mid-interval events. On 50 impacts from 5 players with 10 hits each, the paper reports time-range detection success 49/50 $I_t$0, impact timing success 46/49 $I_t$1, mean absolute timing error $I_t$2\,μs $I_t$3250 μs, contour-detection success 24/26 $I_t$4 without direct sunlight and 3/20 $I_t$5 under sunlight, localization accuracy $I_t$6 pt and $I_t$7 pt implying $I_t$8 mm, and computation time 1.4–2.0 s per impact on a laptop with i7-12700H + RTX3080Ti. The stated limitations are direct sunlight, multi-colored rackets or balls, and the fact that the current 2 D method cannot recover racket tilt (Kase et al., 10 Jun 2025).

“Physics-Guided Fusion for Robust 3D Tracking of Fast Moving Small Objects” describes a RacketVision system for high-precision 3D tracking of fast-moving balls in racket sports. The system uses a ZED 2i stereo RGB-D camera mounted ≈35 ft from court, YOLOv8 fine-tuned on a racquetball dataset, and a physics-based tracker with state $I_t$9. Prediction follows kinematic motion equations,

$\{I_{t-N},\dots,I_t\},N=5$0

and measurement acceptance is gated by $\{I_{t-N},\dots,I_t\},N=5$1. Outliers are rejected when $\{I_{t-N},\dots,I_t\},N=5$2 or depth is NaN/infinite, while missed detections are handled by continuing with the physics prediction. The racquetball RGB-D dataset contains 12 sequences × 5 min each, 80 000+ frames at 720p/1280p + depth, and 10 000 frames manually labeled with 2D boxes using an 80/20 train/val split. Reported results include Scenario 1 with DeepOCSORT 22.97 / 0.69 versus Ours 8.42 / 0.22 in ADE/AMD, corresponding to 63% ADE reduction; Scenario 2 with DeepOCSORT 31.31 / 0.91 versus Ours 26.70 / 0.81, corresponding to 15% ADE reduction; and overall up to 90% ADE reduction vs. StrongSORT and 63–73% vs. best KF-based tracker. End-to-end throughput is sustained ∼40 FPS, with YOLOv8 at 1248×704 running at 24.5 ms or 41 FPS and depth query plus physics update taking <1 ms/frame (Singh et al., 23 Oct 2025).

6. Perception-to-action integration and future directions

RacketVision also appears as an onboard perception subsystem inside “CyboRacket: A Perception-to-Action Framework for Humanoid Racket Sports.” In that framework, the architecture is organized into three tightly-coupled layers: an Onboard Perception Loop at 100 Hz, an Interception Planning Loop at ≈ 20–30 Hz, and a Command Loop at ≥ 100 Hz. The onboard loop handles RGB–D acquisition, 2D detection, temporal tracking, 3D back-projection, and an EKF for state estimation and short-horizon forecasting. The camera is an Intel RealSense D455, or 100 Hz RGB–D head-mounted camera, and the state vector is

$\{I_{t-N},\dots,I_t\},N=5$3

with dynamics

$\{I_{t-N},\dots,I_t\},N=5$4

where $\{I_{t-N},\dots,I_t\},N=5$5. Outlier rejection uses a Mahalanobis gate on the innovation, and bounce handling applies $\{I_{t-N},\dots,I_t\},N=5$6 when $\{I_{t-N},\dots,I_t\},N=5$7 and $\{I_{t-N},\dots,I_t\},N=5$8 (Ren et al., 15 Mar 2026).

The planning layer exports an interception state only when EKF covariance is low enough, time-to-hit lies in the reaction window, and the hit point is reachable. It then selects a feasible hit point, desired base pose, swing direction, and swing phase, with plan representation

$\{I_{t-N},\dots,I_t\},N=5$9

The command interface to SONIC consists of task-space targets $(\hat x_t,\hat y_t)$0, leaving inverse kinematics, balance constraints, and ground-reaction stabilization to the pre-trained whole-body tracker. Reported results are: in simulation, 20 randomized tennis-ball-to-robot trials with 75 % hit rate and 70 % return rate; in the real world, onboard Intel RealSense at 90 Hz, YOLO latency ∼ 15 ms, total perception latency < 30 ms, vision-based tennis-hitting experiments at 10–20 m/s with typical hit success ∼ 60–70 %, EKF position RMSE ∼ 2–3 cm over a 2–4 m range, and short-horizon time-to-impact error < 20 ms (Ren et al., 15 Mar 2026).

The benchmark paper’s future research directions are consistent with these system-level developments. It explicitly identifies 3D racket pose and ball spin estimation leveraging multi-view or synthetic-to-real transfer, physics-informed forecasting combining learned trajectory priors with bounce/spin models, end-to-end architectures that jointly optimize tracking, pose and prediction, and extension to other racket sports such as squash and padel. A plausible implication is that RacketVision now functions less as a single isolated dataset than as a reference problem formulation linking low-level detection, articulated pose, stroke semantics, tactical analysis, and closed-loop interception under real-time constraints (Dong et al., 21 Nov 2025).