--- title: 'RacketVision: Multi-Sport Vision Dataset' url: https://www.emergentmind.com/topics/racketvision type: topic --- # RacketVision: Multi-Sport Vision Dataset RacketVision is a novel dataset and benchmark for advancing computer vision in sports analytics, covering table tennis, tennis, and badminton. It is designed to tackle three interconnected tasks: fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting. The dataset is the first to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. Its baseline study also isolates a central methodological result for multi-modal fusion: while naively concatenating racket pose features degrades performance, a CrossAttention mechanism is essential to unlock their value [2511.17045]. ## 1. Dataset scope and annotation design RacketVision is a large-scale, multi-sport video benchmark designed to advance unified analysis of fast-moving balls and rackets. The sports covered are Table Tennis, Tennis, and Badminton. Its scale and diversity are specified as follows: #Games = 942 professional matches; #Clips = 1 672, each 5–10 s long with total duration ≈ 12 755 s; #Frames = 435 179 at 1080 × 1920 resolution; and environments consisting of broadcast videos spanning indoor/outdoor courts and multiple camera angles [2511.17045]. The annotation types are likewise explicit. Ball positions are annotated as 2D pixel coordinates \((x_t,y_t)\) plus a visibility flag \(v_t\in\{0,1\}\). Racket pose is annotated per frame as a bounding box \((x_{\min},y_{\min},x_{\max},y_{\max})\) plus five 2D keypoints \(\{(x_i,y_i)\}_{i=1}^5\) capturing racket topology. This annotation design couples small-object localization with articulated object structure, which is the principal formal distinction of the benchmark [2511.17045]. The per-sport statistics further indicate the distribution of supervision. TableTennis comprises 50 games, 780 clips, 170\,027 frames, duration 3\,878 s, 19\,495 ball annotations, and 6\,648 racket annotations. Tennis comprises 431 games, 431 clips, 150\,399 frames, duration 4\,285 s, 21\,544 ball annotations, and 7\,395 racket annotations. Badminton comprises 461 games, 461 clips, 114\,753 frames, duration 4\,592 s, 23\,003 ball annotations, and 10\,578 racket annotations. In total, the benchmark contains \(N_{\rm frames}=435\,179\), \(N_{\rm B}=64\,042\), and \(N_{\rm R}=24\,621\) [2511.17045]. ## 2. Tasks, losses, and evaluation protocol RacketVision defines three interconnected tasks—ball tracking, racket pose estimation, and trajectory forecasting—which form a pipeline from low-level perception to high-level prediction. For ball tracking, the problem is: given a frame \(I_t\), or a short video \(\{I_{t-N},\dots,I_t\},N=5\), predict \((\hat x_t,\hat y_t)\) and visibility \(\hat v_t\). The loss is $$ \mathcal{L}_{\rm track} = \lambda_{\rm coord}\|(x_t,y_t)-(\hat x_t,\hat y_t)\|_2^2 \;+\;\lambda_{\rm vis}\,\mathrm{BCE}(v_t,\hat v_t), $$ and the reported metrics are Precision, Recall at \(d<\mathrm{thresh}\), Mean Distance Error (MDE), and mAP@50 [2511.17045]. For racket pose estimation, the task is: given \(I_t\), predict the bounding box and keypoints \(\{(\hat x_i,\hat y_i)\}_{i=1}^5\). The loss is $$ \mathcal{L}_{\rm pose}=\sum_{i=1}^5\|(x_i,y_i)-(\hat x_i,\hat y_i)\|_2^2, $$ with evaluation by PCK@0.2, MPJPE, mOKS, NME, and Detection mAP@50. The formulation makes the racket an articulated object rather than a coarse detection target, which is consequential for downstream conditioning and interaction analysis [2511.17045]. For ball trajectory forecasting, the problem is: given history of ball coordinates \(\{p_{t-N+1},\dots,p_t\}\), and optionally racket poses \(\{r_{t-N+1},\dots,r_t\}\), predict future \(\{\hat p_{t+1},\dots,\hat p_{t+M}\}\). The loss is $$ \mathcal{L}_{\rm traj} = \frac1M\sum_{k=1}^M\|\hat p_{t+k}-p_{t+k}\|_2^2, $$ and the metrics are ADE and FDE. Two settings are standardized: Short with \(N=20,M=5\), and Long with \(N=80,M=20\). Splits are by game: ≈ 70% games for training, 10% for validation, and 20% for testing—each sport independently and a combined multi-sport split for cross-sport experiments. All evaluations use held-out clips and report average metrics across sports [2511.17045]. ## 3. Baseline architectures and the fusion result The baseline stack spans all three tasks. Ball tracking backbones are RTMDet, YOLO11, WASB, and TrackNetV3. Racket pose estimation uses RTMPose with RTMDet for initial detection. Trajectory prediction backbones are a 2-layer LSTM and a 4-layer self-attention Transformer. For multimodal fusion, the benchmark formalizes CrossAttention with ball embeddings \(B\in\mathbb R^{N\times d}\) and racket embeddings \(R\in\mathbb R^{N\times d}\): $$ Q=W_QB,\quad K=W_KR,\quad V=W_VR, $$ $$ \alpha=\mathrm{softmax}\!\bigl(\tfrac{QK^\top}{\sqrt d}\bigr),\qquad F=\alpha V. $$ The compared settings are Ball-only, Ball+Racket via Concatenation, and Ball+Racket via CrossAttention [2511.17045]. For ball tracking, the best reported configuration is MS-TrackNetV3 with #F=4 and background modeling. Its results are: TableTennis, Prec. 0.924, Rec. 0.762, MDE 3.41, mAP 71.1; Tennis, Prec. 0.945, Rec. 0.880, MDE 1.96, mAP 81.9; Badminton, Prec. 0.915, Rec. 0.865, MDE 1.54, mAP 83.1. The benchmark further states that multi-sport training raises mAP by +19.2% in Tennis and +14.6% in Badminton versus single-sport, that background modeling cuts MDE by ≈ 54–61%, and that multi-frame input \((\#F=4)\) improves recall and mAP but may slightly jitter final localization [2511.17045]. For racket pose estimation, MS RTMPose reports: TableTennis, PCK@0.2 81.8, MPJPE 9.71, mOKS 0.498, NME 0.254, mAP 78.4; Tennis, 89.6, 5.34, 0.630, 0.223, 79.4; Badminton, 88.5, 5.00, 0.668, 0.235, 75.5. Multi-sport training boosts PCK@0.2 by +6–6.4% over SS. Structural keypoints—top, bottom, handle—exceed 92% PCK, whereas side keypoints lag at 65–80% due to occlusion and blur [2511.17045]. The trajectory forecasting experiments supply the benchmark’s main caution against naive multimodal integration. In the short setting \((N=20\rightarrow M=5)\), the LSTM on TableTennis gives Ball-only \((41.9 / 64.0)\) in ADE/FDE, Ball+Concat \((58.1 / 86.6)\), and Ball+CrossAttn \((38.3 / 60.4)\). In the long setting \((N=80\rightarrow M=20)\), the same model gives Ball-only \((113.9 / 184.3)\), Ball+Concat \((139.9 / 198.9)\), and Ball+CrossAttn \((101.3 / 161.3)\). The stated interpretation is that naive concatenation degrades performance because racket features are noisy when the ball is in flight, whereas CrossAttention picks out “hit” frames and yields consistent improvement versus ball-only; the overall ADE/FDE gains remain modest because many segments lack racket contact [2511.17045]. ## 4. Technique recognition and tactical analysis A related line of work positions RacketVision beyond ball-and-racket kinematics and into stroke semantics. In “ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis,” the technical report describes ViSTec as a two-stage action-perception system designed to recognize fine-grained stroke techniques in broadcast racket-sport videos and inject high-level tactical knowledge. It comprises a video-encoding backbone based on VideoMAE adapted to extremely short “strokes,” a temporal segmentation (“seg”) module that detects instantaneous stroke events, and a two-part classification stage (“cls” + “grh”) that predicts a stroke’s technique from visual features, then refines it by consulting a learnable tactical graph [2402.15952]. The graph component explicitly models strategic knowledge in stroke sequences. Its node set is \(V_T=\{v_{tec_0},\dots,v_{tec_{m-1}},v_{null}\}\), with directed edges connecting every node to every other node except no incoming edges into “null.” At inference, the previous stroke label selects a transition vector \(W_{tec_p}\), both logits and graph scores are min–max normalized into \([0,1]\), and the combined posterior is $$ \hat P_c=\mathrm{Softmax}(\mathrm{MinMax}(z_i)+\alpha\cdot W_{tec_p}). $$ Adaptive updates depend on whether the predicted label matches the ground truth and on an entropy-based uncertainty term $$ U = 1 - \mathrm{Entropy}(\mathrm{Softmax}(z_i))/\mu,\qquad \mu=\log m. $$ The total loss per video is \(L_{\rm total}=L_{\rm seg}+\lambda\cdot L_{\rm cls}\), with \(\lambda=1.0\), and no additional regularizers were introduced beyond standard weight decay [2402.15952]. The associated dataset is constructed from broadcast videos of World Table Tennis: 18 televised games and 4000 rally clips, automatically segmented via scoreboard change. Stroke timestamps are labeled at frame level by experts. Technique labels are \(\{Serve, Topspin, Short, Block, Push, Flick, Smash, Others\}\), assigned per stroke by professional table-tennis coaches. Sliding windows of 3 consecutive technique labels form basic tactical units for analysis and initialize graph edge frequencies. On this setup, Full ViSTec reports \(79.3, 79.2, 78.5\) on F1@10,25,50; \(83.5\) Acc.; and \(76.3\) Edit, with gains of +2.5% F1@50 and +6.4% Acc over the best baseline, and inference speed of 39.3 fps on a single A100 GPU. Case studies with experts from the Chinese national table tennis team include t-SNE clustering of the 1280-d stroke features and a tactical analysis in which \((Serve, Short, Topspin)\) achieved \(>70\%\) win-rate, while \((Serve, Short, Short)\) drops win-rate to \(\sim 43\%\). This suggests an extension from RacketVision’s benchmarked perception tasks to technical profiling, tactical pattern discovery, and strategy mining [2402.15952]. ## 5. Ball sensing, blur, impact, and 3D tracking Several neighboring studies address failure modes that are central in racket-sport perception. “BlurBall: Joint Ball and Motion Blur Estimation for Table Tennis Ball Tracking” starts from the observation that existing labeling conventions mark the ball at the leading edge of the blur streak. BlurBall instead places the ball center at the midpoint of the straight-line blur streak and annotates two additional blur attributes per frame: orientation \(\theta\) and half-length \(l\). The benchmark released in that work contains 64 119 frames total, with 51 423 training frames and 12 696 test frames, mean inter-frame displacement ≈ 19 px, blur ratio ≈ 62%, and a half-length distribution concentrated around 5 px with maximum observed \(l=73\) px. The proposed model uses HRNet with 1–3 sequential frames stacked along channels and inserts Squeeze-and-Excitation blocks after each stage. Its reported results include, for 1-step + SE at \(\delta=0.5\), F1 \(=96.52\), Acc \(=93.47\), AP \(=98.23\), blur length MAE \(=1.6\) px, and angle MAE \(\approx 6.9^\circ\); tuning \(\delta=0.7\) gives F1 \(=97.17\), Acc \(=94.75\), AP \(=97.34\), blur length MAE \(=1.2\) px, and angle MAE \(\approx 7.3^\circ\). The paper also reports that position-only trajectory fitting yields MAE \(=84.4\pm136.6\) px, whereas position+blur yields \(53.0\pm87.1\) px, described as nearly 40 % reduction in average error and lower variance [2509.18387]. “Locating Tennis Ball Impact on the Racket in Real Time Using an Event Camera” addresses a different sensing regime. Its input is a continuous stream of events \(e_k=(x_k,y_k,p_k,t_k)\), which are accumulated into overlapping event packets and processed in three stages: time-range detection of a swing, impact-timing detection via PATS, and contour extraction of ball and racket. PATS is defined through positive-polarity structure before a reference time and negative-polarity structure after that time: $$ \rho_t = \sum_{x,y} \Bigl|\bigl(\mathbf{F}_{prev}^+ * \mathbf{G}_{prev}\bigr)_{xy}\Bigr| \;\times\; \Bigl|\bigl(\mathbf{F}_{next}^- * \mathbf{G}_{next}\bigr)_{xy}\Bigr|, $$ with a focal-time weighting \(\varphi(\cdot)\) used to emphasize mid-interval events. On 50 impacts from 5 players with 10 hits each, the paper reports time-range detection success 49/50 \((98\%)\), impact timing success 46/49 \((94\%)\), mean absolute timing error \(\bar\Delta t\approx350\)\,μs \(\pm\)250 μs, contour-detection success 24/26 \((92\%)\) without direct sunlight and 3/20 \((15\%)\) under sunlight, localization accuracy \(\Delta u<12.1\%\) pt and \(\Delta v<9.1\%\) pt implying \(\Delta\mathrm{pos}<15\) mm, and computation time 1.4–2.0 s per impact on a laptop with i7-12700H + RTX3080Ti. The stated limitations are direct sunlight, multi-colored rackets or balls, and the fact that the current 2 D method cannot recover racket tilt [2506.08327]. “Physics-Guided Fusion for Robust 3D Tracking of Fast Moving Small Objects” describes a RacketVision system for high-precision 3D tracking of fast-moving balls in racket sports. The system uses a ZED 2i stereo RGB-D camera mounted ≈35 ft from court, YOLOv8 fine-tuned on a racquetball dataset, and a physics-based tracker with state \(s_t=[p_t;v_t]\). Prediction follows kinematic motion equations, $$ p_{t+1}=p_t+v_t\Delta t+\tfrac12 a\Delta t^2,\qquad v_{t+1}=v_t+a\Delta t, $$ and measurement acceptance is gated by \(\|p_t^{meas}-p_t^{pred}\|_2\le Th\). Outliers are rejected when \(\|yolo\_pos-pred\_pos\|>Th\) or depth is NaN/infinite, while missed detections are handled by continuing with the physics prediction. The racquetball RGB-D dataset contains 12 sequences × 5 min each, 80 000+ frames at 720p/1280p + depth, and 10 000 frames manually labeled with 2D boxes using an 80/20 train/val split. Reported results include Scenario 1 with DeepOCSORT 22.97 / 0.69 versus Ours 8.42 / 0.22 in ADE/AMD, corresponding to 63% ADE reduction; Scenario 2 with DeepOCSORT 31.31 / 0.91 versus Ours 26.70 / 0.81, corresponding to 15% ADE reduction; and overall up to 90% ADE reduction vs. StrongSORT and 63–73% vs. best KF-based tracker. End-to-end throughput is sustained ∼40 FPS, with YOLOv8 at 1248×704 running at 24.5 ms or 41 FPS and depth query plus physics update taking <1 ms/frame [2510.20126]. ## 6. Perception-to-action integration and future directions RacketVision also appears as an onboard perception subsystem inside “CyboRacket: A Perception-to-Action Framework for Humanoid Racket Sports.” In that framework, the architecture is organized into three tightly-coupled layers: an Onboard Perception Loop at 100 Hz, an Interception Planning Loop at ≈ 20–30 Hz, and a Command Loop at ≥ 100 Hz. The onboard loop handles RGB–D acquisition, 2D detection, temporal tracking, 3D back-projection, and an EKF for state estimation and short-horizon forecasting. The camera is an Intel RealSense D455, or 100 Hz RGB–D head-mounted camera, and the state vector is $$ \mathbf{x}_t = [p_{x,t},p_{y,t},p_{z,t},v_{x,t},v_{y,t},v_{z,t}]^\top, $$ with dynamics $$ \mathbf{p}_{t+1} = \mathbf{p}_t + \mathbf{v}_t\Delta t + \tfrac12\mathbf{a}(\mathbf{v}_t)\Delta t^2,\qquad \mathbf{v}_{t+1} = \mathbf{v}_t + \mathbf{a}(\mathbf{v}_t)\Delta t, $$ where \(\mathbf{a}(\mathbf{v})=[0,0,-g]^\top-k_1\mathbf{v}-k_2\|\mathbf{v}\|\mathbf{v}\). Outlier rejection uses a Mahalanobis gate on the innovation, and bounce handling applies \(v_z^+=-e\,v_z^-\) when \(\hat p_z