Caltech Tennis Dataset (CalTennis)

• CalTennis is a large-scale, multi-view tennis dataset with over 11 million HD frames designed for monocular-to-3D human pose estimation.
• The dataset uses synchronized iPhone recordings and automated calibration protocols to capture expert athletic motion in unconstrained, real-world settings.
• It introduces novel metrics for footwork and stability, revealing challenges such as translation drift and inconsistent foot-contact in current pose estimation models.

The Caltech Tennis Dataset (CalTennis) is a large-scale, multi-view video corpus and benchmark for evaluating monocular-to-3D pose estimation in the wild, with a focus on tennis motion. CalTennis comprises more than 11 million frames (51 hours) of HD tennis video captured from 40 players under unconstrained conditions using synchronized consumer mobile devices. It is notable for being an order of magnitude larger than previous in-the-wild human motion datasets and for providing the first large-scale, publicly available multi-view resource of expert athletic motion, enabling label-free evaluation of 3D pose estimation algorithms through multi-view consistency (Demler et al., 18 Jun 2026).

1. Dataset Composition and Scale

CalTennis contains 11.03 million frames of 1920×1080 video at 60 Hz, spanning approximately 51 hours of tennis activity. The participant cohort consists of 40 players ranging from collegiate to recreational skill levels. Each recording session uses 2–6 iPhone 14+ devices, rigidly mounted on 1.65 m MagSafe tripods placed at fixed “corners” around each half-court (10.98 m × 11.88 m) with approximately 12 m spacing, ensuring comprehensive multi-side coverage of the player.

A comparative summary is reproduced below:

Dataset	Multi-view?	Frames (M)	Depth range (m, 10–90%)	Pose-space coverage (%)	Hardware cost (k$$)</th> </tr> </thead><tbody><tr> <td>3DPW</td> <td>no</td> <td>0.05</td> <td>3.1–7.4</td> <td>58</td> <td>21</td> </tr> <tr> <td>EMDB</td> <td>no</td> <td>0.11</td> <td>1.9–2.7</td> <td>60</td> <td>31</td> </tr> <tr> <td><a href="https://www.emergentmind.com/topics/ring-imaging-cherenkov-rich-detector" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">RICH</a></td> <td>no</td> <td>0.54</td> <td>4.2–4.7</td> <td>62</td> <td>100</td> </tr> <tr> <td>Human3.6M</td> <td>no</td> <td>1.47</td> <td>4.5–5.8</td> <td>89</td> <td>150</td> </tr> <tr> <td>SportsPose</td> <td>no</td> <td>1.50</td> <td>0.7–3.4</td> <td>47</td> <td>25</td> </tr> <tr> <td><strong>CalTennis</strong></td> <td>yes</td> <td>11.03</td> <td>13.4–16.7</td> <td>85</td> <td>2</td> </tr> </tbody></table></div> <p>CalTennis exceeds prior in-the-wild datasets by at least 10x in frame count and records 3x more data than the largest MOCAP benchmark. The depth variability (most poses 13.4–16.7 m) greatly surpasses that of Human3.6M (~5 m), and the normalized pose-space coverage (Shannon-entropy over 500 PCA clusters) is 85<h2 class='paper-heading' id='data-acquisition-and-calibration-protocol'>2. Data Acquisition and Calibration Protocol</h2> <p>All capture is conducted by Caltech players themselves using standard iPhones, eliminating additional hardware costs. Each phone is fixed on a tripod at canonical orientations relative to the court. The protocol is IRB-approved, and all faces are blurred in post-processing.</p> <p>Calibration is performed automatically:</p> <ul> <li><strong>Spatial calibration</strong>: Detect known tennis court line intersections; intrinsics$$K^i$are extracted from iPhone metadata; extrinsics$(R^i, T^i)$are estimated by minimizing</li> </ul> <p>$\min_{R^i,\,T^i}\sum_{k=1}^n \|\pi(R^i\hat P_k + T^i;K^i) - \hat p_k\|^2,$</p> <p>where$\hat P_k \in \mathbb{R}^3$are court intersection points and$\hat p_k \in \mathbb{R}^2$$are detected image coordinates.</p> <ul> <li><strong>Model frame lifting</strong>: For a per-camera pose translation$$\tau_t^i$$, transformation to world coordinates uses</li> </ul> <p>$$T_{i}^{\mathrm{model}\to W} = \begin{bmatrix} R^{i\top} & -R^{i\top}T^i \ 0 & 1 \end{bmatrix}, \qquad \tilde{\tau}_t^i = T_{i}^{\mathrm{model}\to W}[\tau_t^i; 1].$$</p> <ul> <li><strong>Temporal alignment</strong>: Due to coarse iPhone timestamp quantization (granularity 1 s, yielding up to ±1000 ms offset), a single global offset$$\Delta t$$is estimated by grid search to minimize cross-view pose disagreements:</li> </ul> <p>$$\Delta t^* = \arg\min_{\Delta t} \sum_{t,p} \|\tau^i_{t} - \tau^j_{t+\Delta t}\|^2.$$</p> <p>Model outputs are linearly interpolated in time as needed.</p> <h2 class='paper-heading' id='benchmarking-tasks-and-evaluation-procedures'>3. Benchmarking Tasks and Evaluation Procedures</h2> <p>The <a href="https://www.emergentmind.com/topics/conditional-image-retrieval-core-benchmark" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">core benchmark</a> task is monocular-to-3D human pose estimation: predict 3D body pose, orientation, translation, and <a href="https://www.emergentmind.com/topics/smpl-x" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">SMPL-X</a> shape parameters from a single HD video in a fixed, metric, court-centric frame. Ground-truthing is conducted without labels by evaluating multi-view consistency.</p> <p>Five state-of-the-art monocular 3D pose systems are benchmarked:</p> <ul> <li>TRAM (Daniilidis et al. 2024)</li> <li>GVHMR (Xiaowei et al. 2024)</li> <li>GENMO (Li et al. 2025)</li> <li><a href="https://www.emergentmind.com/topics/weighted-histogram-analysis-method-wham" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">WHAM</a> (Shin et al. 2024)</li> <li>PromptHMR (Wang et al. 2025)</li> </ul> <p>Standard multi-view disagreement metrics include:</p> <ul> <li><strong>Translation error</strong> ($$E_\mathrm{trans}$):</li> </ul> <p>$(R^i, T^i)$0 Pose error ($(R^i, T^i)$1, per-joint L2 after pelvis centering): $(R^i, T^i)$2 MPJPE, PA-MPJPE, and PVE. Multi-view consistency results (first 5 M frames): Model Trans (mm) Pose (mm) MPJPE PA-MPJPE Foot-Vel (m/s) Foot-Ht (mm) Stability PromptHMR 942 105 1785 84 3.23 70 25 WHAM 2664 106 2675 119 0.72 150 44 GVHMR 3587 109 1066 88 2.49 60 21 TRAM 2340 115 958 91 6.65 80 33 GENMO 2560 110 1020 91 4.40 60 16 No single method achieves state-of-the-art across all metrics. PromptHMR leads for translation, pose, and PA-MPJPE; WHAM for foot-velocity; GENMO for foot-height and stability. All methods show substantially less consistency than on established datasets (3DPW, EMDB, RICH). 4. New Metrics for Footwork and Stability Two new cross-view performance metrics address failures under athletic movement: Footwork errors capture foot-contact and "skating" inconsistencies between views: $(R^i, T^i)$3 with $(R^i, T^i)$4 foot-joint velocity and $(R^i, T^i)$5 its height. Stability error quantifies disagreement in support polygon and static balance (Zero-Moment-Point approach): $(R^i, T^i)$6 $(R^i, T^i)$7 where $(R^i, T^i)$8 is the convex hull of grounded foot joints and $(R^i, T^i)$9 is the lateral center of mass. These metrics reveal systematic foot-contact, skating, and support failures overlooked by standard per-joint errors. 5. Empirical Assessment and Failure Modes Joint-angle recovery (relative pose) is now accurate to approximately 11 cm multi-view disagreement, and temporal smoothness is generally strong. However, depth and translation exhibit substantial "pose drifting" errors (0.9–3.6 m RMS) along the camera-subject axis. There are frequent failures in foot-contact (skating or floating), and large variability in estimated foot heights between views. Body shape consistency is a major challenge: for the same subject, estimated SMPL-X shape parameters ($\min_{R^i,\,T^i}\sum_{k=1}^n \|\pi(R^i\hat P_k + T^i;K^i) - \hat p_k\|^2,$0) can differ by ±5–10 cm in limb lengths or proportions between views or methods. Qualitative analyses show that disagreement spikes during fast motion, occlusion, or at larger depths, whereas stationary, fully visible frames produce low multi-view discrepancies. No evaluated model demonstrates robust or consistent depth or foot contact estimation (Demler et al., 18 Jun 2026). 6. Research Directions and Applications CalTennis highlights several research opportunities: Explicit depth modeling (e.g., learned depth priors, improved focal length calibration) to address translation drift. Robust ground-contact constraints (e.g., foot friction or anti-skating penalties) to reduce footwork inconsistency. Cross-temporal and cross-view body-shape regularization (e.g., video-level or multi-view shape estimation). Training algorithms with synthetic multi-view consistency losses for label-free self-supervision. Extending the low-cost, consumer-phone capture paradigm to other sports, surfaces, and participant populations to broaden benchmark generality. Hybrid benchmarks combining phone video with lightweight IMUs for richer supervision. The dataset, protocols, and evaluation code are publicly released to support research into pose estimation and action analysis under realistic and challenging conditions (Demler et al., 18 Jun 2026). Topic to Video (Beta) No one has generated a video about this topic yet. Whiteboard No one has generated a whiteboard explanation for this topic yet. Follow Topic Get notified by email when new papers are published related to Caltech Tennis Dataset (CalTennis). Continue Learning How does the multi-view approach in CalTennis enhance the evaluation of 3D pose estimation methods?  What are the specific calibration and temporal alignment techniques employed in the dataset?  How do the new metrics for footwork and stability compare with traditional pose error metrics?  What challenges does CalTennis reveal about depth estimation and translation in sports-related pose estimation?  Find recent papers about monocular-to-3D pose estimation in sports.  Related Topics THETIS Dataset: Multimodal Tennis Analysis  Million-Scale Pose Dataset  TraceForge-123K Dataset  CameraHMR: Perspective-Aware 3D Recovery  CameraHMR: Monocular Human Mesh Recovery  VRRPI-Bench: 3D Pose Benchmark for VLMs  DexYCB: Multi-View Hand–Object Pose Dataset  3D-CustomBench: Modular 3D Vision Benchmark  Nymeria: Multimodal Egocentric Dataset  Footwork Metrics: Quantitative Gait Analysis  Content Overview References Topic to Video Whiteboard Follow Topic Continue Learning Related Topics Stay informed about trending AI papers: About Labs API Email Digest Chrome Extension RSS Terms Privacy Contact Twitter Discord