--- title: Caltech Tennis Dataset (CalTennis) url: https://www.emergentmind.com/topics/caltech-tennis-dataset-caltennis type: topic --- # Caltech Tennis Dataset (CalTennis) The Caltech Tennis Dataset (CalTennis) is a large-scale, multi-view video corpus and benchmark for evaluating monocular-to-3D pose estimation in the wild, with a focus on tennis motion. CalTennis comprises more than 11 million frames (51 hours) of HD tennis video captured from 40 players under unconstrained conditions using synchronized consumer mobile devices. It is notable for being an order of magnitude larger than previous in-the-wild human motion datasets and for providing the first large-scale, publicly available multi-view resource of expert athletic motion, enabling label-free evaluation of 3D pose estimation algorithms through multi-view consistency [2606.20542]. ## 1. Dataset Composition and Scale CalTennis contains 11.03 million frames of 1920×1080 video at 60 Hz, spanning approximately 51 hours of tennis activity. The participant cohort consists of 40 players ranging from collegiate to recreational skill levels. Each recording session uses 2–6 iPhone 14+ devices, rigidly mounted on 1.65 m MagSafe tripods placed at fixed “corners” around each half-court (10.98 m × 11.88 m) with approximately 12 m spacing, ensuring comprehensive multi-side coverage of the player. A comparative summary is reproduced below: | Dataset | Multi-view? | Frames (M) | Depth range (m, 10–90%) | Pose-space coverage (%) | Hardware cost (k$) | |-------------------|-------------|------------|-------------------------|------------------------|--------------------| | 3DPW | no | 0.05 | 3.1–7.4 | 58 | 21 | | EMDB | no | 0.11 | 1.9–2.7 | 60 | 31 | | RICH | no | 0.54 | 4.2–4.7 | 62 | 100 | | Human3.6M | no | 1.47 | 4.5–5.8 | 89 | 150 | | SportsPose | no | 1.50 | 0.7–3.4 | 47 | 25 | | **CalTennis** | yes | 11.03 | 13.4–16.7 | 85 | 2 | CalTennis exceeds prior in-the-wild datasets by at least 10x in frame count and records 3x more data than the largest MOCAP benchmark. The depth variability (most poses 13.4–16.7 m) greatly surpasses that of Human3.6M (~5 m), and the normalized pose-space coverage (Shannon-entropy over 500 PCA clusters) is 85%, compared to at most 62% for prior real-world datasets [2606.20542]. ## 2. Data Acquisition and Calibration Protocol All capture is conducted by Caltech players themselves using standard iPhones, eliminating additional hardware costs. Each phone is fixed on a tripod at canonical orientations relative to the court. The protocol is IRB-approved, and all faces are blurred in post-processing. Calibration is performed automatically: - **Spatial calibration**: Detect known tennis court line intersections; intrinsics $K^i$ are extracted from iPhone metadata; extrinsics $(R^i, T^i)$ are estimated by minimizing $$ \min_{R^i,\,T^i}\sum_{k=1}^n \|\pi(R^i\hat P_k + T^i;K^i) - \hat p_k\|^2, $$ where $\hat P_k \in \mathbb{R}^3$ are court intersection points and $\hat p_k \in \mathbb{R}^2$ are detected image coordinates. - **Model frame lifting**: For a per-camera pose translation $\tau_t^i$, transformation to world coordinates uses $$ T_{i}^{\mathrm{model}\to W} = \begin{bmatrix} R^{i\top} & -R^{i\top}T^i \\ 0 & 1 \end{bmatrix}, \qquad \tilde{\tau}_t^i = T_{i}^{\mathrm{model}\to W}[\tau_t^i; 1]. $$ - **Temporal alignment**: Due to coarse iPhone timestamp quantization (granularity 1 s, yielding up to ±1000 ms offset), a single global offset $\Delta t$ is estimated by grid search to minimize cross-view pose disagreements: $$ \Delta t^* = \arg\min_{\Delta t} \sum_{t,p} \|\tau^i_{t} - \tau^j_{t+\Delta t}\|^2. $$ Model outputs are linearly interpolated in time as needed. ## 3. Benchmarking Tasks and Evaluation Procedures The core benchmark task is monocular-to-3D human pose estimation: predict 3D body pose, orientation, translation, and SMPL-X shape parameters from a single HD video in a fixed, metric, court-centric frame. Ground-truthing is conducted without labels by evaluating multi-view consistency. Five state-of-the-art monocular 3D pose systems are benchmarked: - TRAM (Daniilidis et al. 2024) - GVHMR (Xiaowei et al. 2024) - GENMO (Li et al. 2025) - WHAM (Shin et al. 2024) - PromptHMR (Wang et al. 2025) Standard multi-view disagreement metrics include: - **Translation error** ($E_\mathrm{trans}$): $$ E_{\rm trans} = \frac{1}{T\,P} \sum_{t,p} \|\tau^i_{p,t} - \tau^j_{p,t}\|_2 $$ - **Pose error** ($E_\mathrm{pose}$, per-joint L2 after pelvis centering): $$ E_{\rm pose} = \frac{1}{T\,P\,K} \sum_{t,p,k} \|J^{i}_{p,k,t} - J^{j}_{p,k,t}\|_2 $$ - **MPJPE, PA-MPJPE,** and **PVE**. Multi-view consistency results (first 5 M frames): | Model | Trans (mm) | Pose (mm) | MPJPE | PA-MPJPE | Foot-Vel (m/s) | Foot-Ht (mm) | Stability | |------------|------------|-----------|-------|----------|----------------|--------------|-----------| | PromptHMR | 942 | 105 | 1785 | 84 | 3.23 | 70 | 25 | | WHAM | 2664 | 106 | 2675 | 119 | 0.72 | 150 | 44 | | GVHMR | 3587 | 109 | 1066 | 88 | 2.49 | 60 | 21 | | TRAM | 2340 | 115 | 958 | 91 | 6.65 | 80 | 33 | | GENMO | 2560 | 110 | 1020 | 91 | 4.40 | 60 | 16 | No single method achieves state-of-the-art across all metrics. PromptHMR leads for translation, pose, and PA-MPJPE; WHAM for foot-velocity; GENMO for foot-height and stability. All methods show substantially less consistency than on established datasets (3DPW, EMDB, RICH). ## 4. New Metrics for Footwork and Stability Two new cross-view performance metrics address failures under athletic movement: - **Footwork errors** capture foot-contact and "skating" inconsistencies between views: $$ E_{\rm skate} = \frac{1}{T\,P\,K_f} \sum_{t,p,k} \|\mathbf v^{(i)}_{p,k,t} - \mathbf v^{(j)}_{p,k,t}\|_2, \qquad E_{\rm height} = \frac{1}{T\,P\,K_f} \sum_{t,p,k} |h^{(i)}_{p,k,t} - h^{(j)}_{p,k,t}| $$ with $\mathbf v_{p,k,t}$ foot-joint velocity and $h_{p,k,t}$ its height. - **Stability error** quantifies disagreement in support polygon and static balance (Zero-Moment-Point approach): $$ E^{(i)}_{\rm stab,p,t} = \begin{cases} \min_{q\in Q} \|\mathrm{CoM}_{xy} - q\|_2, & \text{if } \mathrm{CoM}_{xy} \notin Q \\ 0, & \text{otherwise} \end{cases} $$ $$ E_{\rm stab} = \frac{1}{T\,P} \sum_{t,p} |E^{(i)}_{\rm stab,p,t} - E^{(j)}_{\rm stab,p,t}| $$ where $Q$ is the convex hull of grounded foot joints and $\mathrm{CoM}_{xy}$ is the lateral center of mass. These metrics reveal systematic foot-contact, skating, and support failures overlooked by standard per-joint errors. ## 5. Empirical Assessment and Failure Modes Joint-angle recovery (relative pose) is now accurate to approximately 11 cm multi-view disagreement, and temporal smoothness is generally strong. However, depth and translation exhibit substantial "pose drifting" errors (0.9–3.6 m RMS) along the camera-subject axis. There are frequent failures in foot-contact (skating or floating), and large variability in estimated foot heights between views. Body shape consistency is a major challenge: for the same subject, estimated SMPL-X shape parameters ($\beta$) can differ by ±5–10 cm in limb lengths or proportions between views or methods. Qualitative analyses show that disagreement spikes during fast motion, occlusion, or at larger depths, whereas stationary, fully visible frames produce low multi-view discrepancies. No evaluated model demonstrates robust or consistent depth or foot contact estimation [2606.20542]. ## 6. Research Directions and Applications CalTennis highlights several research opportunities: - Explicit depth modeling (e.g., learned depth priors, improved focal length calibration) to address translation drift. - Robust ground-contact constraints (e.g., foot friction or anti-skating penalties) to reduce footwork inconsistency. - Cross-temporal and cross-view body-shape regularization (e.g., video-level or multi-view shape estimation). - Training algorithms with synthetic multi-view consistency losses for label-free self-supervision. - Extending the low-cost, consumer-phone capture paradigm to other sports, surfaces, and participant populations to broaden benchmark generality. - Hybrid benchmarks combining phone video with lightweight IMUs for richer supervision. The dataset, protocols, and evaluation code are publicly released to support research into pose estimation and action analysis under realistic and challenging conditions [2606.20542]. Source: https://www.emergentmind.com/topics/caltech-tennis-dataset-caltennis