- The paper introduces AdaPT, a decoupled planner–tracker framework that learns Nadal-, Federer-, and Djokovic-style rallying and serving from broadcast video and deploys them on humanoid robots.
- AdaPT combines adaptive speed control, motion generation, keyframed backswing tracking, and toss guidance, achieving up to 96.3% simulated rally success, near-100% simulated serve success, and 66.7–86.7% real serving success.
- The results show that accurate post-bounce prediction, ball-motion randomization, and coordinated planner–tracker adaptation are critical for narrowing the sim-to-real gap, while camera-only perception and long-range locomotion remain open challenges.
AdaPT is a framework for learning professional tennis playing styles on humanoid robots directly from broadcast videos, developed by a team from Noitom Robotics, Shanghai AI Laboratory, Dobot Robotics, and Shanghai Jiao Tong University (2608.20087). The system addresses a problem that prior humanoid ball-sports work largely ignored: most systems optimize task success (hitting, ball return) without reproducing professional athletic styles, even though human sports biomechanics research indicates such styles encode structured whole-body coordination important for efficient force generation and rapid recovery. AdaPT reproduces the distinctive rallying and serving styles of Rafael Nadal, Roger Federer, and Novak Djokovic on the Unitree G1, and demonstrates in-the-wild serving on the full-size (~1.7 m) Dobot Atom without motion capture.
Motivation and framework design
The central architectural choice is a decoupled planning-and-tracking formulation: a high-level planner generates stylistic kinematic motions independently of environment dynamics, while a low-level tracker executes them on hardware. This design is grounded in an empirical distinction between the two tasks. Rally motions are diverse and reactive—formally analogous to self-play against an unknown opponent—whereas serving motions are self-initiated, low-diversity, and largely independent of external dynamics, a claim the authors support with a t-SNE analysis of motion distributions. Accordingly, AdaPT uses different machinery for each task: an MVAE-based motion generator with a latent-conditioned planner for rally (built on Vid2Player3D), and a residual tracker with a speed-adaptation planner for serve (built on AdaMimic).
The motion data pipeline is lightweight and scalable. Broadcast clips of the three players (roughly 179–267 minutes and 5,264–5,664 clips per player) are reconstructed into SMPL poses using GVHMR, retargeted with GMR, and refined into physically plausible motions by a general tracker. Wrist joints are corrected for racket-specific strokes, and randomized wrist perturbations diversify swing directions. Clips are annotated with player identity, stroke type, spin type, and ball contact timing; serving motions additionally carry ball release timing for toss trajectory estimation. The framework also generalizes to professional MoCap data, demonstrating source-agnostic style acquisition.
Speed adaptation as the sim-to-real mechanism
The key technical contribution is explicit speed adaptation on both sides of the planner–tracker interface. During tracker training, the reference motion is temporally blended,
q^tα=(1−α)q^t−1+αq^t,
with α sampled uniformly from a range, exposing the policy to randomized execution speeds. On the planning side, the rally planner outputs both an MVAE latent zt and a speed signal αt conditioned on the future ball trajectory; the serve planner outputs αt alone, since serving requires no generative motion diversity. This dual mechanism addresses a specific failure mode: tracking degradation from imperfect actuator modeling is partially overlooked by autoregressive planning, and perception noise compounds the error over long horizons. The planner's learned speed adapter lets it compensate for the tracker's realized capabilities rather than its nominal ones.
For serving, two additional mechanisms are notable. A keyframing scheme imposes a sparse tracking reward (50× the standard weight) on the racket and wrist poses at the deepest backswing, preventing the policy from converging to a local optimum that skips the backswing while still achieving the strike. Tossing guidance models the toss as a gravity-only parabola between release and strike positions and rewards trajectory adherence; the authors show this is not merely stylistic—without it, the policy produces low tosses that simplify interception but degrade landing control, whereas the guidance enables higher tosses, stronger racket acceleration, wrist pronation, and more accurate serve placement.
Simulation results
All policies are trained with PPO in Mjlab on 4,096 parallel environments. The simulation comparisons establish several findings. RL-Scratch (PPO without motion data) achieves high rally hit rates but large FID deviations from human stroke dynamics, and fails at serving for two of three players, with the third (the Nadal-style data) exhibiting reward hacking—keeping the racket attached to the ball from a stable initial posture. AMP produces human-like strokes but distorted footwork, and can underperform RL-Scratch in serving. Coupled motion-prior methods (PULSE, NCP) achieve responsive control but entangle the latent with robot proprioception, yielding worse style fidelity. Vid2Player3D's decoupled design improves FID substantially but costs task performance. AdaPT achieves the best combined profile: e.g., 96.3% rally success with EFID of 6.1 for the second player, and near-100% serve success with the lowest MPKPE across players. Ablations confirm that both the adaptive tracker and adaptive planner are individually beneficial and complementary.
A particularly useful empirical finding is a structural trade-off: decoupled planning–tracking architectures benefit from longer-horizon future ball trajectory observations, while tightly coupled architectures (PULSE/NCP-style) rely on current observations for responsiveness at the cost of motion naturalness. This explains why LATENT, a PULSE-based humanoid tennis system, suffices with only current ball position and velocity.
Real-robot results and sim-to-real analysis
The real-world gap is substantial and quantified. Vid2Player3D drops from ~82% simulated rally success to 28% (forehand) and 8–28% (backhand) on the G1, with net clearance rates falling to 0% in some settings. AdaPT's adaptation mechanisms recover much of this: for example, 64% forehand and 48% backhand hit rates for the second player, versus 24% and 8% for Vid2Player3D. Neither component alone suffices—the adaptive tracker alone cannot fix timing errors, and the adaptive planner alone remains sensitive to control noise. In serving, AdaPT achieves 66.7–86.7% real success across players, generally exceeding DeepMimic on success and landing accuracy at 6 m and 9 m distances, while preserving player-specific styles.
The sim-to-real analysis yields several practical insights. Post-bounce trajectory prediction errors correlate more strongly with rally failure than pre-bounce errors, making bounce modeling and system identification (calibrated restitution and horizontal damping from 20 MoCap trajectories) critical. Domain randomization ablations for serving are stark: removing all randomization drops success from 90% to 20% with a 40% fall rate; removing only ball perturbation drops it to 40%. Ball trajectory noise injection is essential even under MoCap—rally performance drops from 80% to 55% without it. For perception, YOLO-based detection with stereo triangulation consistently outperforms the HSV-plus-ROI approach used in ESTHER, and for serving YOLO performs comparably to MoCap; however, the multi-camera vision setup exhibits larger localization noise and temporal jitter than MoCap, which degrades rally performance because the adaptive mechanism depends on accurate trajectory prediction. A policy-switching scheme between a standing policy and a rally policy addresses MVAE drift when no ball is present.
Limitations and open questions
The paper is candid about its constraints. The system still relies primarily on motion capture for robot root and ball positions; camera-based rally localization introduces noise that materially degrades performance, and a robust MoCap-free pipeline remains unresolved. Return quality is less consistent in the real world than in simulation, raising difficulties for human–humanoid play. Autoregressive motion generation limits locomotion: hit rates drop at distant ball positions, a weakness less pronounced in coupled NCP/PULSE architectures. The responsiveness–style-fidelity trade-off is acknowledged as requiring better learning algorithms rather than being solved. On hardware, the Unitree Dex-3 dexterous hand is too heavy for professional serving motions (the authors instead demonstrate amateur serving with it at 90% success over 10 trials), and more capable wrist actuation is identified as necessary for stable dexterous serves.
Conclusion
AdaPT demonstrates that professional athletic styles can be transferred from broadcast video to real humanoid robots for both rallying and serving, and that explicit speed adaptation on both the tracker and planner sides is an effective mechanism for closing the sim-to-real gap in decoupled planning–tracking architectures. Its empirical characterization of the coupling trade-off, the criticality of post-bounce prediction and toss guidance, and the quantified sensitivity to domain randomization provide concrete design guidance for future humanoid ball-sports systems. The open problems—robust camera-only perception, MoCap-free localization, and reconciling long-horizon locomotion coverage with autoregressive stylistic generation—remain the primary barriers to fully in-the-wild humanoid tennis.