TaskNPoint: How to Teach Your Humanoid to Hit a Backhand in Minutes

Abstract: How do we learn to hit a tennis backhand? Not from a thousand hours of tennis tournaments on TV - we work with a coach and practice. We argue this is also the right recipe for teaching dynamic skills to humanoid robots. This follows from a structural property of dynamic skills: the outcome is decided by a short, crucial portion of the trajectory - for a backhand, the ~20cm of racket travel around ball contact. Getting this interaction window right requires coordinating the whole motion, so that control, physics, and morphology act in concert. Learning thus reduces to mastering a handful of distinct actions and, for each, practicing until the window comes out right. To this end, we introduce TaskNPoint, a training protocol which makes the coach-learner division of labor explicit. The human coach contributes four inputs: a discrete set of skills (e.g. different shots), one demonstration per skill, identification of the interaction window, and the goal. Learning in a physically realistic simulation environment fills in each action trajectory and provides robustness to unmodeled events. Crucially, randomized target sampling during training lets a single demonstration generalize zero-shot to unseen goal locations. We test this approach on a Unitree G1 humanoid that hits forehands and backhands against balls thrown by a human, kicks incoming soccer balls, and picks and places boxes from novel locations. We find that learning is successful from short human video demonstrations and under an hour of training on a single GPU, with no per-task reward tuning.

• The paper introduces TaskNPoint, a coach-guided framework that transfers dynamic skills to humanoid robots using only seconds of demonstration video.
• It leverages explicit spatio-temporal goal abstraction, kinematic retargeting, and randomized training to enable rapid policy synthesis in simulation.
• Empirical results show high generalization rates and scalability across dynamic tasks, achieving over 90% success in sports and manipulation scenarios.

TaskNPoint: Goal-Conditioned Skill Acquisition for Humanoid Robots from Minimal Human Video

Overview and Motivation

TaskNPoint presents an explicit, coach-guided framework for efficiently transferring dynamic loco-manipulation skills—such as tennis backhands, soccer kicks, and box pick-and-place—to humanoid robots, leveraging only seconds of human demonstration video per motion. Unlike data-hungry, reward-engineering-dependent, or end-to-end imitation learning frameworks, TaskNPoint decouples the teaching process into discrete coach-specified skills, goal-localized interaction windows, and robust RL-based motor policy synthesis in simulation. The approach highlights the spatio-temporal decoupling governing dynamic physical tasks: entire-body motions are hierarchically structured, but outcomes depend acutely on brief, local trajectory segments critical for task success—the so-called "interaction windows." This abstraction, combined with randomized sampling in training, yields zero-shot generalization to novel task instantiations and rapid trainability.

Figure 1: TaskNPoint enables dynamic skill transfer for a humanoid from a single human demonstration, with RL in simulation yielding generalization to unseen goals.

Methodological Framework

Task Abstraction and Structure

Complex, dynamic tasks—returning a tennis ball, kicking a soccer ball, grasping a box—are encoded as hierarchical compositions of:

• Discrete skill selection: The coach enumerates a finite library of atomic skills (e.g., forehand, backhand, kick) and supplies a single motion demonstration per skill.
• Goal and interaction definition: The key event determining task success (racket–ball contact, box grasp, etc.) is manually annotated, with the goal parametrized as $G = (p^*, \nu^*, n^*, t^*)$—the desired spatiotemporal configuration of the relevant end-effector at the moment of interaction.
• Local variation and randomization: After kinematic retargeting from human to robot, training goals are randomized about the nominal demonstration with Gaussian noise, producing a distribution over feasible but diverse targets to foster robust generalization.

This modular abstraction enables discrete-to-continuous generalization over a task space substantially larger than the number of demonstrations.

Figure 2: Overview of the TaskNPoint pipeline, combining single-video demonstration, SMPL-X pose estimation, kinematic retargeting, and a goal-conditioned RL policy.

Human Motion Acquisition and Kinematic Retargeting

Human demonstration videos are processed using state-of-the-art single/multi-view 3D pose estimation (PromptHMR [wang2025prompthmrpromptablehumanmesh]) to reconstruct the full-body trajectory as SMPL-X parameters. In multi-view settings, per-view predictions are fused using a maximum-likelihood estimator with anisotropic covariance for reliable spatial localization, particularly along depth axes. The retargeted trajectory is mapped to the robot using GMR, and critical interaction events are annotated to extract goal tuples for training.

Figure 3: Collection and 3D pose reconstruction from monocular and multi-view demonstration video for high-fidelity motion acquisition.

Goal-Conditioned Policy Learning

The policy is trained using PPO in simulated environments (MJlab [zakka2026mjlab]) with the following key features:

• Observations: Proprioceptive features plus local goal parameters.
• Actions: Joint space position setpoints for low-level PD tracking.
• Rewards: A succinct combination of goal achievement (precise end-effector state in the interaction window) and reference motion imitation, with no auxiliary reward shaping or task-specific tuning required.
• Randomized Curriculum: Each action's relevant parameters (target position, velocity, orientation) are randomly sampled per episode, ensuring that the resulting policy is not merely memorizing specific motions but interpolating across the reachable workspace for each atomic skill.

  Figure 4: The architecture for TaskNPoint, integrating SMPL-X-based demonstration reconstruction, goal sampling, and asymmetric actor-critic policy learning.

  

  Figure 5: Visualization of random sampling in the vicinity of different reference contact points (motion tubules), illustrating dense reward assignment around contact windows.

Motion Selection, State Estimation, and Deployment

Online, perception modules track dynamic objects (using, e.g., OptiTrack and a Kalman filter) and predict future contact points. At each timestep, the planning stack selects which skill to trigger (based on the closest match between predicted object trajectory and library interaction points), updates the corresponding local goal, and commands the robot to execute the relevant policy phase, with action locking to prevent late oscillatory trigger changes.

Empirical Results

Task Space Coverage and Generalization

A salient feature of TaskNPoint is the ability to achieve high workspace coverage with very few demonstrations. Each atomic skill seeds a ball of effective skill coverage via local randomization. As shown in both simulation and hardware, this covers a volumetric space far exceeding that spanned by any individual demonstration, matching or exceeding state-of-the-art methods in 3D operational diversity with an order of magnitude less supervision.

Figure 6: 3D coverage of the task space for a handful of demonstration skills, with each sphere centered at the annotated contact point.

Qualitative and Quantitative Performance

In hardware and simulation, TaskNPoint-trained policies exhibit strong generalization to novel, out-of-distribution targets. Robustness persists under domain randomization and noisy perception. For sports (tennis/soccer), GSR (generalized success rate over diverse initial conditions) exceeds 90% and for box pick-and-place, the rate exceeds 96%. Unlike recent imitation-based methods, which are reinforced only along the demonstration manifold or require extensive reward tuning, TaskNPoint achieves task metrics with a minimal engineering footprint and substantially faster convergence (under an hour per policy on a single GPU).

Figure 7: Qualitative execution snapshots of robot performing tennis shots, soccer kicks, and box pick-and-place—each generalized from a single reference demonstration per skill.

Scaling to Diverse Real-World Motions

Experiments demonstrate scaling TaskNPoint to larger libraries of skills (up to 34 distinct motion primitives): coverage grows, phase/timing error remains stable, and target position error increases only modestly, indicating little performance loss as library complexity rises.

Figure 8: Visualization of sampled coverage volumes as additional "in-the-wild" human demonstrations are added, resulting in increased workspace overlap and skill variety.

Ablative Analyses

TaskNPoint maintains performance under hyperparameter perturbations. Collapsing the reward window to a single interaction frame tightens targeting (error <2cm) but performance degrades gracefully as the window broadens. Variance in location and orientation sampling reveals similarly robust, monotonic degradation only at extreme values, and training remains stable until skill coverage becomes trivial. Critically, training and deployment rates can differ moderately without catastrophic failure.

Figure 9: Phase histogram evidencing precision of hit time under very sparse interaction rewards.

Implications and Future Directions

TaskNPoint advocates explicit spatio-temporal goal abstraction, harnessing human priors for skill decomposition and interaction analysis in contrast to end-to-end data-driven or black-box reward optimization. This modularity has profound implications:

• Rapid adaptation and transfer: Novel skills can be incorporated in minutes, matching human-like learning efficiency.
• Minimal reward engineering: The framework exports nearly all task structure and validation to the coach, removing reliance on fragile reward design or data curation.
• Hardware readiness: Policies are robust to simulation-to-reality gap, failure modes are largely restricted to perception/estimation.
• Scalability: Extending the action library to more skills shows only mild degradation, implying that the "atomic skill + goal local randomization" abstraction scales.

Practical deployments will benefit from integrating tactile/force feedback (to address tasks requiring richer physical contact reasoning), adaptive estimator modeling in the training loop, and higher-level skill sequencing. Theoretically, this decoupling paradigm strengthens arguments for systemic modularization in robot learning architectures, potentially facilitating more interpretable, debuggable, and generalizable learning systems. Future advances may leverage learned goal-selection hierarchies and meta-learning to enable truly open-world skill acquisition.

Conclusion

TaskNPoint operationalizes the human sports coaching paradigm in robot learning: discrete annotation of skills, precise localization of interaction events, and goal-conditioned RL to extend sparse demonstrations across rich, dynamic environments. The outcome is an efficient, robust method for humanoid acquisition of sports and general loco-manipulation skills suitable for real-world deployment with minimal computational or engineering overhead. The framework’s explicit abstraction and empirical validation mark a clear pathway for further research in modular, goal-conditioned robot learning.