TaskNPoint: Humanoid Skills via Interaction Windows

• TaskNPoint is a training protocol that leverages a critical interaction window to determine dynamic task outcomes in humanoid control.
• The method uses discrete skill selection combined with a single human demonstration per skill to build an action library efficiently.
• It employs goal-conditioned reinforcement learning in simulation to adapt skills for tasks like tennis, soccer, and pick-and-place without per-task reward tuning.

Searching arXiv for TaskNPoint and closely related humanoid imitation / goal-conditioned RL papers. arXiv search query: "TaskNPoint humanoid interaction window imitation RL" TaskNPoint is a training protocol for teaching dynamic humanoid skills from minimal human supervision by exploiting a specific structural property of dynamic tasks: the outcome is largely decided by a short “interaction window” within the trajectory, even though realizing that window requires coordinated whole-body motion. In the formulation introduced for a Unitree G1 humanoid, a human coach supplies a discrete skill inventory, one demonstration per skill, an annotation of the interaction window, and a task goal specified as a target interaction in space and time; learning in simulation then fills in the remainder of the trajectory and provides robustness to unmodeled events. The method is evaluated on tennis forehands and backhands, soccer kicks, and box pick-and-place, with the paper reporting successful learning from short human video demonstrations and under an hour of training on a single GPU, without per-task reward tuning (Werner et al., 24 Jun 2026).

1. Conceptual basis and task abstraction

TaskNPoint is built on the claim that many dynamic loco-manipulation tasks decompose into discrete skill selection and continuous action shaping. In this view, the robot does not need to infer an open-ended behavior space from large motion corpora; instead, it selects from a finite skill vocabulary such as forehand, backhand, kick, or pickup, and then adapts the chosen skill to a specific target location, target timing, and environmental state.

The defining abstraction is the interaction window. For a tennis backhand, the paper identifies the relevant quantity not as the entire swing but as roughly the $20$ cm of racket travel around ball contact. The same logic is extended to soccer ball contact and box grasping. The outcome is treated as being determined by a brief interval near contact or release, while the rest of the trajectory exists to make that interval physically feasible given morphology, balance, and dynamics.

This leads to a reduction of the full causal chain

$$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$

to a formulation in which the goal is identified with the interaction itself, written as $G := J$. The key learned map is therefore $G \rightarrow A$: from a specified interaction target to a feasible whole-body action. A plausible implication is that the method is less concerned with discovering skills than with reliably realizing a known skill under varying geometric conditions.

For tennis, the target interaction is parameterized as

$G^* := (p^*, \nu^*, n^*, t^*),$

where $p^*$ is the $3$D impact point, $\nu^*$ is racket velocity direction at impact, $n^*$ is the end-effector orientation axis, and $t^*$ is contact time. The reward-active interval around interaction is denoted

$$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$0

or, in phase form,

$$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$1

2. Human supervision and the coach–learner division of labor

The protocol makes the division between human specification and machine learning explicit. The human coach provides four primary inputs.

First, the coach defines a discrete set of skills, which become an action library

$$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$2

Second, the coach provides one demonstration per skill. The nominal regime is a few seconds of monocular human video, although multi-view video is also supported. The method assumes humanoid morphology so that human motion is a useful prior.

Third, the coach identifies the interaction timestamp $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$3, or more generally an interaction phase interval $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$4. This is the supervision that specifies what portion of the motion determines success.

Fourth, the coach specifies the task goal. In the paper’s formulation, the goal is not the final object outcome, such as the ball landing point, but the desired interaction quality itself. For tennis, this is the tuple $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$5; for soccer and pick-and-place, the same principle is applied to foot-ball contact and hand-box grasp.

Additional assumptions and priors are also fixed by the method: a humanoid robot model, kinematic retargeting from human to robot, the identity of the interaction link, a physically realistic simulation environment, and an external object-state estimator at deployment. The paper uses OptiTrack for object tracking in hardware. This suggests that TaskNPoint is not purely end-to-end; it is a structured system in which representation, motion prior, and deployment-time estimation are externally specified.

3. From demonstration to action library

The pipeline begins with human video reconstruction. Framewise SMPL-X parameters are recovered as

$$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$6

where $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$7 is translation, $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$8 is global orientation, $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$9 is body shape, and $G := J$0 is body pose.

For in-the-wild multi-view tennis, the paper fuses multiple monocular reconstructions into a maximum-likelihood pose estimate. If each camera provides a joint estimate $G := J$1 with covariance $G := J$2, then after lifting to world coordinates,

$G := J$3

$G := J$4

and the fused joint estimate is

$G := J$5

The reconstructed motion is then retargeted to the humanoid using GMR retargeting. The interaction annotation $G := J$6 is crucial at this stage, because it determines the nominal interaction state to be extracted from the retargeted trajectory: end-effector position $G := J$7, velocity direction $G := J$8, and orientation $G := J$9.

The reference action itself is written as

$G \rightarrow A$0

with $G \rightarrow A$1 denoting robot degrees of freedom over time. The resulting retargeted motions populate the discrete library $G \rightarrow A$2.

To generalize from a single demonstration, TaskNPoint samples training goals around the nominal interaction: $G \rightarrow A$3 with $G \rightarrow A$4 and $G \rightarrow A$5 sampled analogously. For motion primitive $G \rightarrow A$6, the method defines a confidence region

$G \rightarrow A$7

with $G \rightarrow A$8 chosen so that

$G \rightarrow A$9

The paper sets $G^* := (p^*, \nu^*, n^*, t^*),$0 and selects the motion set so that

$G^* := (p^*, \nu^*, n^*, t^*),$1

where $G^* := (p^*, \nu^*, n^*, t^*),$2 is the target workspace. This is the formal statement behind the claim that a small set of discrete skills can cover a larger continuous interaction space.

4. Goal-conditioned policy learning in simulation

TaskNPoint trains a goal-conditioned policy in MJLab with PPO and an asymmetric actor–critic setup (Werner et al., 24 Jun 2026). The actor is conditioned on proprioception, the chosen goal, and the reference trajectory for the selected action. Actor observations include joint positions and velocities, angular velocity, projected gravity or anchor orientation, previous action, and motion command information comprising reference body positions together with target positions, target velocities, and target orientations. The critic receives privileged observations including center-of-mass position and velocity, body positions and orientations, and base linear velocity.

The actor outputs desired joint position setpoints. These are tracked by PD controllers, with simulation at $G^* := (p^*, \nu^*, n^*, t^*),$3 Hz, control decimation of $G^* := (p^*, \nu^*, n^*, t^*),$4, policy frequency $G^* := (p^*, \nu^*, n^*, t^*),$5 Hz, and PD tracking at $G^* := (p^*, \nu^*, n^*, t^*),$6 Hz. The actor and critic each use three hidden layers,

$G^* := (p^*, \nu^*, n^*, t^*),$7

The reward combines three components: motion imitation, target achievement inside the interaction window, and regularization. The target terms are written as exponentials that activate only when $G^* := (p^*, \nu^*, n^*, t^*),$8: $G^* := (p^*, \nu^*, n^*, t^*),$9

$p^*$0

$p^*$1

Imitation rewards track anchor orientation, body positions, body orientations, body linear velocities, and body angular velocities relative to the retargeted reference motion. Regularization includes action-rate penalties,

$p^*$2

joint-limit penalties, and self-collision penalties above threshold.

The paper emphasizes that no per-task reward tuning is required beyond optionally scaling achievement rewards according to the size of $p^*$3. This is a central systems claim: the same reward structure is used across tennis, soccer, and box pick-and-place.

Training further uses reference-state initialization and a chained-motion regime. Instead of resetting immediately when a motion ends, the environment pauses briefly and samples a new motion and goal. This is described as an implicit curriculum: early training emphasizes imitation and balance, while later training naturally incorporates transitions and recovery.

Robustness is attributed to multiple sources: randomized target sampling, many parallel simulation environments, domain randomization, and chaining successful motions into subsequent motions. The simulator randomizes pushes, base center-of-mass offset, encoder bias, and foot friction.

5. Deployment-time planning and evaluated tasks

TaskNPoint is hierarchical at deployment. A planner chooses which motion to execute and when; the learned policy shapes the chosen motion toward the predicted interaction target.

Object state is estimated with a Kalman filter, producing $p^*$4. A hybrid rigid-body model predicts future object motion: $p^*$5 where $p^*$6 is gravity, $p^*$7 is restitution, and $p^*$8 is the ground plane. This yields a predicted trajectory $p^*$9.

Each action $3$0 has nominal contact point $3$1 and lead time $3$2. The action-selection rule is

$3$3

The inner minimization gives the predicted time $3$4 when the object passes closest to the nominal contact point; the outer minimization selects the best action. If $3$5, the action is not executed in time. The action is locked once time-to-contact drops below the motion lead time, while the spatial target may still be updated until contact.

The evaluated tasks span four settings. Tennis includes forehands, backhands, and related variants, with demonstrations from human video and extended experiments using up to $3$6 motions. Soccer evaluates incoming-ball kicks. Box pick-and-place evaluates pickup from novel locations and return. A larger in-the-wild tennis library tests scalability with more discrete motions.

The paper reports simulation metrics of success rate (SR), generalized success rate (GSR), and target position or contact error $3$7. Hardware evaluation uses $3$8 trials per condition.

Setting	Main reported results	Notes
Ballistic hitting	TaskNPoint: SR $3$9, GSR $\nu^*$0, $\nu^*$1 m	Compared with SkillMimic, OmniRetarget, HDMI, HumanX
Box pick-and-place	TaskNPoint: SR $\nu^*$2, GSR $\nu^*$3, $\nu^*$4 cm	Best reported generalized performance in table
Hardware tennis	Slow $\nu^*$5–$\nu^*$6 m/s: SR $\nu^*$7; fast $\nu^*$8–$\nu^*$9 m/s: SR $n^*$0	Text also summarizes $n^*$1–$n^*$2 hit rate for $n^*$3–$n^*$4 m/s tosses
Hardware soccer	Slow $n^*$5–$n^*$6 m/s: SR $n^*$7; fast $n^*$8–$n^*$9 m/s: SR $t^*$0	Evaluated on incoming balls
Hardware box pick-and-place	SR $t^*$1	$t^*$2 trials per condition

The comparison table also states that TaskNPoint uses dynamic environment support, one training stage, about $t^*$3 GPU hour, multi-task learning, one demo per task, and about $t^*$4 minutes of imitation data per task. This suggests that the protocol is designed for data efficiency rather than broad pretraining.

6. Empirical findings, ablations, and limitations

The ablation studies strongly reinforce the interaction-window hypothesis. When target reward is applied over a single frame of contact, the paper reports convergence in $t^*$5 iterations, success $t^*$6, and target error $t^*$7 m. As the reward is spread over more frames, targeting degrades: at $t^*$8 frames, target error rises to $t^*$9 m with no convergence; at $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$00 frames, target error is $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$01 m; at $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$02 frames, target error is $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$03 m and the policy reportedly collapses toward pure imitation. This is the clearest quantitative support for the paper’s central thesis that the narrow interaction window, not a temporally diffuse objective, is what matters (Werner et al., 24 Jun 2026).

Local goal generalization is also characterized explicitly. When the mean target is shifted away from the nominal point, error remains manageable up to about $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$04 m but convergence becomes slower; at $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$05 m offset, training no longer converges and error is $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$06 m. Similarly, moderate increases in target covariance are tolerated, but very large variance causes the policy to revert toward nominal imitation. A plausible implication is that each primitive supports a local neighborhood of goal-conditioned deformations rather than unrestricted workspace-wide generalization.

Scaling experiments increase the motion library from $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$07 to $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$08 motions. The paper reports overall success rate $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$09, average target position error $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$10 cm, and unchanged phase error of $$\text{control} \rightarrow \text{trajectory} \rightarrow \text{interaction} \rightarrow \text{goal}$$11. This supports the claim that the discrete-skill library can be enlarged without collapsing performance, although some degradation in targeting precision is observed.

The method’s limitations are explicit. It does not use force feedback during training, which the paper identifies as problematic for tasks where tactile interaction or recovery behavior matters. It does not include object interaction in the RL loop in the same manner as some interaction-rich systems. Hardware failures are attributed largely to imprecise ball trajectory estimation, so deployment depends on accurate external state estimation. Generalization is local rather than global, because each motion primitive only generalizes around its nominal contact point. The task abstraction is also simplified: in tennis, the goal is impact quality rather than final ball landing position, and inversion from desired landing target to impact conditions is left open. The paper states that the approach is best suited to discrete, contact-centric dynamic skills, and is less naturally suited to long-horizon, force-sensitive dexterous manipulation with rich object dynamics.

Taken together, these results position TaskNPoint as a coach-guided, interaction-window-centered formulation of humanoid skill learning: discrete skill choice is human-specified, continuous adaptation is learned in simulation, and zero-shot target generalization emerges from randomized training around a single demonstrated interaction.