---
title: 'PingpongDuo: Hierarchical Framework for Humanoid Table-Tennis Doubles'
url: https://www.emergentmind.com/topics/pingpongduo
type: topic
---

# PingpongDuo: Hierarchical Framework for Humanoid Table-Tennis Doubles

PingpongDuo is a hierarchical framework for agile humanoid table-tennis doubles with safe team coordination. It enables two humanoid robots to strike in strict alternation while sharing a confined workspace, combining a risk-aware team-positioning planner, individually learned footwork and striking skills, transition-aware skill composition, and a racket-arm residual controller. The system is implemented on two 29-degree-of-freedom Unitree G1 humanoids and uses external motion capture for robot and ball state estimation [2610.20579].

## 1. Problem formulation and system architecture

Humanoid table-tennis doubles imposes constraints absent from conventional singles play. The robot assigned to the $k$-th incoming ball must differ from the robot assigned to the preceding ball:

$$
h_{k+1}\neq h_k.
$$

Consequently, a valid return requires not only successful racket–ball contact, but also compliance with the prescribed alternating-striker order. Both humanoids share the central workspace: the current striker requires sufficient room for preparation and follow-through, while the previous striker must clear the region without compromising its own future recovery.

PingpongDuo separates the system into hierarchical layers:

1. **Ball prediction and strike planning** estimate the contact state and racket command.
2. **Team-level positioning** jointly selects lateral targets for both humanoids.
3. **Individual skills** execute footwork, clearance, recovery, or striking.
4. **Racket-arm residual control** corrects the nominal strike using simulated return feedback.
5. **A state machine** enforces striking alternation and transitions among striking, clearance, waiting, preparation, and recovery.

For each robot $i$, the system maintains a robot state $x_t^i$, ball state $b_t$, state-machine phase $\phi_t^i$, and the identity $h_k$ of the robot assigned to the next incoming ball. The planner is event-driven at the incoming-ball level, whereas the learned controllers operate at the low-level control frequency.

The system is evaluated in Isaac Lab with physics simulation at 200 Hz and control at 50 Hz. Hardware experiments use two Unitree G1 humanoids. Real-world state estimation relies on external motion capture rather than fully onboard perception.

## 2. Ball prediction and nominal striking

PingpongDuo uses an adaptive extended Kalman filter to estimate ball position and velocity. A physics model predicts flight under gravity, aerodynamic drag, and table bounces. The system then selects a contact position and contact time and computes a racket command for the designated striker.

For robot $i$, the strike command is

$$
c_t^{i,\mathrm{hit}} =
\left(
p_r^{i,\star},
v_r^{i,\star},
\tau_t^i
\right),
$$

where $p_r^{i,\star}$ is the desired racket position at contact, $v_r^{i,\star}$ is the desired racket velocity, and $\tau_t^i$ is the remaining time to contact.

The nominal striking policy is

$$
a_t^{i,\mathrm{hit}}
=
\pi_{\mathrm{strike}}
\left(
o_t^{i,\mathrm{prop}},
m_t^{i,\mathrm{hit}},
c_t^{i,\mathrm{hit}}
\right),
$$

where $o_t^{i,\mathrm{prop}}$ contains proprioception and recent control history, $m_t^{i,\mathrm{hit}}$ is a selected whole-body reference-motion feature, and $a_t^{i,\mathrm{hit}}$ is a whole-body joint-position target.

Training combines motion imitation, task rewards, stability terms, and control regularization. The task rewards encourage the racket to reach the commanded position and velocity and to align the racket-face normal with the desired strike direction near contact. The strike policy alone does not solve the doubles problem because it does not determine how the non-striker should move.

## 3. Joint team-positioning planner

At planning step $n$, the team policy proposes a pair of lateral base targets:

$$
u_n =
\begin{pmatrix}
y_n^A\\
y_n^B
\end{pmatrix}
\sim
\pi_\theta
\left(
\cdot\mid o_n^{\mathrm{plan}}
\right).
$$

The planning observation contains the predicted strike command, the current base positions of both robots, and the previously accepted target pair. The robots remain within a suitable longitudinal range for striking, while the planner primarily selects lateral positions because lateral displacement governs clearance and subsequent recovery.

The two target components have distinct functions. The striker target supports the immediate return, whereas the non-striker target clears the hitting region while preserving its ability to prepare for the next turn.

The planner is trained over sequences of incoming balls using PPO-style constrained policy optimization. Its abstract objective is

$$
\max_{\theta}\;
\mathbb{E}_{\pi_\theta}
\left[
\sum_n R_n^{\mathrm{team}}
\right]
\quad
\text{subject to}
\quad
\mathbb{E}_{\pi_\theta}
\left[
\sum_n C_n^{\mathrm{sep}}
\right]
\leq d_{\max},
$$

where $R_n^{\mathrm{team}}$ represents team-level task performance and $C_n^{\mathrm{sep}}$ measures inter-robot separation violations. The low-level controller and risk estimator remain fixed during planner training, isolating target selection from execution.

This joint formulation differs from independently positioning each robot. It permits the planner to account simultaneously for immediate striking quality, clearance, future recovery, and role alternation.

## 4. Learned risk filtering and role-aware safety

PingpongDuo estimates collision risk from closed-loop behavior rather than from target geometry alone. It monitors three classes of inter-robot separation:

$$
\mathcal Q =
\{
\mathrm{hand},
\mathrm{foot},
\mathrm{base}
\}.
$$

For candidate target pair $u_n$, the fixed low-level controller is rolled out over an interval $\mathcal I_n$. For body-part class $q$, the risk label is

$$
z_{n,q}
=
\mathbb{I}
\left[
\min_{t\in\mathcal I_n}
\delta_q(t)
<
\delta_q^{\min}
\right],
$$

where $\delta_q(t)$ is the corresponding distance and $\delta_q^{\min}$ is the minimum allowed separation.

An MLP $r_\psi$ predicts per-part risks:

$$
r_\psi
\left(
o_n^{\mathrm{risk}},u_n
\right)
=
\left(
r_{\psi,\mathrm{hand}},
r_{\psi,\mathrm{foot}},
r_{\psi,\mathrm{base}}
\right).
$$

The scalar risk used for filtering is their mean:

$$
\bar r_\psi(o,u)
=
\frac{1}{|\mathcal Q|}
\sum_{q\in\mathcal Q}
r_{\psi,q}(o,u).
$$

If

$$
\bar r_\psi
\left(
o_n^{\mathrm{risk}},u_n
\right)
\leq \eta,
$$

the proposed pair is accepted. Otherwise, the system searches for the closest admissible pair satisfying the risk constraint:

$$
\begin{aligned}
\tilde u_n
=
\arg\min_{u\in\mathcal U}\quad&
w_{\mathrm{str}}
\left(
u_{\mathrm{str}}-u_{n,\mathrm{str}}
\right)^2
+
w_{\mathrm{nstr}}
\left(
u_{\mathrm{nstr}}-u_{n,\mathrm{nstr}}
\right)^2\\
\text{subject to}\quad&
\bar r_\psi
\left(
o_n^{\mathrm{risk}},u
\right)
\leq\eta .
\end{aligned}
$$

The defining role-aware condition is

$$
w_{\mathrm{str}}>w_{\mathrm{nstr}}>0.
$$

The filter therefore preferentially preserves the striker’s target and modifies the non-striker’s target to resolve the conflict. This reflects the greater immediate sensitivity of the striker’s position: changing it can degrade racket reach, timing, or return quality, whereas the non-striker generally has more freedom to move laterally.

Target search is implemented by grid search over the admissible target set $\mathcal U$. If no safe pair exists, both robots return to predefined default targets and the next strike handoff is withheld. This fallback prioritizes safe coordination over continuation of an invalid alternating exchange.

Compared with a rule-based planner and a heuristic control-barrier-function filter, the learned-risk approach avoids contacts in cases where those baselines fail. The learned planner is also important: filtering rule-based targets improves safety, but learned target selection improves future preparation and produces longer return sequences.

## 5. Skill learning, transition composition, and residual striking

### Footwork and human motion references

The authors collected approximately 200 human table-tennis doubles footwork clips covering strike preparation, non-striker clearance, recovery, and different lateral movement distances. The clips were retargeted to the humanoid using General Motion Retargeting and annotated with lateral displacement.

For a requested displacement $\Delta y_t^i$, the reference motion is selected by

$$
m^\star
=
\arg\min_{m\in\mathcal M(\Delta y_t^i)}
\left|
\Delta y_t^i-\Delta y_m
\right|,
$$

where

$$
\Delta y_t^i
=
y_{t,\mathrm{cmd}}^i-y_t^i.
$$

The footwork policy is trained with position and velocity tracking, motion imitation, stability objectives, and control regularization. During movement, target reaching is emphasized; during waiting, braking and maintenance of a ready posture are emphasized.

### State-machine execution

For robot $i$, the nominal action is

$$
a_t^{i,\mathrm{nom}}
=
\begin{cases}
\pi_{\mathrm{strike}}
\left(
o_t^{i,\mathrm{prop}},
m_t^{i,\mathrm{hit}},
c_t^{i,\mathrm{hit}}
\right),
&
\phi_t^i\in\Phi_{\mathrm{hit}},
\\[4pt]
\pi_{\mathrm{move}}
\left(
o_t^{i,\mathrm{prop}},
m_t^{i,\mathrm{move}},
c_t^{i,\mathrm{move}}
\right),
&
\text{otherwise}.
\end{cases}
$$

The striking phase includes the strike and a short follow-through. After striking, the robot becomes the non-striker and clears the central region, while the teammate moves toward the next striking position. Clearance and preparation overlap rather than requiring sequential completion.

The same skills support opposite sides through mirroring of observations, commands, reference motions, and actions.

### Transition-aware training

Independent policy switching can fail because the state at a transition is not a nominal standing state. A robot may retain linear or angular velocity after striking, may still be moving after clearance, or may need to strike before footwork has settled.

PingpongDuo therefore trains each skill from states generated by the other:

$$
x_0^{\mathrm{move}}
\sim
\mathcal D_{\mathrm{hit}\rightarrow\mathrm{move}},
\qquad
x_0^{\mathrm{hit}}
\sim
\mathcal D_{\mathrm{move}\rightarrow\mathrm{hit}}.
$$

The transition datasets retain body pose, orientation, linear and angular velocity, and joint positions and velocities. Control histories are maintained across switches, and references are blended over a short transition interval.

### Racket-arm residual

Fast footwork can leave the humanoid slightly misaligned or moving at contact. PingpongDuo introduces a bounded residual over the seven racket-arm joints:

$$
\begin{aligned}
\Delta a_{t,\mathrm{arm}}^i
&=
\pi_{\mathrm{res}}
\left(
o_t^{i,\mathrm{hit}},
a_t^{i,\mathrm{nom}},
\hat b_t^i
\right),\\
a_t^{i,\mathrm{exec}}
&=
a_t^{i,\mathrm{nom}}
+
\alpha g_t^i
S_{\mathrm{arm}}^i
\Delta a_{t,\mathrm{arm}}^i .
\end{aligned}
$$

Here $\hat b_t^i$ is the estimated ball position and velocity, $S_{\mathrm{arm}}^i$ embeds the arm correction into the full action vector, $\alpha$ is the residual scale, and $g_t^i$ is a phase-dependent gate that activates the residual near contact and removes it afterward.

The residual is trained using return feedback rather than only racket tracking error. The simulated outcome considers whether the ball is contacted, crosses the net, lands on the opponent’s table, lands accurately, and maintains adequate net clearance. Misses, low net crossings, out-of-bounds landings, unintended ball–body contacts, and excessive residual magnitude are penalized.

This objective reflects the fact that the smallest racket-position error does not necessarily yield the best return: contact orientation, racket velocity, body motion, and timing jointly determine outgoing-ball behavior.

## 6. Calibration, evaluation, and limitations

### Real-to-sim-to-real calibration

The simulator is calibrated using real measurements of vertical drops, oblique ball–table impacts, flight trajectories, rebound heights, contact times, and post-impact velocities. The identified quantities include aerodynamic drag, table restitution, friction, speed-dependent normal restitution, and effective tangential attenuation for moving-racket contact.

The fitting procedure first estimates flight and table dynamics and then fits racket-contact parameters. This staged procedure reduces the possibility that one incorrect model component compensates for another. The calibrated simulator is subsequently randomized over ball mass, drag, table restitution, racket-contact response, observation perturbations, and actuator delays.

### Evaluation protocol

The experiments use Isaac Lab with physics at 200 Hz and control at 50 Hz. Evaluations use five seeds. Planner evaluations contain 100 attempts per seed; policy evaluations use 100 balls and one clearance trial per seed; movement-and-striking refinement conditions use 100 trials.

Reported metrics include mean returns, $S_k$ for the percentage of attempts with at least $k$ alternating legal returns, racket-contact success $SR_C$, legal-return success $SR_H$, racket position and velocity errors, racket-normal error, lateral clearance-target error $E_M$, and motion-reference MPJPE.

A legal return must cross the net, land on the opponent’s table, and obey the alternating-striker order.

### Quantitative results

The team-planner comparison is:

| Method | Mean returns | $S_2$ | $S_3$ | $S_4$ | $E_p$ (cm) |
|---|---:|---:|---:|---:|---:|
| Rule-based | $3.60\pm0.40$ | 57.4% | 48.0% | 38.4% | 7.92 |
| Rule + heuristic CBF | $3.79\pm0.38$ | 57.4% | 48.0% | 39.2% | 7.96 |
| Rule + learned risk | $3.79\pm0.36$ | 57.4% | 48.0% | 38.8% | 7.94 |
| RL + learned risk | **$4.87\pm0.74$** | **64.0%** | **54.8%** | **46.6%** | **7.42** |

For composed footwork and striking, the system achieves $SR_C=67.4\%$, legal-return success $SR_H=55.6\%$, clearance error $E_M=2.11$ cm, and motion-reference MPJPE of $89.08$ mm. Without arm refinement, $SR_C$ falls to 55.2% and $SR_H$ to 44.2%, while clearance remains $2.11$ cm. Thus the residual increases contact success by 12.2 percentage points and legal-return success by 11.4 percentage points without altering the clearance behavior.

The abstract reports a 13.2% improvement in ball-return success over the state-of-the-art baseline. In real-world experiments, the robots complete a 2.15 m clearance-and-recovery cycle in 2.4 s with a peak speed of 2.23 m/s, approximately 2 cm positioning error during clearance, and more than 10 consecutive alternating strikes.

### Limitations

PingpongDuo has several stated limitations:

- **External motion capture**: the real system does not yet provide fully onboard perception of the robots and ball.
- **No spin modeling**: spin measurements are unavailable, and spin effects are omitted from trajectory prediction and racket–ball interaction.
- **Simulation-derived risk**: the risk estimator depends on the fidelity of simulated closed-loop rollouts and their transfer to hardware.
- **Local discrete filtering**: grid search over candidate target pairs does not provide a formal global safety guarantee.
- **Limited high-level strategy**: the system returns incoming balls under a prescribed alternation, rather than autonomously selecting attacking or defensive tactics.
- **Safe-play interruption**: when no safe target pair exists, the system falls back to default targets and withholds the next handoff.

PingpongDuo’s principal contribution is the integration of joint role-aware planning, learned closed-loop safety filtering, human-guided doubles footwork, transition-aware skill composition, and return-feedback arm refinement. These components address the coupled requirements of strict alternation, rapid clearance, collision avoidance, and accurate striking in a shared humanoid workspace.

Source: https://www.emergentmind.com/topics/pingpongduo