CoorDex: Coordinated Body & Hand Control

• CoorDex is a pipeline that decomposes humanoid control into separate proprioception-conditioned body and hand latent priors for seamless loco-manipulation.
• It employs simulated demonstrations to train privileged tracking teachers that are distilled via a VAE-style bottleneck, enabling structured latent residual actions.
• The coordinated latent residual control strategy facilitates non-stop tasks like bottle grasping, fridge opening, and cube pick-and-turn with enhanced stability and natural motion.

Searching arXiv for the CoorDex paper and closely related work. arXiv search query: "CoorDex Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation" CoorDex is a learning pipeline for continuous dexterous humanoid loco-manipulation that treats locomotion and multi-finger manipulation as coupled behaviors rather than as a stop-and-go sequence of walking, stopping, manipulating, and resuming motion. Its central mechanism is to convert high-dimensional body-and-hand control into structured latent residual control: separate body and hand motion priors are learned from simulated demonstrations, distilled into proprioception-conditioned latent spaces, frozen, and then composed by a coordinated residual reinforcement-learning policy. In the reported system, this formulation enables a Unitree G1 humanoid with a 20-DoF WUJI hand to execute dexterous manipulation while in motion, including non-stop bottle grasping and carrying, fridge door opening on the move, and cube pick-and-turn (Li et al., 22 Jun 2026).

1. Problem formulation and scope

CoorDex targets a specific failure mode in humanoid loco-manipulation: when dexterous hands are added to a mobile humanoid, the control problem must simultaneously manage balance, whole-body motion, wrist placement, reaching, finger preshape, contact formation, object transport, and contact perturbations that feed back into the body. The paper states that direct learning in full joint space makes exploration extremely hard, and that many existing pipelines therefore simplify the task into a stop-and-go pattern consisting of walking to an object, stopping, grasping or manipulating, and then walking again (Li et al., 22 Jun 2026).

The framework is designed for on-the-move dexterous manipulation, where wrist pose depends on whole-body dynamics and finger contact must be refined while the robot is still moving. To address that coupling, CoorDex separates the overall problem into three components: a body prior for locomotion, reaching, and wrist placement; a hand prior for finger-level dexterity; and a coordinated residual policy that composes the two during downstream reinforcement learning (Li et al., 22 Jun 2026).

A plausible implication is that CoorDex is not merely a manipulation method attached to an existing gait controller, but a hierarchy in which locomotor and contact behaviors are jointly regulated through a shared task-conditioned interface. The paper’s emphasis is that whole-body mobility and dexterous contact should be treated as coupled behaviors, not stitched-together phases (Li et al., 22 Jun 2026).

2. Pipeline architecture and demonstration generation

CoorDex is organized as a four-stage pipeline: collect simulated demonstrations, train privileged motion-tracking teachers, distill teachers into proprioception-conditioned latent priors, and freeze those priors for residual PPO (Li et al., 22 Jun 2026). The pipeline begins in Isaac Lab with separate demonstration streams for the body side and the hand side.

For body-side demonstrations, lower-body locomotion is generated using an AGILE-based locomotion controller. The operator provides right wrist and hand motion through Apple Vision Pro using a CloudXR-based XR teleoperation interface, and the tracked human wrist pose becomes the target for a Pink inverse-kinematics solver. This produces whole-body trajectories containing walking, torso motion, arm motion, and wrist placement (Li et al., 22 Jun 2026).

For hand-side demonstrations, hand motions are retargeted using optimization-based dex-retargeting from ManipTrans-style data. The paper uses floating-hand environments in which wrist pose and velocity are written directly into simulation, so that the learning problem can focus on finger coordination rather than wrist trajectory generation (Li et al., 22 Jun 2026). This separation is structurally important: the body prior handles wrist placement, whereas the hand prior handles finger dexterity (Li et al., 22 Jun 2026).

The paper explicitly does not train a single unified policy directly from raw demonstrations. Instead, demonstrations are used to train two separate motion priors with different roles (Li et al., 22 Jun 2026). This suggests that CoorDex treats demonstration data primarily as a source of subsystem-specific motion regularities rather than as end-to-end imitation targets.

3. Privileged teachers and latent-prior distillation

The first learned stage consists of privileged motion-tracking teachers. The body teacher is a whole-body motion tracker, denoted by policy $\pi_T^b$, which takes body proprioception $\mathbf{s}^{b,p}_t$ and reference goal $\mathbf{s}^{b,g}_t$ as input and outputs body joint position targets $\mathbf{a}^{b,T}_t$. It tracks root or pelvis motion, torso and arm links, wrist links, and posture or reach-related full-body behavior (Li et al., 22 Jun 2026).

The hand teacher, denoted by $\pi_T^h$, is trained in a floating-hand setup. It takes hand proprioception $\mathbf{s}^{h,p}_t$ and reference goal $\mathbf{s}^{h,g}_t$ as input and outputs active finger joint targets $\mathbf{a}^{h,T}_t$. Because the reference wrist pose and velocity are written directly into simulation, this teacher controls only the 20 active finger joints of the WUJI hand (Li et al., 22 Jun 2026).

After teacher training, each subsystem is distilled into a latent prior using a VAE-style proprioception-conditioned bottleneck inspired by PULSE. For each subsystem $x \in \{b,h\}$, the student consists of an encoder $\mathcal{E}_x$, a prior $\mathbf{s}^{b,p}_t$0, and a decoder $\mathbf{s}^{b,p}_t$1 (Li et al., 22 Jun 2026). The manuscript gives the distillation objective in a partially corrupted form and also provides a cleaner appendix expression:

$\mathbf{s}^{b,p}_t$2

In the notation described by the paper, $\mathbf{s}^{b,p}_t$3 is the decoded action prediction, $\mathbf{s}^{b,p}_t$4 is the teacher action, $\mathbf{s}^{b,p}_t$5 is the encoder posterior mean, and $\mathbf{s}^{b,p}_t$6 is the previous encoder mean used for temporal smoothness (Li et al., 22 Jun 2026). The result is a latent space in which the default command can be inferred from proprioception alone, the latent is compact and reusable, and downstream control acts by modifying a low-dimensional latent instead of raw joint targets (Li et al., 22 Jun 2026).

After distillation, the prior $\mathbf{s}^{b,p}_t$7 and decoder $\mathbf{s}^{b,p}_t$8 are frozen, and the prior mean becomes the default latent command (Li et al., 22 Jun 2026). This frozen latent-prior interface is the basis for the downstream control formulation.

4. Coordinated latent residual control

Once the body and hand priors have been learned, CoorDex uses them as the action space for downstream reinforcement learning. At each control step, the frozen priors output latent means

$\mathbf{s}^{b,p}_t$9

The policy predicts latent residuals rather than joint targets,

$\mathbf{s}^{b,g}_t$0

which are added to the prior means to form corrected latents. The frozen decoders then produce body and hand joint targets, and these are executed through low-level PD control (Li et al., 22 Jun 2026).

The paper identifies this as the latent-prior interface: exploration no longer occurs in full joint space, but in a structured 28D residual latent space comprising 16 dimensions for the body and 12 for the hand (Li et al., 22 Jun 2026). The policy architecture is not a single residual predictor over the concatenated latent. Instead, it has a shared coordination trunk, a body residual head, and a hand residual head (Li et al., 22 Jun 2026). The shared trunk is defined as

$\mathbf{s}^{b,g}_t$1

followed by specialized residual heads for body and hand refinement (Li et al., 22 Jun 2026).

The coordination trunk receives body proprioception, hand proprioception, task-level state, hand-object state, the prior means, and the previous latent residual. Task state can include object pose, goal information, projected gravity, and contact features; hand-object state includes object pose in the hand frame and fingertip-object contact features (Li et al., 22 Jun 2026). The body head $\mathbf{s}^{b,g}_t$2 refines stepping, torso posture, reaching, and wrist placement, while the hand head $\mathbf{s}^{b,g}_t$3 refines finger preshape, closure, and contact refinement (Li et al., 22 Jun 2026).

The paper argues that these specialized heads matter because the two subsystems have different control roles: body dynamics concern global balance and placement, whereas hand dynamics concern local contact and dexterity (Li et al., 22 Jun 2026). A plausible interpretation is that CoorDex imposes modularity at the policy-output level while maintaining cross-subsystem coupling through a shared state-conditioned trunk.

5. Experimental configuration and benchmark tasks

The main experiments use a Unitree G1 humanoid with 29 actuated body DoF and a WUJI five-finger hand with 20 DoF (Li et al., 22 Jun 2026). The latent dimensions are 16 for the body prior and 12 for the hand prior, giving a total 28D residual action space (Li et al., 22 Jun 2026).

Training is conducted in Isaac Lab at a control rate of 60 Hz with 4096 parallel environments. PPO uses 24 rollout steps per environment, 5 epochs, and 4 minibatches. The reported hyperparameters are discount $\mathbf{s}^{b,g}_t$4, GAE $\mathbf{s}^{b,g}_t$5, clip parameter 0.2, entropy coefficient 0.005, adaptive learning rate $\mathbf{s}^{b,g}_t$6, KL target 0.01, hidden layers $\mathbf{s}^{b,g}_t$7, and ELU activation (Li et al., 22 Jun 2026).

Three benchmark tasks are evaluated.

Task	Description	Skill type
WalkGrab	Walk forward continuously, grasp and lift a bottle from a side table, and carry it while moving	Dynamic grasping during locomotion
OpenFridge	Grasp a fridge handle, open the door while stepping backward, and maintain contact while moving	Contact-rich articulated manipulation
WalkPickTurn	Approach a table, pick up a cube, and rotate $\mathbf{s}^{b,g}_t$8 while holding it	Long-horizon skill composition

These tasks were chosen to span dynamic grasping during locomotion, contact-rich articulated manipulation, and long-horizon skill composition (Li et al., 22 Jun 2026).

The reported task outcomes are: WalkGrab success 0.55, fall 0.00, drop 0.40; OpenFridge success 0.66, fall 0.00; and WalkPickTurn success 0.89, fall 0.01, drop 0.10 (Li et al., 22 Jun 2026). For WalkGrab, the paper emphasizes that the robot maintains non-stop locomotion, with a velocity analysis showing that near the bottle it still moves forward at about 0.25 m/s rather than solving the task by stopping and switching to stationary grasping (Li et al., 22 Jun 2026).

6. Ablations, failure modes, and comparative interpretation

The ablations focus on the walk-grasp-carry setting and are central to the paper’s argument that both the latent-prior action interface and the coordinated residual structure are necessary (Li et al., 22 Jun 2026).

In the action-space ablation, three methods are compared: All Joint Space, Body Prior + Hand Joint Space, and CoorDex. The reported results are as follows.

Method	Success	Reach	Grasp	Stop	Fall
All Joint Space	0	1.00	0.00	0.86	0.04
Body Prior + Hand Joint Space	0	0.96	0.01	0.90	0.04
CoorDex	0.55	1.00	0.55	0.00	0.00

The paper’s interpretation is that raw joint-space PPO can reach the bottle but fails to learn grasping and often produces unnatural whole-body postures. Adding a body prior improves walking and wrist placement, but finger coordination remains too difficult in raw hand joint space, and the policy tends to stop and turn the task into a stationary grasp problem. By contrast, CoorDex learns grasping while walking, avoids stopping, and succeeds with zero falls (Li et al., 22 Jun 2026).

A second ablation compares Monolithic Latent Residual with CoorDex while keeping the same frozen priors, same latent dimensions, and same task setup. Only the policy architecture differs.

Method	Success	Action rate	Fall
Monolithic Latent Residual	0.00	0.40	0.02
CoorDex	0.55	0.22	0.00

The paper attributes the monolithic model’s failure to more jittery motion, less natural body behavior, and unreliable grasping under the same reward budget (Li et al., 22 Jun 2026). The coordinated design improves smoothness, stability, and cross-body/hand task reasoning (Li et al., 22 Jun 2026). This suggests that the architecture is not simply using low-dimensional priors as an optimization aid; it also relies on an explicit coordination structure to prevent body and hand residuals from becoming entangled.

The overall formulation is summarized in the paper as

$\mathbf{s}^{b,g}_t$9

That sequence is presented as the main novelty of CoorDex (Li et al., 22 Jun 2026).

7. Relation to grasp generation, limitations, and outlook

The paper positions CoorDex as a simulation-centric framework for continuous dexterous loco-manipulation. Its demonstrated capabilities include non-stop grasping while walking, door opening while stepping back, and pick-and-turn behavior while holding an object, all with natural whole-body motion and finger-level contact refinement (Li et al., 22 Jun 2026). The framework is especially aimed at preserving locomotion while manipulating, avoiding stop-and-go decomposition, and making high-DoF contact-rich control trainable (Li et al., 22 Jun 2026).

Several limitations are stated explicitly. First, the policies use privileged state observations such as object poses and contact signals; no vision or perception pipeline is included (Li et al., 22 Jun 2026). Second, the main quantitative evaluation is in Isaac Lab, while hardware results are qualitative replay or visualization (Li et al., 22 Jun 2026). Third, experiments focus on a fixed G1 + WUJI setup, so broader deployment across hands and hardware is not yet demonstrated (Li et al., 22 Jun 2026). Fourth, long-horizon exploration still requires curriculum support, as WalkPickTurn uses NoDemoRSI (Li et al., 22 Jun 2026). Fifth, generalization to more objects, hands, and real-world transfer remains future work (Li et al., 22 Jun 2026). The paper also notes that the hardware visualization uses a G1 + Dex3-1 hand rather than the G1 + WUJI configuration used in the main simulation results, so those visualizations are qualitative and not on the exact evaluation platform (Li et al., 22 Jun 2026).

A related line of work mentioned in the provided material is CEDex, a cross-embodiment dexterous grasp synthesis pipeline that directly serves the goal of CoorDex by generating grasps that transfer across robotic hands with different morphologies while preserving human-like kinematic plausibility and physical feasibility (Wu et al., 29 Sep 2025). CEDex operates at the object-contact representation level, generating human-like contact structure with a CVAE, aligning it to robot morphology through topological merging, and then optimizing grasp pose with SDF-based physical constraints (Wu et al., 29 Sep 2025). A plausible implication is that such contact-centric grasp priors could complement CoorDex’s body-hand coordination framework by broadening the range of hand embodiments and grasp repertoires available to the downstream loco-manipulation system.

Taken together, CoorDex defines a specific architectural thesis: dexterous humanoid loco-manipulation becomes trainable when body motion and finger dexterity are first regularized into separate proprioception-conditioned latent priors and then recomposed by a shared-context residual controller. Within the reported experiments, direct joint-space RL and monolithic latent prediction fail under the same reward budget, whereas the latent-prior interface and coordinated residual structure permit continuous on-the-move manipulation on a high-DoF humanoid platform (Li et al., 22 Jun 2026).