CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation

Abstract: Humanoid loco-manipulation is often simplified into a stop-and-go process: walking to an object, stopping to manipulate it, and then resuming locomotion. It also commonly relies on low degree-of-freedom (DoF) end effectors that behave like an open-close grasp primitive. We introduce CoorDex, a learning pipeline that converts high-dimensional body and dexterous hand control into coordinated latent residual control, enabling high-DoF dexterous loco-manipulation on the move. Starting from simulated whole-body and hand demonstrations, CoorDex trains privileged motion tracking teachers for the humanoid body and dexterous hand, distills them into proprioception-conditioned latent priors, and uses the frozen priors as the action space for downstream residual reinforcement learning. A coordinated latent residual policy composes these priors through shared task context and separate body-hand residual heads, preserving natural whole-body motion while improving finger-level contact reliability. CoorDex enables a Unitree G1 humanoid with a 20-DoF WUJI hand to execute dexterous manipulation while in motion, including non-stop bottle grasping and carrying, fridge door opening on the move, and cube pick-and-turn. Ablations on the walk-grasp-carry task show that joint-space PPO, joint-space hand control, and monolithic latent prediction all fail under the same reward budget, while the latent-prior interface and coordinated residual structure make high-dimensional contact-rich loco-manipulation trainable. Project Page: https://skevinci.github.io/coordex/

• The paper introduces a unified latent residual policy that coordinates humanoid body and hand priors, enabling continuous loco-manipulation without stopping.
• It employs a dual latent space factorization with specialized policy heads, significantly boosting sample efficiency and task generalization.
• Experimental results on tasks like WalkGrab and OpenFridge show high success rates and robust non-stop manipulation in both simulation and real-world tests.

Coordinated Latent Residual Policies for Loco-Manipulation with High-DoF Humanoids

Introduction

CoorDex addresses the challenge of continuous loco-manipulation for humanoids with high-DoF dexterous hands, aiming to overcome the prevalent stop-and-go paradigm in current systems. Most existing pipelines either freeze the manipulator’s trajectory during walking or collapse hand control into low-dimensional primitives, forcing a separation between locomotion and dexterity. CoorDex proposes a unified pipeline that enables simultaneous coordination of full-body locomotion and finger-object contact, allowing tasks such as on-the-move grasping, articulated object interaction during walking, and complex manipulation trajectories requiring combined body/hand adaptation.

Figure 1: A humanoid executing continuous loco-manipulation involving walking, grasping, carrying, and opening a fridge door without stopping, enabled by coordinated body-hand control.

Methodology

Coordinated Latent Residual Architecture

The approach leverages privileged teacher policies for both the full humanoid body and the dexterous hand, training them on reference demonstrations that capture whole-body motion and precise hand interactions. These teachers are distilled into proprioception-conditioned latent priors. CoorDex factorizes control into two latent action spaces—one each for the body and the hand. During downstream RL, a coordinated residual policy utilizes these priors: it predicts residuals in both latent spaces, composed through a shared context trunk and specialized residual heads for locomotor and hand adaptation, producing joint actions via frozen decoders.

Figure 2: Diagrammatic overview of CoorDex: tracking teachers for body and hand are distilled into separate priors, which are used as bases for downstream coordinated latent-residual RL policies.

This decomposition eliminates the need for direct high-dimensional joint-space exploration and allows decoupling of wrist placement (emergent from body prior) and fine-grained finger dynamics (hand prior), with both reusable across downstream tasks.

Training and Reward Structure

The RL policy is trained in the Isaac Lab simulator using PPO, with task-dependent reward shaping driving complex skills such as non-stop dynamic grasping, articulated object opening during walking, and long-horizon manipulation involving turning and retention. The architecture’s main innovation lies in maintaining body/hand separation at both the prior and policy head levels, enhancing sample efficiency and task transferability. Evaluation is conducted on a 29-DoF Unitree G1 humanoid with a 20-DoF WUJI hand.

Experimental Results

Performance Across Diverse Loco-Manipulation Tasks

Experiments are performed on three representative tasks: WalkGrab (non-stop bottle grasping during locomotion), OpenFridge (handle grasp and backward stepping for door opening), and WalkPickTurn (approach, grasp, and object turn). CoorDex achieves high task success rates, especially on longer-horizon and contact-rich behaviors:

• WalkGrab: 0.55 (success), fall rate 0.00.
• OpenFridge: 0.66 (success), fall rate 0.00, mean door open angle 57.8°.
• WalkPickTurn: 0.89 (success), fall rate 0.01, min heading error 9.98°.

Notably, the same policy structure generalizes across task compositions with varying temporal and spatial coordination demands.

Action Space and Architecture Ablations

Ablation experiments on WalkGrab highlight the necessity for both the latent-prior action interface and body/hand residual decoupling. Direct joint-space control in either the full or hand subsystem yields trivial policies that exploit reward loopholes but never achieve functional grasp. Removing the hand prior (Body Prior + Hand Joint Space) leads to policies that reach but cannot generate robust finger coordination, often defaulting to a stationary grasp attempt—directly contradicting non-stop requirements.

Coordinated residual policies outperform monolithic latent structures by a statistically significant margin (success 0.55 versus 0.0), with smoother trajectory generation (lower action rates) and more reliable grasp/contact stability. This supports the architectural claim that separate residual heads conditioned through a shared context yield superior physical plausibility and task throughput.

Real-World Validation

Figure 3: Real-world execution of WalkPickTurn on G1+Dex3-1, demonstrating policy replay compatibility with hardware kinematics for approach, pick, and turn.

Figure 4: Real-world WalkGrab rollout showing bottle grasp and carry on hardware.

Figure 5: OpenFridge demonstration on real robot using a mock-up, maintaining grasp during backward stepping and door opening.

Policy rollouts recorded in simulation are replayed on physical G1+Dex3-1 robots, demonstrating the compatibility of the generated joint-space trajectories with the real platform even under hardware hand swaps, further validating factorization of body-hand priors.

Theoretical and Practical Implications

CoorDex empirically and structurally validates that dexterous loco-manipulation with high-DoF systems is tractable under a coordinated latent residual framework. By explicitly modeling body/hand priors and maintaining architectural separation at the RL/policy level, the method navigates the exploration bottleneck inherent in high-dimensional continuous control, providing a pathway towards skill reuse and platform transfer (including hand replacements). Policy architectures that collapse body/hand reasoning into single-head monolithic networks fail to capture the underlying coordination manifold, particularly for dynamic non-prehensile and prehensile interaction during locomotion.

Practically, this enables advanced humanoids to manipulate objects without pausing, interact with articulated structures while moving, and maintain robust grasp/transport under dynamic balance—all required for real-world mobile manipulation in human environments. CoorDex is configuration-agnostic, and its modular tracking-distillation-coordination stack allows rapid retargeting to new morphologies, facilitating hardware transfer and fleet-wide deployment.

Limitations and Future Prospects

The current instantiation relies on privileged proprioceptive and contact information, omits perception, and is evaluated with fixed morphologies and constrained object sets. Sim-to-real transfer for raw policies, adaptation to fully vision-based feedback, generalization to unseen morphologies, and performance on extended manipulation programs remain open areas. Extending CoorDex with task-conditioned priors, perception-based state estimation, and autonomous curriculum generation are future directions that could expand applicability to unconstrained environments and arbitrary manipulation pipelines.

Conclusion

CoorDex introduces a scalable framework for simultaneous locomotion and dexterous manipulation in humanoid robots that leverages coordinated latent residual policies over body and hand priors. Empirical results across challenging loco-manipulation tasks and hardware replays substantiate claims regarding trainability, compositionality, and generalization of high-DoF control in a unified architecture. The explicit decoupling of body and hand priors, combined with coordinated policy heads, proves essential for robust execution of complex, dynamic, contact-rich skills required for practical humanoid deployment.

---

Reference:

CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation (2606.23680)