HEAD: Hand-Eye Autonomous Delivery

• HEAD is a modular framework for humanoid robots that learns coordinated navigation, locomotion, and reaching from human motion and vision data.
• It employs a two-level architecture, with a high-level egocentric planner and a low-level whole-body controller tracking sparse hand and eye targets.
• The system overcomes isolated navigation or manipulation limitations by integrating human-inspired cues for robust task-level hand-eye delivery in complex indoor environments.

Hand-Eye Autonomous Delivery (HEAD) denotes a modular framework for humanoid robots that learns navigation, locomotion, and reaching skills directly from human motion and vision perception data. In this formulation, a high-level planner commands the target position and orientation of the hands and eyes of the humanoid, and a low-level policy controls the whole-body movements required to realize those targets. The framework addresses a compound embodied problem: moving through human-designed environments, keeping a target in view, approaching it safely, and finally reaching or touching it with the hands (Chen et al., 5 Aug 2025).

1. Problem setting and conceptual scope

HEAD is motivated by the observation that human manipulation starts with moving to the object. A humanoid that only has a reaching controller but no navigation, or only a navigation controller but no reaching, cannot robustly perform useful physical tasks in cluttered human environments. The framework is therefore organized around a shared hand-eye abstraction in which perception and action are coordinated through the robot’s eyes and hands rather than through a monolithic end-to-end visuomotor policy (Chen et al., 5 Aug 2025).

Within this abstraction, navigation is not treated as planar locomotion alone. The robot must control posture, head orientation, balance, and reaching simultaneously, often in 3D clutter where obstacles and targets occur at different heights. HEAD is explicitly designed to overcome two limitations identified in prior system designs: separating navigation and manipulation into weakly coordinated modules, and attempting a fully end-to-end policy that is difficult to train because it requires heterogeneous data and faces a large embodiment gap. This suggests that HEAD should be understood not as a narrow reaching method, but as a task-level framework for coordinated humanoid mobility and contact.

The operational target of the system is “hand-eye delivery” capability: moving through a human space, keeping a goal visible, transitioning from room-scale motion to local approach, and making contact. In the reported real-world tasks, this capability is evaluated as reaching objects placed at different locations and heights in indoor layouts unseen during training (Chen et al., 5 Aug 2025).

2. Modular architecture and hand-eye representation

HEAD uses a two-level decomposition. The high-level policy reasons from egocentric visual input and predicts target trajectories for the robot’s eyes and hands. The low-level whole-body controller then tracks those targets with the robot’s full body. The paper describes this modular approach as one that decouples the ego-centric vision perception from physical actions, promoting efficient learning and scalability to novel scenes (Chen et al., 5 Aug 2025).

The high-level policy is itself split into a navigation module and a reaching module. The navigation module drives the robot across room-scale distances while keeping the goal in view. The reaching module takes over when the target is close enough and visible in a downward-facing RGB-D camera, then computes target hand and head poses for final approach and contact. The low-level controller receives desired target poses for three body points—eyes or head, left hand, and right hand—and outputs joint commands that realize those targets through coordinated whole-body motion.

A central representation is the sparse three-point target. Rather than tracking a full-body motion trajectory, the controller tracks head, left-hand, and right-hand targets. In the main text this is written as

$$\mathbf{g} = [\mathbf{p}^\text{tar}_\text{eye}\ominus\mathbf{p}_\text{eye};\mathbf{p}^\text{tar}_\text{l-hand}\ominus\mathbf{p}_\text{l-hand};\mathbf{p}^\text{tar}_\text{r-hand}\ominus\mathbf{p}_\text{r-hand}]$$

where $\ominus$ denotes the relative transformation operator and “tar” denotes target pose (Chen et al., 5 Aug 2025).

The transition from navigation to reaching is explicitly staged. Room-scale navigation uses a wide-angle webcam. Final reaching uses a downward-facing RGB-D camera. When the target enters the downward-looking RGB-D camera’s view and is close enough to be within reaching range, the goal is transferred to the RGB-D frame using LoFTR correspondence matching. The reaching module then approximates the goal as a 3D hand position, and inverse kinematics via Mink is used to infer the target hand orientations and the head pose. The inverse-kinematics procedure is initialized from the current robot state and includes a smoothness objective discouraging large changes in center-of-mass position and pelvis orientation, which supports a continuous transition from walking to reaching (Chen et al., 5 Aug 2025).

3. Low-level whole-body controller

The low-level policy is trained in simulation and deployed on the real robot. The humanoid has 33 links and 27 controllable joints. The state input uses two historical frames and is processed by a GRU. The state space is

$$\mathbf{s}_t \in \mathbb{R}^{33 \times 7 \times 2},$$

where each link is represented by position plus quaternion orientation in the local frame of the root link. The control state is

$\mathbf{c}_t \in \mathbb{R}^{30},$

including root angular velocity and joint velocity locally, but not root linear velocity because that is difficult to access on the real robot. The goal vector is

$\mathbf{g} \in \mathbb{R}^{3\times3\times7},$

containing the target positions and orientations for the left hand, right hand, and head over the next three frames. The final observation is

$$\mathbf{o}_t = (\mathbf{s}_t,\mathbf{c}_t,\mathbf{g}_t),$$

and the action is

$\mathbf{a}_t \in \mathbb{R}^{27},$

executed via PD servos at 30 Hz (Chen et al., 5 Aug 2025).

The controller is trained with a multi-objective RL framework and a GAN-like imitation setup. The final optimization objective is

$$\max \mathbb{E}_t\left[\sum\nolimits_\kappa w_k \bar{A}_{t,k} \log \pi(\mathbf{a}_t | \mathbf{o}_t) \right]$$

with $w_{\text{imit}, i} = 0.2$ for each imitation objective and $w_g = 0.6$ for the goal-directed objective (Chen et al., 5 Aug 2025).

A distinctive design choice is the use of two discriminator groups, one for upper-body motion and one for lower-body motion. The paper reports that a single discriminator over all motion leads to worse tracking and poorer sim-to-sim transfer. Separate discriminators are intended to preserve compositionality between arm manipulation and walking or balance behaviors. This matters because walking clips naturally include arm swings, whereas manipulation may require a crouched or stationary upper body.

The goal-directed reward combines tracking and regularization terms:

$\ominus$0

The tracking term averages position and orientation errors over the set $\ominus$1, while the auxiliary terms penalize insufficient swing-foot airtime, foot sliding, and actuation effort. Training uses PPO as the RL backbone, Adam optimization, and domain randomization over mass, friction, PD gains, and action delay (Chen et al., 5 Aug 2025).

A plausible implication is that the low-level controller is best viewed as a sparse-target whole-body tracker rather than as a task-specific reaching policy. Because only three points are specified, the controller must infer full-body coordination from the distribution of human motion rather than from a rigid demonstration trace.

4. High-level policy, egocentric perception, and data adaptation

The high-level policy is trained from a combination of human egocentric data collected with Aria glasses, robot-specific trajectory data in the training room, and out-of-distribution human egocentric data from the Aria Digital Twin (ADT) dataset. For navigation, the training tuples are

$\ominus$2

where $\ominus$3 is the current egocentric image, $\ominus$4 is the future camera trajectory, and $\ominus$5 is the 2D goal point in the image (Chen et al., 5 Aug 2025).

At inference time, the navigation model receives the current RGB image and a tracked 2D goal point from a point tracker. It extracts DINO features from the image, appends the goal coordinate, and feeds them to a transformer decoder. The output is a future eye or camera trajectory $\ominus$6, represented as transformations relative to the previous frame. This makes the navigation module a learned short-horizon visual motion planner conditioned on an egocentric target location.

Aria glasses are central because they provide accurate camera poses, static point clouds, and gaze estimation. The training pipeline converts raw Aria footage into goal-conditioned navigation data by finding the closest point in the static point cloud along the future gaze vector and projecting that point into the image plane using the current camera pose. This yields the 2D goal point used for supervision.

Human egocentric data are not used without adaptation. Because humans and the Unitree G1 robot differ in camera geometry and body posture, the method applies undistortion of Aria fisheye images, a homography transform to match the robot’s pitch and viewing angle, and temporal subsampling because the robot moves about $\ominus$7 slower than humans. The paper reports that human data alone are not sufficient because of embodiment mismatch; robot data help align human egocentric trajectories with the robot’s actual motion dynamics; and the best performance comes from combining robot data from the training room, human data in the same room, and ADT human data for scale and generalization (Chen et al., 5 Aug 2025).

The paper also notes that a shared decoder for human and robot data works better than separate branches for navigation. This suggests that, for the navigation subproblem, cross-domain regularities in egocentric motion are better captured by a shared latent structure than by a hard architectural separation.

5. Training corpus, evaluation protocol, and empirical findings

The low-level controller is trained on about 5 hours of retargeted motion collected from AMASS and OMOMO. The paper manually selects 1363 clips of locomotion and locomotion-plus-manipulation motion, which are retargeted to the G1 robot using keypoint matching. High-level navigation training uses 200 human clips per room collected with Aria glasses, 38 robot trajectory clips in the lab and 20 in the kitchen, and the ADT dataset with about 400 minutes of human egocentric indoor activity (Chen et al., 5 Aug 2025).

The system is deployed on a Unitree G1 humanoid. Sensors include a wide-angle USB webcam for navigation and the G1’s built-in RealSense D435 RGB-D camera for reaching. Computation runs on a PC with an RTX 4090 and an i9-14900K. Real-world experiments are conducted in two indoor environments, a lab room and a kitchen, with furniture arranged to create different layouts. Tasks involve reaching 4 objects placed at different locations and heights in each layout (Chen et al., 5 Aug 2025).

The whole-body controller is evaluated using position error in meters, orientation error in radians, and failure rate. A timestep is counted as a failure if head height deviates from the target by at least 0.4 m. In simulation on Isaac Gym unseen motions, the full method achieves 0.075 m position error, 0.120 rad orientation error, and 0% failure. For sim-to-sim transfer in MuJoCo, it achieves 0.153 m position error, 0.326 rad orientation error, and 3% failure. The reported ablations show that the single-discriminator variant has worse errors and higher failure, while the no-discriminator variant fails catastrophically (Chen et al., 5 Aug 2025).

Navigation is evaluated by success rate and mean position error in open-loop prediction, where a prediction is successful if the final error is within 0.6 m, roughly the humanoid’s effective manipulation range. The strongest navigation result is obtained by combining robot, human, and ADT data in a shared decoder: Lab room SR = 0.86, Lab room error = 0.380, Kitchen SR = 0.60, and Kitchen error = 0.608. The paper emphasizes that robot data alone work reasonably in-domain but generalize poorly to the new kitchen, that human data in the same environment improve performance, that ADT significantly helps transfer to the unseen kitchen, and that using both human and robot data is best (Chen et al., 5 Aug 2025).

Across the end-to-end real-world task, the full HEAD system achieves a 71% success rate overall. Reported failure modes include motion blur causing the point tracker to lose the object or switch targets, aggressive paths seen in human data that may be infeasible for the robot and cause collisions, reflective surfaces and narrow corridors in the kitchen, and failures in the transition from navigation to reaching when goal tracking becomes unreliable. The authors also note that the system only uses head and wrist or hand poses as the human control interface, can be conservative in complex environments, and may hesitate because the mapping from sparse 3-point control to lower-body intention is ambiguous. They suggest that additional egocentric cues such as foot pose could be beneficial (Chen et al., 5 Aug 2025).

6. Related formulations, precursor ideas, and acronym ambiguity

The term “HEAD” is not unique across robotics and autonomous systems. In connected and autonomous vehicles, HEAD refers to a bandwidth-efficient cooperative perception method that fuses features from the classification and regression heads in 3D object detection networks. That method addresses the trade-off between communication bandwidth and perception performance, supports heterogeneous detectors such as LiDAR PointPillars, SECOND, VoxelNet, and camera Bird’s-eye View Encoder, and is evaluated on V2V4Real and OPV2V. Despite sharing the acronym, it is unrelated to humanoid hand-eye delivery (Qu et al., 2024).

A closer conceptual relative is DronePick, a teleoperation system for remote object picking and delivery by a human-controlled quadcopter. DronePick combines a modified Crazyflie 2.0 quadcopter, a magnetic grabber, a tactile glove with ERM vibromotors and a flex sensor, Vicon-based hand tracking, a VR application in Unity 3D, and an HTC Vive headset. The operator teleoperates the quadcopter by moving the hand; vibrotactile patterns indicate whether the object is left, right, forward, or directly under the drone; and a clasp gesture sends the pick command. The study reports an average 99.0% recognition rate and an average 2.36 s recognition time for the tactile patterns. The paper, however, explicitly characterizes DronePick as a human-in-the-loop, hand-eye teleoperated delivery system with tactile guidance rather than a fully autonomous delivery robot (Ibrahimov et al., 2019).

Another relevant antecedent is work on prioritized kinematic control of joint-constrained head-eye robots. That line of research formulates gaze as an operational-space orientation task, uses recursive null-space projections for prioritized fixation trajectories, and applies the Intermediate Desired Value approach to avoid discontinuities when joint-limit tasks are inserted or removed. The controller is reported to give a biomimetic vestibulo-ocular reflex under self-generated movements and is validated on the Dreamer humanoid robot. While this is not a delivery architecture, it provides control-theoretic machinery directly relevant to any system that must preserve high-priority gaze targets while lower-priority head motions proceed (Jorgensen et al., 2018).

These comparisons clarify two common sources of confusion. First, not every “HEAD” system concerns hand-eye humanoid delivery; the acronym is reused in unrelated fields. Second, not every hand-eye delivery system is autonomous in the same sense. DronePick couples hand motion, visual scene understanding, and object delivery in a closed interaction loop, but the operator continuously steers the platform. By contrast, the humanoid HEAD framework learns navigation, locomotion, and reaching from human motion and egocentric perception data, then executes those behaviors on a Unitree G1 in human indoor environments (Chen et al., 5 Aug 2025).