HumanEgo: Egocentric Robot Skill Transfer

• HumanEgo is a framework designed to extract transferable manipulation skills from first-person video using hand-object representations and generative flow-matching policies.
• It employs visual appearance normalization and kinematic abstraction to bridge embodiment gaps between human demonstrations and robotic execution.
• The framework demonstrates high data efficiency and robust skill transfer across diverse robotic hardware with only 15–30 minutes of training video.

HumanEgo is a robot learning framework that enables the extraction of transferable manipulation skills from minutes of human egocentric video, allowing deployment on diverse robotic hardware in a zero-shot, data-efficient manner. Bridging the longstanding embodiment gap in both appearance and kinematics between humans and robots, HumanEgo leverages entity-level hand-object representations and generative flow-matching policies to achieve high transfer performance using only first-person video, with no robot data required at training time (Zhi et al., 24 May 2026).

1. Motivations and Challenges in Egocentric Human-to-Robot Transfer

Egocentric video—collected from wearable, head-mounted devices—captures manipulation trajectories directly from the demonstrator's perspective. While this signal is rich in spatial and temporal cues, direct transfer of skills from such video to a robot is impeded by two key embodiment gaps: (i) differences in visual appearance, such as human anatomy, hand morphology, and backgrounds, and (ii) differences in kinematics and actuation, as seen in high-DoF human hands versus parallel-jaw grippers.

HumanEgo explicitly targets solutions to both gaps:

• Visual appearance normalization through hand/arm segmentation and inpainting, followed by virtual gripper rendering, producing robot-like inputs.
• Kinematic abstraction via Interaction-Centric Tokens (ICTs), which represent each hand and object in a 6-DoF frame, supporting hardware-agnostic and viewpoint-invariant skill transfer.

The overarching aim is to drastically reduce required human data—only 15–30 minutes per task—while supporting immediate zero-shot deployment on varied robotics platforms and environments (Zhi et al., 24 May 2026).

2. Egocentric Data Acquisition and Preprocessing

HumanEgo employs head-mounted, sensor-rich stereo devices (Project Aria Gen1) that record synchronized RGB video (30Hz, 2MP), stereo monochrome for SLAM-based tracking, IMUs, and eye-tracking. The hardware yields online-calibrated 6-DoF camera trajectories and accurate 3D hand keypoints.

Key elements:

• Task coverage: 15–30 minutes (27k–54k frames) of unconstrained manipulation per task, with diverse backgrounds and lighting. Collection requires no workspace calibration or robot in the scene.
• Multi-view anchoring: Before each demonstration, users perform a "scene sweep" (1–2 s head motion) for improved 3D object triangulation.
• Appearance normalization: Human hands and arms are segmented and inpainted using SAM2 and LaMa, removing species-specific visual cues, then canonical virtual gripper models are rendered in place.

This robot-data-free, scalable collection suits real environments and arbitrary scene configurations (Zhi et al., 24 May 2026).

3. Interaction-Centric Token Representation

At the core of HumanEgo is the Interaction-Centric Token (ICT), an entity-level token encoding all relevant agents and objects in a scene. Each ICT is a 29-dimensional vector for every hand and object, encapsulating:

• $\tau$: Entity type indicator (hand or object).
• ${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$: Entity's 6-DoF pose in a canonical reference frame.
• ${}^{E}\!T_{LH/ RH}$: Relative 6-DoF pose of both hands in the local frame of the entity.
• $g$: Grasp state (binary for hands; sentinel for objects).

Each SE(3) is flattened into a 9D vector (3D translation $+$ 6D continuous rotation [Zhou et al., 2019]). ICTs are computed per timestep and stacked for all active entities (Zhi et al., 24 May 2026). This object-centric, relative encoding provides invariance to viewpoint, morphology, and absolute positioning, enabling direct conditioning of policy networks across human and robot embodiments.

4. Flow-Matching Policy and Auxiliary Objectives

HumanEgo formulates action generation as a conditional flow-matching problem. The policy predicts a $K$-step action "chunk" $\mathbf{x}_1\in\mathbb{R}^{K\times D_a}$ (packing all hand/gripper DoF and grasp states) using a transformer that models a conditional velocity field $v_\theta$ trained via the flow-matching loss:

$$\mathcal{L}_\mathrm{FM} = \mathbb{E}_{t\sim U(0,1),\,\mathbf{x}_0\sim\mathcal{N}(0,I),\,\mathbf{x}_1} \big[ w_p\|\Delta\mathbf{p}\|^2 + w_r\|\Delta\mathbf{r}\|^2 + w_g\|\Delta g\|^2 \big]$$

with $\Delta(\cdot)$ defined as

${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$0

Dense auxiliary objectives are simultaneously optimized:

• ${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$1: MSE on ground-truth future object 6-DoF poses.
• ${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$2: MSE on predicted keypoint 2D traces.
• ${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$3: MSE in latent ICT space for multi-step consistency.

The combined loss is:

${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$4

These auxiliary heads extract signal from every frame and trajectory, substantially increasing policy data-efficiency and robustness, with the greatest effect in the lowest-data regime (gains of +20 percentage points at 8 minutes demonstrated) (Zhi et al., 24 May 2026).

5. Learning Pipeline and Zero-Shot Deployment

The policy architecture is an explicit context-conditioned transformer decoder (6 layers, 8 heads, embedding dimension 384), aggregating both visual and ICT context streams. Key training details include:

• AdamW optimizer (${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$5) with cosine decay, 400 epochs, batch size 32, gradient clipping.
• Photometric and spatial augmentations, temporal interpolation, state noise injection on ICTs.
• Region-attention focusing to bias vision toward the manipulation anchor.

At inference:

• The learned velocity field is integrated via fixed-step Euler over 20 steps, producing a 50-step action chunk.
• Replanning occurs at 10 Hz, with actions executed at 5 Hz.
• Grasp is triggered when predicted probability ${}^{\mathrm{REF}}\!T_E\in\text{SE}(3)$6.
• Output position and orientation are smoothed (EMA for position, SLERP for orientation), and per-step motion is clamped for safety.

Deployment is robot-agnostic, requiring only a top-down RGB camera for localization and a parallel-jaw manipulator (Trossen, Franka, UR series); no adaptation or finetuning is performed (Zhi et al., 24 May 2026).

6. Empirical Results: Data Efficiency, Generalization, and Comparison

HumanEgo achieves 92.5% average success across four manipulation tasks—Serve Bread, Downstack Cups, Water Flowers, Adjust Table—using only 30 minutes of human demonstration video per task; with only 15 minutes, the average success is 75%. This performance exceeds five recent zero-shot imitation learning baselines (range: 1.9%–45%) and substantially outperforms matched-time robot teleoperation learning (HumanEgo: 92.5%, ACT@30min: 51.2%), a +41 percentage point absolute margin (Zhi et al., 24 May 2026).

Zero-shot transfer is robust to:

• Variations in camera viewpoint, lighting, environment.
• Unseen object instances.
• Different robot actuators and camera hardware.
• Arbitrary scene configurations without retraining.

Auxiliary losses provide pronounced gains when training data is most limited, underscoring the importance of multi-headed supervision for efficient and robust policy learning.

7. Limitations and Research Directions

Current limitations include dependence on accurate stereo hand tracking (Aria MPS); reduced performance with monocular keypoint estimators; single-frame object pose tracking susceptible to errors during in-hand re-orientation; reliance on pipelined third-party modules leading to possible error cascades; and the lack of sub-centimeter precision, which may necessitate incorporating reinforcement learning or small amounts of robot data for refinement (Zhi et al., 24 May 2026).

Future research will likely explore end-to-end training for perception/action frontends, integration of dense real-time object tracking, and scalable methods for more finely resolved manipulation.

---

In summary, HumanEgo establishes a new paradigm for human-to-robot skill transfer from wearable egocentric video, achieving state-of-the-art zero-shot manipulation through entity-centric abstraction and dense data-efficient policy learning (Zhi et al., 24 May 2026).