HumanEgo: Zero-Shot Robot Learning

• The paper presents a method that converts minutes of human egocentric video into entity-centric representations for zero-shot robot learning.
• The methodology employs a flow-matching policy with dense auxiliary objectives to achieve up to 95% success in varied manipulation tasks.
• The approach enables hardware-agnostic transfer without any robot data, outperforming traditional teleoperation by over 41 percentage points.

HumanEgo: Zero-Shot Robot Learning refers to a data-efficient, robot-data-free manipulation learning framework that leverages minutes of human egocentric videos to train transferable closed-loop robot policies without requiring any robot demonstrations or in-domain robot data. HumanEgo addresses the embodiment gap—differences in hand-object appearance and kinematics—by converting in-the-wild human demonstrations into entity-centric representations and employs a flow matching policy augmented with dense auxiliary supervision to achieve robust, hardware-agnostic, zero-shot transfer to novel robots and environments (Zhi et al., 24 May 2026).

1. Entity-Level Representation and Embodiment Bridging

HumanEgo begins by parsing a human demonstration as an egocentric video sequence $\{I_t\}_{t=1}^T$ and extracting semantic, entity-level states for every frame:

$$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$

where $x_t^{(k)}\in \mathrm{SE}(3)$ is the 6-DoF pose of object $k$, $h_t^L, h_t^R\in \mathrm{SE}(3) \times [0,1]$ are left/right hand end-effector poses with normalized grasp state, and $N$ is the variable number of physical entities present.

To bridge the appearance and kinematics gap, HumanEgo performs arm/hand segmentation using SAM2, removes them via LaMa inpainting, and re-renders a virtual robot gripper and projected object keypoints into the reconstructed scene. Spatial representation is encoded as "Interaction-Centric Tokens" (ICTs) for each entity:

$$ICT_k = \left[\tau;\ {}^{\mathrm{REF}}T_E;\ {}^E T_{LH};\ {}^E T_{RH};\ g\right] \in \mathbb{R}^{29}$$

where each ICT aggregates type, reference-frame pose, hand poses relative to the entity, and the grasp scalar. This tokenization creates viewpoint- and morphology-invariant descriptors.

2. Flow-Matching Policy and Dense Auxiliary Objectives

The core learning objective employs a conditional flow-matching paradigm. The policy is defined as a velocity field

$v_\theta(\mathbf{x}_t, t\,|\,s_t)$

that, via linear interpolation $x_t = (1-t)x_0 + t x_1$ for $t\sim \mathcal{U}(0,1)$ between Gaussian noise $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$0 and the ground-truth action $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$1, matches the action delta with weighted per-component MSE:

$$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$2

with $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$3, $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$4, $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$5.

To maximize supervision from the dense human trajectories, HumanEgo incorporates auxiliary losses:

• Object Motion Loss $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$6: MSE on future object $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$7 trajectories.
• 2D Trace Loss $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$8: L2 loss on predicted future 2D keypoint traces.
• Latent Consistency Loss $$s_t = \left\{x_t^{(k)},\ h_t^L,\ h_t^R\right\}_{k=1}^N$$9: MSE between predicted and target ICT tokens $x_t^{(k)}\in \mathrm{SE}(3)$0 steps ahead.

The total loss for policy training is

$x_t^{(k)}\in \mathrm{SE}(3)$1

which enables amplification of supervision at every step.

3. Model Architecture, Training, and Inference

The policy network is built from a ViT-style patch embedding module (for $x_t^{(k)}\in \mathrm{SE}(3)$2 RGB images), linear embeddings of the $x_t^{(k)}\in \mathrm{SE}(3)$3 ICT tokens, and a 6-layer Transformer decoder (8 heads, embedding dim 384, dropout 0.05). Image and ICT context are fused using cross-attention, and $x_t^{(k)}\in \mathrm{SE}(3)$4-step action tokens interact through self-attention.

Training employs AdamW with a base learning rate $x_t^{(k)}\in \mathrm{SE}(3)$5, batch size 32, 400 epochs, and on-the-fly data augmentation (photometric jitter, random crop/erase, added target noise, region-biased attention). During inference, a fixed-step Euler integrator produces a 50-step trajectory chunk, replanning at 10 Hz and execution at 5 Hz with smoothing (EMA on position, SLERP on rotation, overlap blending, and delta safety cages).

4. Zero-Shot Transfer, Real-World Evaluation, and Generalization

HumanEgo is robot-data-free and hardware-agnostic: no in-domain robot demonstrations or internet-scale pretraining are required; all training relies on minutes of Aria-smart-glasses egocentric human video per task.

Deployment on Trossen WidowX, Franka, UR10, and environments with various cameras (RealSense, ZED) proceeds zero-shot using the same ICT-format entity tokenization for both human and robot scenes. The system achieves a mean success rate of 92.5% across four diverse real-world manipulation tasks given 30 minutes of human video per task (Serve Bread: 95%; Downstack Cups: 87.5%; Water Flowers: 95%; Adjust Table: 92%), and still attains 75% with only 15 minutes per task. Matched-time robot teleoperation is outperformed by +41.3 percentage points, and generalization to OOD settings without fine-tuning maintains 85–91% success (Zhi et al., 24 May 2026).

Task	HumanEgo (30min)	Teleop (30min)
Serve Bread	95.0%	51.2%
Downstack Cups	87.5%	45.0%
Water Flowers	95.0%	30.0%
Adjust Table	92.0%	60.0%
Average	92.5%	51.2%

Ablation experiments indicate the critical role of the auxiliary objectives (e.g., +17.5 pp for object-motion, +12.5 pp for latent consistency, +25 pp total gain), and ICT representations (from 7.5% with raw RGB to 95% with full visual+ICT). Human video data yields 4x higher spatial density and 3x smoother motion than teleoperation.

5. Comparative Context and Related Methodologies

HumanEgo introduces a paradigm distinction from prior zero-shot methods relying either on synthetic overlays (Dessalene et al., 4 Oct 2025), plan translation modules with policy adaptation (Bharadhwaj et al., 2023), or direct action prediction and regressive mapping from passive videos (Bharadhwaj et al., 2023). Unlike systems that require robot-labeled data, paired imitation, or internet-scale pretraining, HumanEgo achieves full closed-loop control with hardware-aligned perceptual and kinematic representations extracted entirely from naturalistic human data.

A key property is the use of entity-centric, viewpoint-invariant ICTs, which enables policies to generalize morphologically (across robotic hardware), spatially (across environments and viewpoints), and semantically (across object/task variations) without any robot-in-the-loop adaptation or pre-deployment calibration, as evidenced by robust zero-shot rollout in unseen scenarios.

6. Limitations, Open Challenges, and Future Research

Observed limitations include dependence on stereo Aria tracking (monocular methods degrade to 0–45% success), offline object pose estimation requiring reliable multi-view/tracking pipelines (dynamic and in-hand manipulation necessitate real-time trackers), and susceptibility to cascading failures from independent perception modules. Precision currently saturates at ~1 cm; high-precision or dexterous manipulation may require future integration with reinforcement learning or tactile (haptic) feedback.

Planned research directions include monocular or learned stereo hand estimation, joint/recurrent perception-trajectory models, sub-centimeter precision via hybrid RL, and extension of ICT representations to deformable objects and fine contact modeling (Zhi et al., 24 May 2026).

Bibliographic Note and Landscape

HumanEgo was introduced by Lin, Majaj, and collaborators in 2026, building on lines of work in entity-centric representation for passive-to-robot transfer, synthetic overlays, flow-matching imitation, and dense human video annotation pipelines. It advances the state of the art in data-efficient, zero-shot, hardware-agnostic robot policy learning from naturalistic human video by combining robust perceptual abstraction with dense supervisory regularization, achieving state-of-the-art real-world success rates without recourse to robot data or calibration (Zhi et al., 24 May 2026).