Ego-Pi: Cross-Embodiment Dexterous Manipulation

• Ego-Pi is a framework that integrates egocentric human data with humanoid demonstrations to imbue robots with high-level task semantics like sorting logic and compositional skills.
• It employs explicit cross-embodiment alignment, mapping human hand and wrist observations to robot action spaces while preserving compatibility with a pretrained vision–language–action model.
• Empirical results demonstrate that co-training with human data significantly improves task success rates, outperforming robot-only models in complex dexterous manipulation tasks.

Ego-Pi is a framework for fine-tuning a pretrained vision–language–action foundation model, $\pi_{0.5}$, on jointly curated egocentric human demonstrations and humanoid robot demonstrations for dexterous five-finger manipulation. Its central claim is that human egocentric data can supply task semantics that are absent from robot datasets, allowing a robot to acquire sorting logic, skill composition, and rule-based ordering even when no corresponding robot demonstrations exist for those semantics. The framework is explicitly cross-embodiment: it aligns human hand and wrist observations to humanoid robot action spaces while preserving compatibility with the pretrained $\pi_{0.5}$ action head (Kim et al., 6 Jun 2026).

1. Problem setting and scope

Ego-Pi is motivated by data scarcity in robotics. Unlike language and vision, robot learning does not have internet-scale data, because collecting interaction data requires expensive hardware and real-world operation. Ego-Pi addresses this by using egocentric human data, which can be collected more easily and at greater breadth, and co-training it with robot data for dexterous humanoid manipulation (Kim et al., 6 Jun 2026).

The framework targets a setting in which both human and humanoid embodiments have dexterous five-finger hands. This makes the problem materially different from gripper-centric VLA systems. The action space is high-dimensional, retargeting human hands into robot joint space is nontrivial, and naive inverse kinematics can produce self-collisions or unnatural poses. Ego-Pi therefore treats cross-embodiment alignment as a first-class systems problem rather than as a peripheral preprocessing step (Kim et al., 6 Jun 2026).

A common misunderstanding is that the framework treats human data as a replacement for robot data. The reported experiments support a narrower claim: human data provides high-level semantics that are missing from robot demonstrations, while robot data still supplies the base skills on which those semantics are composed. In the reported tasks, robot data covers isolated skills or generic placements, whereas human data contributes the novel rule, ordering, or compositional structure (Kim et al., 6 Jun 2026).

2. Foundation model and action formalism

Ego-Pi is built on $\pi_{0.5}$, a flow-matching VLA that extends a pretrained vision–LLM to predict short-horizon continuous actions. At time $t$, the model consumes language instruction $\ell_t$, images $(I_t^{ego}, I_t^{L}, I_t^{R})$ from head and wrist cameras, and proprioceptive state $s_t$. It can also optionally emit a subtask language output $\hat{\ell}_t$ before action prediction (Kim et al., 6 Jun 2026).

Its core action objective is the flow-matching loss

$$L_{\text{flow}}(\theta)=\mathbb{E}_{\tau,\omega}\left[\left\|\omega-a_{t:t+H}-f_\theta\!\left(a_{t:t+H}^{\tau,\omega},o_t,\ell_t\right)\right\|_2^2\right],$$

with

$$a_{t:t+H}^{\tau,\omega}=\tau a_{t:t+H}+(1-\tau)\omega,$$

where $\pi_{0.5}$0, $\pi_{0.5}$1, and $\pi_{0.5}$2. This preserves the pretrained continuous-action head of $\pi_{0.5}$3 rather than replacing it with a new decoder (Kim et al., 6 Jun 2026).

For dexterous control, Ego-Pi uses a per-hand action representation

$\pi_{0.5}$4

where $\pi_{0.5}$5 is wrist position, $\pi_{0.5}$6 is wrist orientation in 6D rotation, and $\pi_{0.5}$7 are Tesollo hand joint angles; for Inspire hands, $\pi_{0.5}$8. Bimanual Tesollo control therefore yields 58-dimensional actions (Kim et al., 6 Jun 2026).

Because $\pi_{0.5}$9 supports up to 32-dimensional action tokens, Ego-Pi introduces action interleaving rather than modifying the pretrained head. If

$\pi_{0.5}$0

then the interleaved sequence is

$\pi_{0.5}$1

This halves the effective horizon from $\pi_{0.5}$2 to $\pi_{0.5}$3, but the reported training remains stable and convergent. The token layout places visual, language, and state embeddings before <SUBTASK>, <ACT_L>, and <ACT_R>, with causal masking and subtask tokens predicted before action tokens (Kim et al., 6 Jun 2026).

3. Cross-embodiment alignment and optimization

Ego-Pi uses an explicit robot-centric mapping from human demonstrations into robot action space. For hand joints, if $\pi_{0.5}$4 denotes human joint angles from Manus gloves or HaMeR reconstructions, the mapping is

$\pi_{0.5}$5

where $\pi_{0.5}$6 is a per-joint offset and $\pi_{0.5}$7 is a per-joint scale. This defines a direct projection $\pi_{0.5}$8, or $\pi_{0.5}$9 for Inspire hands. The paper presents this as an alternative to wrist-plus-fingertip inverse kinematics, specifically to avoid unnatural or colliding poses (Kim et al., 6 Jun 2026).

Wrist pose is mapped separately. Quest controller tracking provides 6D wrist pose relative to the headset, and the robot reproduces it with inverse kinematics and PD control:

$t$0

The complete action mapping from human embodiment $t$1 to robot embodiment $t$2 is

$t$3

with $t$4 a diagonal scaling on joint angles and optional sign changes for certain joints (Kim et al., 6 Jun 2026).

Visual alignment is reinforced with colored skeleton overlays that explicitly encode finger correspondence and occlusions across human and robot embodiments. The overlays are described as occlusion-aware and finger-specific. Notably, Ego-Pi does not introduce a learned cross-embodiment alignment loss; alignment is achieved through the explicit mapping $t$5 and these visual overlays (Kim et al., 6 Jun 2026).

The optional subtask head adds a language objective

$t$6

and the full training objective is

$t$7

Fine-tuning uses AdamW with $t$8, $t$9, weight decay $\ell_t$0, gradient clip $\ell_t$1, cosine learning-rate schedule, warmup ratio $\ell_t$2, batch size $\ell_t$3, and $\ell_t$4–$\ell_t$5 training steps. Human and robot data are co-trained in 50/50 batches, and robot wrist images are dropped out 40% of the time to mitigate the modality mismatch created by the absence of wrist images in human data (Kim et al., 6 Jun 2026).

4. Data collection, hardware, and evaluation protocol

The robot platform is the Galaxea R1 Pro. Its sensor suite includes a head-mounted ZED mini camera and Ardurocam wrist cameras on both end-effectors with $\ell_t$6 FoV. The hands are either Tesollo with 20 joints or Inspire with 6 joints. Finger joints are recorded with Manus gloves, and wrist pose is captured by Quest controllers; both streams run at 100 Hz (Kim et al., 6 Jun 2026).

Human data collection uses Manus gloves and Quest controllers for hand and wrist tracking, plus a ZED mini on a table for third-person view. Human data does not include wrist images. The synchronization protocol records human and robot kinematics at 100 Hz, vision at the respective cameras, and uses subtask labels when the auxiliary training head is enabled (Kim et al., 6 Jun 2026).

The experiments cover three tabletop tasks: Tomato sorting by color, Boxing, and Packaging. The dataset composition reported for those tasks is as follows.

Task	Human demos	Robot demos
Tomato Sort	89 (13 min)	150 (60 min)
Boxing	60 (5 min)	144 (21 min)
Packaging	96 (11 min)	185 (27 min)

Evaluation uses success rate as the primary metric, defined as task completion under correct semantics. The environments include bowls, boxes, blocks, and a bear doll. Ablations compare robot-only training against co-training with human data, assess the contribution of the subtask generation head and skeleton overlays, compare Tesollo against Inspire hands, and test the effect of removing wrist cameras at inference time (Kim et al., 6 Jun 2026).

5. Empirical behavior, transfer effects, and ablations

The reported results show that co-training with human egocentric data transfers semantic structure that robot-only training does not recover. In Tomato sorting, co-training reaches 92% success, whereas robot-only training drops to 40%; the failure mode is that the robot places tomatoes randomly rather than following the color-based multi-bowl sorting rule. In Packaging, co-training reaches 90% success, whereas robot-only drops to 10%; the reported failures involve wrong object ordering and frequent collisions (Kim et al., 6 Jun 2026).

The Boxing task is the clearest test of multi-step semantic composition. Simple co-training without the subtask head reaches 27% success, and the policy often reaches for the block before opening the box or attempts both actions simultaneously, causing collisions. With subtask generation, co-training reaches 93% success on the Inspire hand. On Tesollo, the same configuration reaches 67% success: ordering improves, but the transition from opening to placement remains difficult, which the paper attributes to sensitivity to states and possible embodiment mismatch associated with the larger hand (Kim et al., 6 Jun 2026).

Task	Setting	Success
Tomato sorting	Co-training	92%
Tomato sorting	Robot-only	40%
Packaging	Co-training	90%
Packaging	Robot-only	10%
Boxing	Co-training, no subtask head	27%
Boxing	Co-training + subtask, Inspire	93%
Boxing	Co-training + subtask, Tesollo	67%

Several ablations qualify these gains. Wrist cameras remain critical at test time: removing them significantly degrades grasp stability, including in Tomato sorting. This is important because the human data stream has no wrist images; the reported result is that training-time wrist-image dropout helps modality alignment, but does not make wrist cameras dispensable at deployment. The paper also reports that embodiment similarity matters: Inspire, being closer to human size, outperforms Tesollo on complex sequencing (Kim et al., 6 Jun 2026).

These findings constrain the interpretation of “human data enables robots to learn new task semantics.” The evidence supports transfer of semantics such as color-based sorting, “box first, then bear,” and prerequisite ordering in box opening before placement. It does not support the stronger claim that human data alone suffices for full dexterous control; the experiments rely on co-training with robot data that already covers isolated skills or generic placements (Kim et al., 6 Jun 2026).

6. Position within egocentric research and open directions

Ego-Pi belongs to a broader egocentric systems landscape, but its problem formulation is distinct. It is a cross-embodiment VLA fine-tuning method for dexterous manipulation, not a perception-only model. Adjacent egocentric work addresses complementary problems: rectilinear HMD body-pose recovery with probabilistic joint rotations in SimpleEgo (Cuevas-Velasquez et al., 2024), synthetic data generation for egocentric mapping, tracking, and human mesh recovery in EgoGen (Li et al., 2024), and efficient multisensory inference under compute and energy constraints in EgoAdapt (Chowdhury et al., 26 Jun 2025). These systems do not define Ego-Pi as a named VLA framework, but they address the same class of egocentric sensing, data, and deployment pressures.

This context clarifies Ego-Pi’s technical niche. SimpleEgo emphasizes uncertainty-aware egocentric body modeling on resource-constrained head-mounted devices (Cuevas-Velasquez et al., 2024). EgoGen supplies large-scale, richly annotated synthetic egocentric data and closes the loop between embodied perception and motion (Li et al., 2024). EgoAdapt shows that egocentric systems often require explicit compute-aware modality selection and distillation to meet edge constraints (Chowdhury et al., 26 Jun 2025). A plausible implication is that future Ego-Pi-like systems may increasingly integrate dexterous manipulation transfer with synthetic pretraining, on-head perception, and efficiency-aware deployment.

The paper’s stated limitations remain substantial. Transfer can break down in long-horizon dexterous tasks; the system is sensitive to state inputs; occlusions or inaccurate hand-pose reconstructions can degrade the human-to-robot mapping; and large embodiment mismatches make sequencing brittle. The reported future directions are better retargeting, adding wrist cameras or multi-view human recordings to reduce modality gaps, scaling human corpora to more complex tasks and tool use, incorporating tactile and force sensing, extending to long-horizon and mobile manipulation, combining imitation with closed-loop RL fine-tuning, and exploring unified cross-embodiment objectives that jointly optimize visual and kinematic domains (Kim et al., 6 Jun 2026).

In that sense, Ego-Pi is best understood not as a generic synonym for egocentric AI, but as a specific proposal for transferring task semantics from egocentric human demonstrations into dexterous humanoid robot control through explicit action-space alignment, token-level action interleaving, and auxiliary subtask prediction (Kim et al., 6 Jun 2026).