Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Data

Abstract: Robotics faces a fundamental challenge of data scarcity. Unlike language or vision research, there is no internet-scale dataset for robotic manipulation. A promising path forward is to leverage egocentric human data, which can be collected more easily, with greater breadth, and at a larger scale. Towards this end, we investigate key design choices for learning across human and humanoid embodiments equipped with dexterous five-finger hands, using the $π_{0.5}$ model as a foundation. Our results show that human data enables robots to learn new task semantics and compose existing skills into novel behaviors without corresponding robot data. The paper website is here: https://egopipaper.github.io/

• The paper introduces a cross-embodiment VLA fine-tuning framework (Ego-Pi) that transfers high-level semantics from egocentric human demonstrations to humanoid robots.
• It employs a token interleaving mechanism and direct per-joint mapping to manage high-dimensional bimanual control without perturbing pretrained weights.
• Experiments on tasks like tomato sorting, skill composition, and packaging show significant performance gains with human-robot co-training over robot-only baselines.

Ego-Pi: Cross-Embodiment VLA Fine-Tuning for Dexterous Humanoid Manipulation

Motivation and Problem Statement

The core challenge addressed by "Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Data" (2606.08107) is the severe data scarcity afflicting robot learning, especially for high-dimensional dexterous manipulation. While internet-scale datasets accelerate progress in language and vision, physical robotics lacks equivalent large-scale sources, constraining task-space generalization, skill composition, and acquisition of semantically complex behaviors. Recent advances in anthropomorphic robot morphology and proliferation of egocentric wearable devices present a latent opportunity: leveraging scalable human demonstration data to enable novel cross-embodiment policy generalization. However, there are significant representational, algorithmic, and architectural mismatches between human and robot hand action spaces that have thwarted seamless semantic transfer at scale.

The Ego-Pi framework is engineered to close these cross-embodiment gaps. Building upon large, pretrained vision-language-action models (VLAs), the work investigates whether humanoid robots equipped with dexterous hands can acquire high-level task semantics present only in human data—specifically sorting, skill chaining, and rule-based ordering—by co-training on both human and robot modalities. Notably, Ego-Pi demonstrates explicit transfer of novel behaviors achievable solely through egocentric human demonstrations, advancing the frontier relative to prior approaches restricted to either in-distribution augmentations or sub-symbolic robustness.

Architectural Contributions

Central to Ego-Pi is its extension of the $\pi_{0.5}$ flow-matching VLA to accommodate high-dimensional, bimanual action spaces without perturbing pretrained weights. The authors identify that naive increases to VLA output projection dimensionality result in catastrophic interference, impairing policy transfer. Instead, they propose a token interleaving mechanism: bimanual control vectors are split across sequential action tokens, each covering one hand per step, maintaining architectural invariance and enabling efficient fine-tuning. This preserves both the inductive biases and the generalization capacity of the VLA foundation.

Figure 1: VLA Architecture with token interleaving supports bimanual, high-DOF control for dexterous manipulation without modifying pretrained action projection layers.

A second key contribution is the design of a robot-centric alignment technique for hand pose retargeting. Rather than relying on unreliable inverse kinematics from fingertip positions—which often leads to pose infeasibility and self-collisions in high-DOF robot hands (e.g., Tesollo)—the work proposes direct per-joint mapping from calibrated human glove keypoints to robot hand joint angles. Scaling and offset parameters are optimized per joint to minimize discrepancy in pose and functional workspace overlap, as shown in the joint mapping code and schematics in the supplementary material.

Additionally, visual cross-embodiment consistency is promoted via depth-aware, color-coded skeleton overlays on hand images, ensuring finger correspondence visibility irrespective of occlusion and camera viewpoint differences between human and robot data.

Figure 2: Depth-aware skeleton overlays facilitate finger correspondence between human and robot hands, supporting visual alignment for policy transfer.

Experimental Design

The evaluation focuses on three compositional and semantically rich manipulation tasks embodying scenario-level generalization:

1. Tomato Sorting by Color: Robot demonstrations cover single-bowl placement, while human data introduces a color-based sorting rule requiring the use of multiple bowls.
2. Skill Composition (Boxing): Robots individually learn box opening and object placing, whereas human demonstrations provide the composite skill—opening a box then inserting a block.
3. Rule-Based Packaging: Robots learn to place a box or a teddy bear, but only human data exposes the correct ordering: box first, then bear.

   Figure 3: Benchmarked tasks target high-level task semantics not available in robot data; only co-training with human data exposes correct rules for sorting, composition, and order-enforced packaging.

All experiments are conducted using a physically realized Galaxea R1 Pro equipped with either Tesollo (20-DOF) or Inspire (6-DOF) anthropomorphic robot hands and a multi-view camera system. Teleoperation leverages Manus gloves for fine-grained joint angle collection, with real-time retargeting to the robotic embodiment.

Egocentric human data is collected using the same motion capture interface, but critically, only the robot uses wrist cameras; human data is restricted to head-mounted third-person images for practical scalability.

Methods for Enhanced Cross-Embodiment Transfer

In addition to the core co-training regime, Ego-Pi integrates two auxiliary strategies to maximize complex task transfer:

• Subtask Generation: Datasets are annotated with intermediate subtask labels, enabling the VLM backbone of the VLA to first predict semantic subgoals ("think before acting"), which are subsequently realized via low-level action generation.
• Visual Skeleton Overlays: By embedding visually aligned hand skeletons into both modalities, the policy is encouraged to develop shared spatial priors over hand configurations.

Both strategies are empirically validated against standard co-training baselines on challenging skill composition and ordering tasks.

Empirical Results and Analysis

Ego-Pi's results demonstrate that co-training with human data dramatically increases semantic generalization in dexterous manipulation. Tomato sorting and packaging tasks achieve 92% and 90% success rates respectively. However, compositional behaviors in the boxing task proved unlearnable via standard co-training alone (27% success). Augmenting the policy with subtask generation enables near-perfect sequencing (93% success) by decomposing the solution into explicit, model-internal subgoals. Purely robotic data fails to unlock these high-level behaviors—robot-only models exhibit significant semantic confusion and consistently execute incorrect subtask orderings or grasp failures.

Figure 4: Co-training with human data (and subtask prediction where necessary) enables superior task performance relative to robot-only baselines; Inspire hands consistently outperform Tesollo due to morphological similarity to human hands.

The Inspire hand, whose form factor closely matches that of a human operator, consistently outperforms Tesollo in both compositional and packaging experiments, suggesting that morphological proximity facilitates cross-embodiment generalization not only in action but also in perception.

When the visual skeleton overlay is ablated, no significant quantitative improvement is registered; however, qualitative analyses suggest that overlays support more interpretable attention maps during visual grounding.

Experimentation with wrist camera ablation during test-time indicates that the policy exploits wrist perspectives heavily for precise manipulation, especially in high-precision tasks (e.g., tomato sorting).

Figure 5: Joint mapping for direct correspondence between Manus glove input and robot hand articulation, enabling kinematically valid cross-embodiment retargeting.

Implications, Limitations, and Future Directions

Ego-Pi demonstrates that a hybridization of scalable egocentric human data with pretrained foundation model architectures yields explicit high-level semantic transfer in real-world, dexterous humanoid robots. Notably, through action interleaving and robot-centric pose retargeting, Ego-Pi circumvents the architectural bottlenecks previously limiting such policy generalization to low-DOF or simulated embodiments. The work establishes that task-level semantics, especially in rule-based or skill-composed endeavors, can be acquired from human demonstration alone—even when robot data is entirely absent for those target behaviors.

The high performance achieved is nonetheless contingent on certain constraints: tasks are short-horizon, visually uncluttered, and assume all necessary low-level skills are already present in the robot. Additionally, human data serves primarily as a source of symbolic "programming" for novel task logic, not for learning new atomic skills—thus, the challenges of low-level motor primitive transfer, long-horizon and cluttered scene composition, and adaptation to fully dynamic mobile platforms remain open problems.

Looking ahead, there is significant scope for integrating Ego-Pi-style alignment into settings where human and robot morphology diverge more strongly, scaling to multi-agent and fully mobile manipulation, and exploiting unsupervised cross-embodiment priors to relax the need for tight sensorimotor correspondence.

Conclusion

Ego-Pi extends vision-language-action policy transfer to cross-embodiment dexterous manipulation, reliably enabling humanoid robots to inherit novel semantics and composite skills from egocentric human demonstrations. The action interleaving scheme, direct joint mapping, and integration of semantic subtask modeling constitute a robust recipe for advancing cross-modal, symbolically rich skill transfer. This work positions egocentric human data as an essential, scalable curriculum for enabling foundation model-powered robotics to perform open-ended, human-level tasks.

(2606.08107)