EgoLAP: Transferring Human Motion to Robot Control
- EgoLAP is a vision-language-action pre-training framework designed to transfer motion knowledge from egocentric human data to robot control, bridging the human-robot embodiment gap.
- The framework uses structured language actions and motion-level reasoning grounded in scene geometry, physics, and object affordances, improving real-world task progress by 80.1% and simulation success by 38.3%.
- EgoLAP achieves better performance compared to alternatives like FAST and OAT, with motion-level reasoning substantially improving transfer, as shown by a 35.6 percentage point increase in real-world progress.
EgoLAP is a vision-language-action pre-training framework for transferring motion knowledge from egocentric human data to robot control. Introduced in “EgoLAP: Learning from Egocentric Human Data through Language-Action Reasoning” (Zha et al., 6 Oct 2026), it addresses the human–robot embodiment gap by representing short motion segments as structured language actions and pairing them with motion-level reasoning grounded in scene geometry, physics, object affordances, and contact relations. EgoLAP jointly learns from human and robot trajectories through a shared language-based action chain-of-thought and a continuous action expert. It reports 80.1% mean real-world task progress and 38.3% mean simulation success, with motion-level reasoning substantially improving transfer relative to alternative action representations (Zha et al., 6 Oct 2026).
1. Motivation and problem formulation
Robot learning is constrained by the cost and embodiment specificity of robot demonstrations. Human egocentric video offers diverse manipulation experience, long-horizon behavior, and interaction with everyday objects, but human trajectories are not directly executable by robots. Human hands and robot end effectors differ in morphology, degrees of freedom, kinematic structure, gripper mechanics, control frequency, dynamics, camera placement, and coordinate frames.
EgoLAP distinguishes between low-level action and the underlying motion intent. A human and a robot may use different joint-space trajectories while performing an equivalent motion, such as moving an end effector toward a container, lifting an object to clear a surface, or rotating a grasped object to fit an opening. The framework therefore seeks to transfer intent rather than reproduce human trajectories verbatim.
Robot samples are represented as
where denotes RGB observations, is the task instruction, is proprioceptive state, and is a continuous action chunk. Human samples are represented as
where denotes one or two tracked hand poses in the head-camera frame. Relative motion between poses defines a human continuous action chunk.
Each sample may provide:
- a structured language-action target ;
- an optional motion rationale ;
- a continuous action chunk.
An embodiment-specific canonicalizer maps human or robot trajectories to a common discrete target:
0
The canonicalizer preserves the distinction between human, single-arm robot, and bimanual robot trajectories through metadata while producing a common language-action representation.
2. Egocentric canonicalization and language actions
EgoLAP defines an egocentric action frame
1
at the beginning of each prediction window. The reference frame is selected according to the embodiment:
- the end-effector frame for a single-arm robot with a wrist camera;
- the robot base frame for a bimanual robot;
- the base frame for a single-arm robot without a wrist camera;
- the head-mounted camera frame for egocentric human data.
For a pose 2, the action-frame coordinates are
3
Over an approximately one-second window, the framework computes
4
and
5
For bimanual data, the transformation is computed independently for the left and right trajectories while preserving arm identity.
The resulting motion tuple is
6
where 7 denotes the gripper or hand-state transition. EgoLAP serializes this tuple using a fixed language grammar:
8
Translations are rounded to integer centimeters and rotations to multiples of 9. Components whose rounded magnitude is zero are omitted, while the final hand or gripper state is retained. Translation components are ordered longitudinally, vertically, and laterally; rotations are ordered by roll, pitch, and yaw.
The vocabulary includes:
| Motion component | Positive direction | Negative direction |
|---|---|---|
| Translation 0 | move forward | move back |
| Translation 1 | move up | move down |
| Translation 2 | move left | move right |
| Roll | tilt left | tilt right |
| Pitch | tilt back | tilt forward |
| Yaw | rotate counterclockwise | rotate clockwise |
A single-arm action follows the form:
[Left|Right] arm: [motion components], [open|close] gripper
Bimanual actions use separate left- and right-arm descriptions. Human hands and fingers are represented using the corresponding “arm” and “gripper” terminology.
This representation is temporally abstracted, structured, machine-verifiable, compatible with a pretrained VLM, and intended to describe motion intent rather than a complete low-level trajectory. Its discretization introduces a trade-off: it supports cross-embodiment consistency but may be too coarse for high-speed, precision, or contact-rich manipulation.
3. Motion-level reasoning
Language actions describe what motion should occur, but not why it is physically appropriate. EgoLAP therefore introduces an optional rationale 3 that identifies:
- the hand–object or end-effector–object relation;
- visible contact or support evidence;
- relevant object affordances;
- the immediate physical effect of the motion;
- geometric constraints such as clearance, alignment, openings, rims, or obstacles.
The rationale is local and motion-focused rather than a high-level subtask plan. It connects visual state to intended physical effect:
4
For example, a rationale for manipulating a pump may state that the gripper has grasped the nozzle and that a forward or downward motion depresses the pump while avoiding bottle tipping. A rationale for moving an object toward a cabinet may explain that upward and backward motion clears the countertop and nearby obstacles.
The framework’s example for kneading dough describes the dough resting in a metal bowl, both palms contacting the top and sides, deformation against the bowl surface, and backward hand translation with wrist tilt and finger opening before the next pressing stroke. Such reasoning combines scene geometry, physics, affordances, and temporal state.
The paper distinguishes motion-level reasoning from composite reasoning formats that combine subtask information, object bounding boxes, and visual traces. Its experiments report that motion-level reasoning is more effective for EgoLAP than that composite format. The reported interpretation is that the rationale contains the physical relationship between visual state and motion, rather than merely adding auxiliary visual or task descriptors.
4. Model architecture and training
EgoLAP uses a Mixture-of-Transformers architecture consisting of a PaliGemma-2B vision-language backbone and a 300-million-parameter continuous action expert initialized from scratch. The model accepts up to three RGB views: one base-camera view and two wrist-camera views. Missing cameras are zero-padded and masked.
Normalized proprioceptive state is discretized into 256 uniform bins and serialized as text. For Cartesian policies, the state includes end-effector position, continuous 6D rotation, and gripper state. The VLM and action expert use separate parameters but share masked-attention operations.
For batch size 5, camera slots 6, and a bimanual system, the principal tensor dimensions are:
- images: 7;
- state: 8;
- action target: 9.
The VLM uses 18 transformer blocks with width 2048, FFN width 16,384, eight query heads, and one key–value head. The action expert also uses 18 transformer blocks, with width 1024, FFN width 4096, eight query heads, and one key–value head. Its input is a noisy 0 action chunk together with a sinusoidal flow-time embedding, and its output is a velocity field of the same action-chunk shape.
The attention structure separates training-time language supervision from control-time action prediction. Images, instructions, and state form a bidirectional prefix. Rationale and language-action tokens attend causally to this prefix and to preceding textual tokens. The action expert attends to its noisy action tokens and the observation–instruction–state prefix, but not to rationale or language-action tokens. Consequently, textual reasoning need not be generated during inference.
EgoLAP also uses knowledge insulation: gradients from the continuous action expert are stopped at the VLM representation boundary. This prevents the randomly initialized action expert from degrading the pretrained language representation.
Flow-matching objective
For an action chunk 1, Gaussian noise 2, and
3
the interpolated action is
4
with target velocity
5
The flow-matching loss is
6
where 7 is the VLM prefix representation, 8 is the action expert, and 9 denotes stop-gradient.
The overall training objective is
0
with 1 and 2.
Stochastic reasoning supervision
Rationale supervision is stochastic. Independent Bernoulli variables determine whether a rationale is inserted as a teacher-forced prefix and whether rationale tokens receive loss:
3
with
4
Language-action tokens are supervised in every case. Among samples with rationale labels, half directly predict the action without a rationale, one quarter condition on a rationale without supervising its tokens, and one quarter both condition on and supervise rationale tokens. This prevents the model from depending exclusively on generated rationales while retaining their training benefit.
5. Data sources and pseudo-labeling
The training mixture contains approximately 10,000 hours of raw data:
- 30% ABC robot data from the YAM platform;
- 25% egocentric human data from EgoVerse, including MECKA, Scale, and Aria;
- 25% additional bimanual robot data from AgiBot, Galaxea R1 Lite, and MolmoAct2 YAM;
- 20% single-arm OXE robot datasets, with DROID contributing approximately 12%.
Only DROID and MECKA contain motion-level reasoning labels. Together they account for 32% of each training batch. Other data provide language-action and continuous-action supervision without rationale labels.
Reasoning labels are generated offline using Gemini-3.5-Flash for DROID and Gemini-3.7-Flash for MECKA, with temperature 5. The annotator receives task instructions, temporally ordered image observations, action history, current-state information, and the ground-truth language action. Providing the action target was found empirically to produce more grounded explanations than supplying images alone.
For DROID, adaptive segments use a nominal stride of 15 filtered steps, approximately one second, and may terminate early following a persistent gripper-state change or a large translational-direction change. The annotator receives primary and wrist-camera images, gripper metadata, the language action, and up to five seconds of compact action history.
For MECKA, subsequences from the same recording are concatenated in order, adaptive anchors are selected 15–30 frames apart at 30 Hz, and the annotator receives the head-mounted image, instruction, stitched subtask context, bimanual language action, and action history. Hand-specific language is used for contacts, support, fingers, and release.
The annotation process produces trajectory summaries, subtask plans, completed/current/next subtasks, observation–action grounding, visual relations, motion effects, and alternative-motion diagnostics. EgoLAP uses the main motion-level rationale as training supervision. These are therefore offline vision-language-model pseudo-labels rather than dense manually authored annotations.
6. Inference, reinforcement learning, evaluation, and limitations
Inference
At inference time, RGB observations, task instruction, and discretized state are encoded. The action expert starts from Gaussian noise at 6, integrates the learned velocity field toward 7 using ten Euler steps, and produces a 8 action chunk for continuous execution.
Rationales and language actions are not generated during standard control-time inference. They provide training-time supervision, while the continuous action expert directly consumes the shared visual-language representation. Language actions can nevertheless be generated for reinforcement-learning post-training.
Reinforcement learning over language actions
Because language actions are deterministic and parseable, EgoLAP can assign a verifiable motion-matching reward without a learned reward model. For arm 9, translation, rotation, and gripper-state discrepancies are combined into
0
where
1
The reward is 2. A 2 cm translation error, a 3 orientation error, and one incorrect gripper state each contribute approximately one cost unit. The RL study uses GRPO with 32 observations and 64 completions per observation. It is simulation-only and optimizes motion-matching reward rather than task success.
Generating rationales during RL produced a smaller improvement than generating language actions alone: the reported improvements were 12.7% with rationale generation and 19.4% without it. The paper attributes this to the rationales being out of distribution because ABC rationales were not present during pre-training.
Experimental platforms
Real-world evaluation uses a customized bimanual YAM robot with two table-mounted arms separated by approximately 0.8 m, a RealSense D435 base camera, and two RealSense D405 wrist cameras. Five behaviors are evaluated:
- Pick Place (Simple);
- Fold Pants;
- Handover and Place;
- Wipe Object;
- Pick Place (Complex).
There are 10 trials per task except Pick Place (Complex), which has 12. Each task is decomposed into stages. If task 4 has 5 stages, normalized progress is
6
Simulation evaluation uses 14 ABC tasks, comprising picking, placement, color selection, and cloth folding. Each model is evaluated for 100 episodes per task, for 1,400 episodes in total, without simulation-data training.
Quantitative results
EgoLAP reports 80.1% mean real-world task progress and 38.3% mean simulation success. The real-world progress metric measures the fraction of ordered task stages completed on average, rather than the fraction of trials completed perfectly.
Without reasoning supervision, the reported results are:
| Representation | Simulation success | Real-world progress |
|---|---|---|
| Language actions | 16.4% | 44.5% |
| FAST | 1.6% | 35.2% |
| OAT | 2.7% | 18.0% |
Adding motion-level reasoning increases language-action performance from 44.5% to 80.1% in real-world progress and from 16.4% to 38.3% in simulation success. The corresponding gains are 35.6 and 21.9 percentage points. FAST and OAT obtain substantially smaller gains, and FAST decreases from 35.2% to 26.1% in real-world progress after reasoning is added.
Adding human data to a robot-only language-action policy improves simulation success from 7.2% to 16.4%. With reasoning, robot-only performance is 18.4%, whereas joint robot–human training reaches 38.3%. The paper therefore reports that human data improve both language supervision and continuous action-expert training.
EgoLAP is also compared with a composite reasoning format combining subtask information, object bounding boxes, visual traces, and language actions. The composite format reaches 5.3% simulation success alone and 14.4% when stacked before motion-level reasoning, compared with 38.3% for motion-level reasoning alone. The reported interpretation is that physically grounded motion reasoning is more useful than simply adding auxiliary tokens.
Cross-embodiment retrieval further reports improvements in shared skill structure. Robot-to-human three-skill retrieval at rank one increases from 51% to 76%, human-to-robot three-skill retrieval increases from 42% to 63%, and robot-to-human two-skill retrieval increases from 69% to 92%. The embeddings remain linearly separable by embodiment, indicating that the shared representation does not eliminate human–robot identity.
Alternative representations and limitations
The principal alternatives are EgoFAST, which uses FAST action tokens; EgoOAT, which uses OAT action tokens; LAP, which uses language actions without motion-level reasoning and human data; and ABC-VLA, a released checkpoint trained directly on the ABC YAM platform. FAST and OAT retain the same continuous action expert while replacing the discrete language-action target.
EgoLAP has several stated limitations. Its one-second action horizon can omit high-frequency contact events, rapid corrections, force-sensitive behavior, and precision requirements. Translation and rotation quantization may be too coarse for delicate insertion or tight alignment. The evaluation focuses on bimanual tabletop manipulation and does not establish performance for mobile manipulation, locomotion, dexterous multi-finger control, large-scale navigation, or force-controlled manipulation.
Motion rationales are pseudo-labeled by vision-LLMs and may be incomplete, imprecise, or visually overconfident. Human hand tracking introduces noise and kinematic mismatch. The reinforcement-learning study optimizes a parsed motion-matching proxy rather than task-level reward. Finally, contact-rich manipulation may require adaptive temporal resolution and fine corrections beyond the current coarse language-action representation.
EgoLAP’s central contribution is consequently a shared language–physics–action interface for human and robot data. It treats egocentric human trajectories neither as direct robot demonstrations nor solely as visual pre-training, but as supervision for a representation of motion intent that can be grounded in scene geometry, physical effects, and affordances. The framework’s empirical results support the use of structured language actions, motion-level reasoning, joint human–robot training, and continuous action experts for cross-embodiment robot learning (Zha et al., 6 Oct 2026).