Papers
Topics
Authors
Recent
Search
2000 character limit reached

EgoLAP: Transferring Human Motion to Robot Control

Updated 10 October 2026
  • EgoLAP is a vision-language-action pre-training framework designed to transfer motion knowledge from egocentric human data to robot control, bridging the human-robot embodiment gap.
  • The framework uses structured language actions and motion-level reasoning grounded in scene geometry, physics, and object affordances, improving real-world task progress by 80.1% and simulation success by 38.3%.
  • EgoLAP achieves better performance compared to alternatives like FAST and OAT, with motion-level reasoning substantially improving transfer, as shown by a 35.6 percentage point increase in real-world progress.

EgoLAP is a vision-language-action pre-training framework for transferring motion knowledge from egocentric human data to robot control. Introduced in “EgoLAP: Learning from Egocentric Human Data through Language-Action Reasoning” (Zha et al., 6 Oct 2026), it addresses the human–robot embodiment gap by representing short motion segments as structured language actions and pairing them with motion-level reasoning grounded in scene geometry, physics, object affordances, and contact relations. EgoLAP jointly learns from human and robot trajectories through a shared language-based action chain-of-thought and a continuous action expert. It reports 80.1% mean real-world task progress and 38.3% mean simulation success, with motion-level reasoning substantially improving transfer relative to alternative action representations (Zha et al., 6 Oct 2026).

1. Motivation and problem formulation

Robot learning is constrained by the cost and embodiment specificity of robot demonstrations. Human egocentric video offers diverse manipulation experience, long-horizon behavior, and interaction with everyday objects, but human trajectories are not directly executable by robots. Human hands and robot end effectors differ in morphology, degrees of freedom, kinematic structure, gripper mechanics, control frequency, dynamics, camera placement, and coordinate frames.

EgoLAP distinguishes between low-level action and the underlying motion intent. A human and a robot may use different joint-space trajectories while performing an equivalent motion, such as moving an end effector toward a container, lifting an object to clear a surface, or rotating a grasped object to fit an opening. The framework therefore seeks to transfer intent rather than reproduce human trajectories verbatim.

Robot samples are represented as

(ot,ℓ,st,at:t+K)∈DR,(o_t,\ell,s_t,a_{t:t+K})\in\mathcal{D}_R,

where oto_t denotes RGB observations, ℓ\ell is the task instruction, sts_t is proprioceptive state, and at:t+Ka_{t:t+K} is a continuous action chunk. Human samples are represented as

(ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,

where qq denotes one or two tracked hand poses in the head-camera frame. Relative motion between poses defines a human continuous action chunk.

Each sample may provide:

  1. a structured language-action target mtm_t;
  2. an optional motion rationale rtr_t;
  3. a continuous action chunk.

An embodiment-specific canonicalizer GeG_e maps human or robot trajectories to a common discrete target:

oto_t0

The canonicalizer preserves the distinction between human, single-arm robot, and bimanual robot trajectories through metadata while producing a common language-action representation.

2. Egocentric canonicalization and language actions

EgoLAP defines an egocentric action frame

oto_t1

at the beginning of each prediction window. The reference frame is selected according to the embodiment:

  • the end-effector frame for a single-arm robot with a wrist camera;
  • the robot base frame for a bimanual robot;
  • the base frame for a single-arm robot without a wrist camera;
  • the head-mounted camera frame for egocentric human data.

For a pose oto_t2, the action-frame coordinates are

oto_t3

Over an approximately one-second window, the framework computes

oto_t4

and

oto_t5

For bimanual data, the transformation is computed independently for the left and right trajectories while preserving arm identity.

The resulting motion tuple is

oto_t6

where oto_t7 denotes the gripper or hand-state transition. EgoLAP serializes this tuple using a fixed language grammar:

oto_t8

Translations are rounded to integer centimeters and rotations to multiples of oto_t9. Components whose rounded magnitude is zero are omitted, while the final hand or gripper state is retained. Translation components are ordered longitudinally, vertically, and laterally; rotations are ordered by roll, pitch, and yaw.

The vocabulary includes:

Motion component Positive direction Negative direction
Translation ℓ\ell0 move forward move back
Translation ℓ\ell1 move up move down
Translation ℓ\ell2 move left move right
Roll tilt left tilt right
Pitch tilt back tilt forward
Yaw rotate counterclockwise rotate clockwise

A single-arm action follows the form:

[Left|Right] arm: [motion components], [open|close] gripper

Bimanual actions use separate left- and right-arm descriptions. Human hands and fingers are represented using the corresponding “arm” and “gripper” terminology.

This representation is temporally abstracted, structured, machine-verifiable, compatible with a pretrained VLM, and intended to describe motion intent rather than a complete low-level trajectory. Its discretization introduces a trade-off: it supports cross-embodiment consistency but may be too coarse for high-speed, precision, or contact-rich manipulation.

3. Motion-level reasoning

Language actions describe what motion should occur, but not why it is physically appropriate. EgoLAP therefore introduces an optional rationale ℓ\ell3 that identifies:

  1. the hand–object or end-effector–object relation;
  2. visible contact or support evidence;
  3. relevant object affordances;
  4. the immediate physical effect of the motion;
  5. geometric constraints such as clearance, alignment, openings, rims, or obstacles.

The rationale is local and motion-focused rather than a high-level subtask plan. It connects visual state to intended physical effect:

ℓ\ell4

For example, a rationale for manipulating a pump may state that the gripper has grasped the nozzle and that a forward or downward motion depresses the pump while avoiding bottle tipping. A rationale for moving an object toward a cabinet may explain that upward and backward motion clears the countertop and nearby obstacles.

The framework’s example for kneading dough describes the dough resting in a metal bowl, both palms contacting the top and sides, deformation against the bowl surface, and backward hand translation with wrist tilt and finger opening before the next pressing stroke. Such reasoning combines scene geometry, physics, affordances, and temporal state.

The paper distinguishes motion-level reasoning from composite reasoning formats that combine subtask information, object bounding boxes, and visual traces. Its experiments report that motion-level reasoning is more effective for EgoLAP than that composite format. The reported interpretation is that the rationale contains the physical relationship between visual state and motion, rather than merely adding auxiliary visual or task descriptors.

4. Model architecture and training

EgoLAP uses a Mixture-of-Transformers architecture consisting of a PaliGemma-2B vision-language backbone and a 300-million-parameter continuous action expert initialized from scratch. The model accepts up to three RGB views: one base-camera view and two wrist-camera views. Missing cameras are zero-padded and masked.

Normalized proprioceptive state is discretized into 256 uniform bins and serialized as text. For Cartesian policies, the state includes end-effector position, continuous 6D rotation, and gripper state. The VLM and action expert use separate parameters but share masked-attention operations.

For batch size ℓ\ell5, camera slots ℓ\ell6, and a bimanual system, the principal tensor dimensions are:

  • images: ℓ\ell7;
  • state: ℓ\ell8;
  • action target: ℓ\ell9.

The VLM uses 18 transformer blocks with width 2048, FFN width 16,384, eight query heads, and one key–value head. The action expert also uses 18 transformer blocks, with width 1024, FFN width 4096, eight query heads, and one key–value head. Its input is a noisy sts_t0 action chunk together with a sinusoidal flow-time embedding, and its output is a velocity field of the same action-chunk shape.

The attention structure separates training-time language supervision from control-time action prediction. Images, instructions, and state form a bidirectional prefix. Rationale and language-action tokens attend causally to this prefix and to preceding textual tokens. The action expert attends to its noisy action tokens and the observation–instruction–state prefix, but not to rationale or language-action tokens. Consequently, textual reasoning need not be generated during inference.

EgoLAP also uses knowledge insulation: gradients from the continuous action expert are stopped at the VLM representation boundary. This prevents the randomly initialized action expert from degrading the pretrained language representation.

Flow-matching objective

For an action chunk sts_t1, Gaussian noise sts_t2, and

sts_t3

the interpolated action is

sts_t4

with target velocity

sts_t5

The flow-matching loss is

sts_t6

where sts_t7 is the VLM prefix representation, sts_t8 is the action expert, and sts_t9 denotes stop-gradient.

The overall training objective is

at:t+Ka_{t:t+K}0

with at:t+Ka_{t:t+K}1 and at:t+Ka_{t:t+K}2.

Stochastic reasoning supervision

Rationale supervision is stochastic. Independent Bernoulli variables determine whether a rationale is inserted as a teacher-forced prefix and whether rationale tokens receive loss:

at:t+Ka_{t:t+K}3

with

at:t+Ka_{t:t+K}4

Language-action tokens are supervised in every case. Among samples with rationale labels, half directly predict the action without a rationale, one quarter condition on a rationale without supervising its tokens, and one quarter both condition on and supervise rationale tokens. This prevents the model from depending exclusively on generated rationales while retaining their training benefit.

5. Data sources and pseudo-labeling

The training mixture contains approximately 10,000 hours of raw data:

  • 30% ABC robot data from the YAM platform;
  • 25% egocentric human data from EgoVerse, including MECKA, Scale, and Aria;
  • 25% additional bimanual robot data from AgiBot, Galaxea R1 Lite, and MolmoAct2 YAM;
  • 20% single-arm OXE robot datasets, with DROID contributing approximately 12%.

Only DROID and MECKA contain motion-level reasoning labels. Together they account for 32% of each training batch. Other data provide language-action and continuous-action supervision without rationale labels.

Reasoning labels are generated offline using Gemini-3.5-Flash for DROID and Gemini-3.7-Flash for MECKA, with temperature at:t+Ka_{t:t+K}5. The annotator receives task instructions, temporally ordered image observations, action history, current-state information, and the ground-truth language action. Providing the action target was found empirically to produce more grounded explanations than supplying images alone.

For DROID, adaptive segments use a nominal stride of 15 filtered steps, approximately one second, and may terminate early following a persistent gripper-state change or a large translational-direction change. The annotator receives primary and wrist-camera images, gripper metadata, the language action, and up to five seconds of compact action history.

For MECKA, subsequences from the same recording are concatenated in order, adaptive anchors are selected 15–30 frames apart at 30 Hz, and the annotator receives the head-mounted image, instruction, stitched subtask context, bimanual language action, and action history. Hand-specific language is used for contacts, support, fingers, and release.

The annotation process produces trajectory summaries, subtask plans, completed/current/next subtasks, observation–action grounding, visual relations, motion effects, and alternative-motion diagnostics. EgoLAP uses the main motion-level rationale as training supervision. These are therefore offline vision-language-model pseudo-labels rather than dense manually authored annotations.

6. Inference, reinforcement learning, evaluation, and limitations

Inference

At inference time, RGB observations, task instruction, and discretized state are encoded. The action expert starts from Gaussian noise at at:t+Ka_{t:t+K}6, integrates the learned velocity field toward at:t+Ka_{t:t+K}7 using ten Euler steps, and produces a at:t+Ka_{t:t+K}8 action chunk for continuous execution.

Rationales and language actions are not generated during standard control-time inference. They provide training-time supervision, while the continuous action expert directly consumes the shared visual-language representation. Language actions can nevertheless be generated for reinforcement-learning post-training.

Reinforcement learning over language actions

Because language actions are deterministic and parseable, EgoLAP can assign a verifiable motion-matching reward without a learned reward model. For arm at:t+Ka_{t:t+K}9, translation, rotation, and gripper-state discrepancies are combined into

(ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,0

where

(ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,1

The reward is (ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,2. A 2 cm translation error, a (ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,3 orientation error, and one incorrect gripper state each contribute approximately one cost unit. The RL study uses GRPO with 32 observations and 64 completions per observation. It is simulation-only and optimizes motion-matching reward rather than task success.

Generating rationales during RL produced a smaller improvement than generating language actions alone: the reported improvements were 12.7% with rationale generation and 19.4% without it. The paper attributes this to the rationales being out of distribution because ABC rationales were not present during pre-training.

Experimental platforms

Real-world evaluation uses a customized bimanual YAM robot with two table-mounted arms separated by approximately 0.8 m, a RealSense D435 base camera, and two RealSense D405 wrist cameras. Five behaviors are evaluated:

  1. Pick Place (Simple);
  2. Fold Pants;
  3. Handover and Place;
  4. Wipe Object;
  5. Pick Place (Complex).

There are 10 trials per task except Pick Place (Complex), which has 12. Each task is decomposed into stages. If task (ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,4 has (ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,5 stages, normalized progress is

(ot,ℓ,qt:t+K)∈DH,(o_t,\ell,q_{t:t+K})\in\mathcal{D}_H,6

Simulation evaluation uses 14 ABC tasks, comprising picking, placement, color selection, and cloth folding. Each model is evaluated for 100 episodes per task, for 1,400 episodes in total, without simulation-data training.

Quantitative results

EgoLAP reports 80.1% mean real-world task progress and 38.3% mean simulation success. The real-world progress metric measures the fraction of ordered task stages completed on average, rather than the fraction of trials completed perfectly.

Without reasoning supervision, the reported results are:

Representation Simulation success Real-world progress
Language actions 16.4% 44.5%
FAST 1.6% 35.2%
OAT 2.7% 18.0%

Adding motion-level reasoning increases language-action performance from 44.5% to 80.1% in real-world progress and from 16.4% to 38.3% in simulation success. The corresponding gains are 35.6 and 21.9 percentage points. FAST and OAT obtain substantially smaller gains, and FAST decreases from 35.2% to 26.1% in real-world progress after reasoning is added.

Adding human data to a robot-only language-action policy improves simulation success from 7.2% to 16.4%. With reasoning, robot-only performance is 18.4%, whereas joint robot–human training reaches 38.3%. The paper therefore reports that human data improve both language supervision and continuous action-expert training.

EgoLAP is also compared with a composite reasoning format combining subtask information, object bounding boxes, visual traces, and language actions. The composite format reaches 5.3% simulation success alone and 14.4% when stacked before motion-level reasoning, compared with 38.3% for motion-level reasoning alone. The reported interpretation is that physically grounded motion reasoning is more useful than simply adding auxiliary tokens.

Cross-embodiment retrieval further reports improvements in shared skill structure. Robot-to-human three-skill retrieval at rank one increases from 51% to 76%, human-to-robot three-skill retrieval increases from 42% to 63%, and robot-to-human two-skill retrieval increases from 69% to 92%. The embeddings remain linearly separable by embodiment, indicating that the shared representation does not eliminate human–robot identity.

Alternative representations and limitations

The principal alternatives are EgoFAST, which uses FAST action tokens; EgoOAT, which uses OAT action tokens; LAP, which uses language actions without motion-level reasoning and human data; and ABC-VLA, a released checkpoint trained directly on the ABC YAM platform. FAST and OAT retain the same continuous action expert while replacing the discrete language-action target.

EgoLAP has several stated limitations. Its one-second action horizon can omit high-frequency contact events, rapid corrections, force-sensitive behavior, and precision requirements. Translation and rotation quantization may be too coarse for delicate insertion or tight alignment. The evaluation focuses on bimanual tabletop manipulation and does not establish performance for mobile manipulation, locomotion, dexterous multi-finger control, large-scale navigation, or force-controlled manipulation.

Motion rationales are pseudo-labeled by vision-LLMs and may be incomplete, imprecise, or visually overconfident. Human hand tracking introduces noise and kinematic mismatch. The reinforcement-learning study optimizes a parsed motion-matching proxy rather than task-level reward. Finally, contact-rich manipulation may require adaptive temporal resolution and fine corrections beyond the current coarse language-action representation.

EgoLAP’s central contribution is consequently a shared language–physics–action interface for human and robot data. It treats egocentric human trajectories neither as direct robot demonstrations nor solely as visual pre-training, but as supervision for a representation of motion intent that can be grounded in scene geometry, physical effects, and affordances. The framework’s empirical results support the use of structured language actions, motion-level reasoning, joint human–robot training, and continuous action experts for cross-embodiment robot learning (Zha et al., 6 Oct 2026).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to EgoLAP.