EgoDemoGen: Paired Egocentric Demo Generation

• EgoDemoGen is a framework that generates paired egocentric demonstrations by combining kinematics-based action retargeting with a video repair model to maintain action-observation alignment.
• It addresses viewpoint shifts by reprojecting scene and robot motion data using precise geometric transformations and inverse kinematics for synchronized demonstration outputs.
• Empirical results in both simulation and real-world settings show significant improvements in policy success rates, especially under novel egocentric viewpoints.

Searching arXiv for EgoDemoGen and closely related egocentric demonstration generation work. EgoDemoGen is a framework for generating paired novel egocentric demonstrations for robotic manipulation under viewpoint shifts, with the objective of improving policy robustness when the egocentric camera pose changes relative to the robot base. In the formulation reported for "EgoDemoGen: Novel Egocentric Demonstration Generation Enables Viewpoint-Robust Manipulation" (Xu et al., 26 Sep 2025), the central premise is that egocentric viewpoint shifts mechanically couple the camera and robot base, so demonstrations must be regenerated as action–observation pairs in the shifted frame rather than as appearance-only augmentations. EgoDemoGen therefore combines kinematics-based action retargeting with a generative video repair model, EgoViewTransfer, to synthesize temporally aligned novel-view videos and corresponding retargeted joint trajectories. In the reported experiments, training on a mixture of original demonstrations and EgoDemoGen-generated demonstrations improves policy success rates in both simulation and real-world settings, including gains under novel egocentric viewpoints (Xu et al., 26 Sep 2025).

1. Definition and problem setting

EgoDemoGen is a data-generation framework that creates paired demonstrations from novel egocentric viewpoints in order to train viewpoint-robust manipulation policies (Xu et al., 26 Sep 2025). Its task setting begins from a standard egocentric demonstration collected with a camera rigidly mounted to the robot “head” or base assembly. A standard egocentric viewpoint is the camera pose at data collection time, whereas a viewpoint shift is a change in the camera pose induced by robot base motion, parameterized as $v = (\Delta x, \Delta y, \Delta \theta)$ (Xu et al., 26 Sep 2025).

The framework is motivated by a specific failure mode of single-view imitation learning. Policies trained on demonstrations from one egocentric viewpoint learn view-specific correlations. Under a novel egocentric viewpoint, the same actions produce different images because the camera moves with the robot; conversely, the same images require different actions because the robot’s base pose has changed (Xu et al., 26 Sep 2025). The paper states that simply synthesizing novel views without updating actions causes a visual–action mismatch and failure. EgoDemoGen is designed to remove that mismatch by regenerating both modalities jointly.

This problem formulation places EgoDemoGen within a broader research program on egocentric data generation and control. Related systems pursue neighboring goals with different emphases: EgoEngine transforms egocentric human videos into synchronized robot observation–action pairs (Liu et al., 10 Jun 2026); EgoSim models egocentric interaction generation with an updatable 3D world state (Hao et al., 1 Apr 2026); E$^3$C controls egocentric video synthesis through 3D environmental memory and ego–exo pose conditioning (Gu et al., 25 May 2026); EgoControl conditions egocentric video prediction on full-body pose sequences (Pallotta et al., 22 Nov 2025). EgoDemoGen is narrower and more concrete: it addresses egocentric viewpoint shifts in robotic manipulation by regenerating demonstrations that remain action-consistent in the new camera frame (Xu et al., 26 Sep 2025).

2. Geometric formulation of egocentric viewpoint shifts

EgoDemoGen formalizes a novel egocentric viewpoint as an $SE(3)$ transform of the robot base, $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$, induced by the viewpoint-shift parameterization $v = (\Delta x, \Delta y, \Delta \theta)$ (Xu et al., 26 Sep 2025). The robot base frames are denoted $B_{\mathrm{src}}$ and $B_{\mathrm{nov}}$, and the camera extrinsics are written as $T_{\mathrm{cam}\leftarrow\mathrm{base}}$ for the source viewpoint and $T'_{\mathrm{cam}\leftarrow\mathrm{base}}$ for the novel viewpoint (Xu et al., 26 Sep 2025). In the experiments, wrist camera observations remain unchanged, while the head egocentric camera changes with $v$ (Xu et al., 26 Sep 2025).

The coordinate transformation between camera frames is given as

$^3$0

Projection of a 3D point into the image is modeled with the standard pinhole form

$^3$1

Novel-view scene reprojection uses a warping operator with a depth-aware z-buffer,

$^3$2

implemented as backproject-to-3D, rigid transform, then project and splat with min-depth per pixel (Xu et al., 26 Sep 2025).

These equations are not incidental. They encode the framework’s central claim that egocentric data augmentation must be grounded in the joint geometry of camera, robot base, and end-effector trajectory. A plausible implication is that EgoDemoGen should be understood less as a generic video augmentation pipeline than as a geometric retargeting-and-rendering system whose output is only valid if observation and action remain synchronized in the same transformed frame.

3. Paired demonstration generation pipeline

EgoDemoGen produces paired novel demonstrations $^3$3 from source demonstrations $^3$4 through a three-stage pipeline (Xu et al., 26 Sep 2025). The two outputs are the synthesized egocentric video $^3$5 and the retargeted joint trajectory $^3$6, aligned step-by-step in the same novel egocentric frame.

The first stage is action retargeting. EgoDemoGen computes end-effector poses through forward kinematics in the source base frame, transforms those targets into the novel base frame using the sampled base motion, and solves inverse kinematics for joint trajectories in the novel frame, yielding $^3$7 (Xu et al., 26 Sep 2025). For each arm $^3$8 and time $^3$9, the paper gives

$SE(3)$0

$SE(3)$1

$SE(3)$2

and the dual-arm retargeted sequence is

$SE(3)$3

The same procedure is summarized more generally as

$SE(3)$4

The reported implementation uses tolerance escalation, random restarts, and median smoothing in robust inverse kinematics; gripper commands are preserved by concatenating the original gripper trajectories to $SE(3)$5 (Xu et al., 26 Sep 2025).

The second stage is video synthesis through EgoViewTransfer. The source robot is removed from source frames using rendered robot masks; the robot-free scene is reprojected to the novel view and hole-filled to produce $SE(3)$6. In parallel, a robot-only video is rendered in the novel view from the retargeted actions $SE(3)$7, yielding $SE(3)$8. A finetuned generator, EgoViewTransfer, then conditions on both and synthesizes the final egocentric observation video $SE(3)$9 (Xu et al., 26 Sep 2025). The paper gives the final output relation as

$T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$0

The third stage is paired output construction. “Paired” is defined precisely: the synthesized video $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$1 and action trajectory $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$2 are aligned in the same novel egocentric frame and temporally synchronized step-by-step (Xu et al., 26 Sep 2025). This property is essential for imitation learning because it prevents the visual–action mismatch that would arise if actions were left in the source frame.

4. EgoViewTransfer and self-supervised double reprojection

EgoViewTransfer is the generative video repair model used by EgoDemoGen to synthesize egocentric observations after geometric reprojection and action retargeting (Xu et al., 26 Sep 2025). Its backbone is CogVideoX-5b-I2V video diffusion, finetuned for dual-video conditioning, with input channels expanded to 48 so that it can ingest both the reprojected scene video $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$3 and the rendered robot video $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$4 (Xu et al., 26 Sep 2025).

The conditioning design is explicitly dual-stream. The reprojected scene video provides scene geometry cues, while the rendered robot video visualizes the precise robot motion implied by $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$5 (Xu et al., 26 Sep 2025). This is intended to produce temporally consistent, action-aligned egocentric videos. The model is trained self-supervised by double reprojection. Starting from source RGB-D demonstrations, the scene is first reprojected from source to novel view and then back from novel to source, yielding an artifacted scene video $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$6; after hole filling, the repaired scene $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$7 and the rendered source robot video $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$8 serve as conditioning inputs, while the original source video $T_{\mathrm{nov}\leftarrow\mathrm{src}} \in SE(3)$9 is the target (Xu et al., 26 Sep 2025).

The training objective is the diffusion denoising loss

$v = (\Delta x, \Delta y, \Delta \theta)$0

with noisy target

$v = (\Delta x, \Delta y, \Delta \theta)$1

The paper states that EgoViewTransfer trains with the diffusion denoising objective and that additional $v = (\Delta x, \Delta y, \Delta \theta)$2 reconstruction losses or adversarial/perceptual losses were not used (Xu et al., 26 Sep 2025). Consistency is instead achieved through geometric supervision implicit in double reprojection and through conditioning on robot motion (Xu et al., 26 Sep 2025). Occlusion handling combines robot masks, z-buffer warping, hole filling, and robot rendering to resolve ambiguities introduced by viewpoint changes (Xu et al., 26 Sep 2025).

This design distinguishes EgoDemoGen from appearance-only novel-view synthesis. The generative component does not replace geometric retargeting; it repairs reprojection artifacts and reconciles the rendered robot with the scene while preserving the alignment constraints established upstream.

5. Policy training, datasets, and empirical performance

After generation, EgoDemoGen-generated demonstrations are mixed with original demonstrations for policy training (Xu et al., 26 Sep 2025). In simulation, the policy architecture is ACT (Action Chunking Transformer) with standard settings, input resolution $v = (\Delta x, \Delta y, \Delta \theta)$3 per camera, action chunk size 50, batch size 16, learning rate $v = (\Delta x, \Delta y, \Delta \theta)$4, 40k steps, temporal aggregation at test time, and training on $v = (\Delta x, \Delta y, \Delta \theta)$5 RTX 4090 (Xu et al., 26 Sep 2025). In the real world, the policy is $v = (\Delta x, \Delta y, \Delta \theta)$6 with default settings, batch size 128, input resolution $v = (\Delta x, \Delta y, \Delta \theta)$7, 40k steps, and training on $v = (\Delta x, \Delta y, \Delta \theta)$8 H20 GPUs (Xu et al., 26 Sep 2025).

The observation modalities retain wrist cameras unchanged while replacing the egocentric head camera video with $v = (\Delta x, \Delta y, \Delta \theta)$9 for novel-view demonstrations (Xu et al., 26 Sep 2025). The paper gives a generic behavioral cloning loss

$B_{\mathrm{src}}$0

or the model’s native supervised objective for ACT and $B_{\mathrm{src}}$1, trained on mixed datasets (Xu et al., 26 Sep 2025). Source data are mixed with EgoDemoGen-generated pairs at ratios $B_{\mathrm{src}}$2 (Xu et al., 26 Sep 2025).

The experiments are reported on both simulation and real-world platforms.

Setting	Baseline	EgoDemoGen
Simulation, average standard view	78.0%	80.7%
Simulation, average novel view	14.7%	30.0%
Real world, average standard view	60.0%	78.3%
Real world, average novel view	36.7%	62.5%

In simulation on RoboTwin2.0, the average standard-view success improves from 78.0% to 80.7% and the average novel-view success improves from 14.7% to 30.0% (Xu et al., 26 Sep 2025). Per-task novel-view gains are reported as Lift Pot 18% to 43%, Handover Mic 7% to 22%, and Shake Bottle 19% to 25% (Xu et al., 26 Sep 2025). In the real world, on Mobile ALOHA, average standard-view success improves from 60.0% to 78.3%, and average novel-view success improves from 36.7% to 62.5% (Xu et al., 26 Sep 2025).

The comparison to naive composition isolates the contribution of EgoViewTransfer beyond kinematics retargeting. In simulation, novel-view success increases from 15.3% to 30.0% and standard-view success from 75.7% to 80.7% (Xu et al., 26 Sep 2025). In the real world, standard-view success increases from 51.7% to 78.3% and novel-view success from 43.3% to 62.5% (Xu et al., 26 Sep 2025). Video quality metrics further show that EgoViewTransfer reduces FVD from 460.62 to 211.99 and LPIPS from 0.1175 to 0.1145 in simulation, while in real validation it improves PSNR from 19.95 to 26.93, SSIM from 0.7645 to 0.8895, reduces LPIPS from 0.2453 to 0.0870, and reduces FVD from 896.46 to 148.61 (Xu et al., 26 Sep 2025).

The mixing-ratio study reports that increasing the share of EgoDemoGen pairs improves both standard and novel views, with larger gains on novel views; a clear knee appears at $B_{\mathrm{src}}$3, while $B_{\mathrm{src}}$4 yields diminishing returns (Xu et al., 26 Sep 2025). No trade-off is observed for standard view (Xu et al., 26 Sep 2025).

6. Action–observation alignment as the core design principle

The most direct validation of EgoDemoGen’s central claim is the action-retargeting replay experiment. When original actions are executed in novel views, the average success rate is 8.3%; retargeted actions raise this to 78.3%, an absolute gain of +70.0% and approximately $B_{\mathrm{src}}$5 improvement (Xu et al., 26 Sep 2025). Per task, the paper reports Lift Pot +95% absolute, Handover Mic +45%, and Shake Bottle +70% (Xu et al., 26 Sep 2025). The conclusion stated in the paper is explicit: source-frame actions do not transfer, and retargeting is essential (Xu et al., 26 Sep 2025).

This result clarifies a common misconception in egocentric augmentation. Novel-view generation is not sufficient if the control trajectory remains expressed in the original egocentric frame. EgoDemoGen argues, and empirically supports, that robustness to viewpoint shifts requires preserving the coupling between what the camera sees and what the robot does (Xu et al., 26 Sep 2025). This contrasts with data augmentation and multi-view training approaches that modify visuals while keeping original actions fixed, thereby inducing egocentric visual–action mismatch (Xu et al., 26 Sep 2025).

The same theme appears across adjacent literature, although with different embodiments and modalities. EgoEngine identifies executable action generation as the primary factor for downstream success when converting egocentric human videos into robot data (Liu et al., 10 Jun 2026). UMIGen emphasizes that egocentric observations must match what a wrist-mounted camera would actually see, using visibility-aware optimization to preserve observation–action consistency in point-cloud space (Huang et al., 12 Nov 2025). EgoVerse’s large-scale human-to-robot study similarly finds that effective scaling depends on alignment between human data and robot learning objectives rather than mere data quantity (Punamiya et al., 8 Apr 2026). EgoDemoGen can therefore be situated within a broader shift from purely visual augmentation toward synchronized observation–action regeneration.

7. Position within egocentric demonstration generation research

EgoDemoGen belongs to a broader category of systems concerned with egocentric demonstration generation, but its specific contribution is to address viewpoint robustness through paired regeneration of observations and actions (Xu et al., 26 Sep 2025). Related works in the supplied literature illuminate adjacent formulations.

Human-to-robot demonstration conversion is exemplified by EgoEngine, which takes egocentric human manipulation videos and produces a high-fidelity robot observation video plus an executable robot action trajectory, using an object-centric digital twin and adaptive Replay $B_{\mathrm{src}}$6 MPC $B_{\mathrm{src}}$7 RL mode switching (Liu et al., 10 Jun 2026). EgoEngine’s emphasis is morphology and feasibility transfer from human to robot rather than viewpoint shift of a robot’s own egocentric camera.

Closed-loop egocentric world simulation is represented by EgoSim, which models the world as an editable 3D state and alternates observation synthesis with state updating, enabling multi-stage, long-horizon egocentric demonstrations with persistent object-state changes (Hao et al., 1 Apr 2026). EgoSim addresses structural drift and state persistence rather than robot-base viewpoint coupling per se.

Controllable egocentric video generation appears in E$B_{\mathrm{src}}$8C and EgoControl. E$B_{\mathrm{src}}$9C builds a semi-dense point-cloud 3D memory and controls ego and exo humans separately, improving camera-motion accuracy, object consistency, and control adherence on Nymeria (Gu et al., 25 May 2026). EgoControl instead uses a DiT-based diffusion model conditioned on a full-body pose representation that separates differential head motion, pelvis motion, and pelvis-relative articulated joints, achieving strong improvements in visual quality and motion alignment (Pallotta et al., 22 Nov 2025). These systems are generative and control-centric, whereas EgoDemoGen is a data-engineering pipeline for policy training.

Egocentric synthetic data generation predates these with EgoGen, which synthesizes first-person trajectories by coupling egocentric sensing and motion through collision-avoiding motion primitives and two-stage PPO (Li et al., 2024). EgoGen targets perception tasks such as mapping/localization, camera tracking, and human mesh recovery rather than manipulation robustness to viewpoint shift.

Egocentric datasets for imitation and cross-view learning include EgoMe, EgoExoLearn, and EgoVerse. EgoMe provides 7,902 paired exo–ego videos with gaze and IMU signals to study observation-to-imitation transfer (Qiu et al., 31 Jan 2025). EgoExoLearn captures asynchronous exocentric demonstrations and egocentric executions with gaze and multimodal procedural annotations (Huang et al., 2024). EgoVerse scales egocentric human manipulation data to 1,362 hours and reports that co-training with human data improves robot performance when the data are aligned with robot task definitions (Punamiya et al., 8 Apr 2026). These datasets are complementary to EgoDemoGen: they supply egocentric data diversity and cross-view structure, while EgoDemoGen focuses on viewpoint-robust regeneration of robot demonstrations.

A plausible synthesis is that EgoDemoGen occupies the “paired egocentric augmentation” niche within this ecosystem. Its distinctive mechanism is not general video synthesis, digital-twin world modeling, or human-to-robot translation, but the coordinated regeneration of action trajectories and egocentric observations under robot-base viewpoint transformations.

8. Limitations, assumptions, and practical implications

The reported evaluation range for viewpoint shifts is moderate: in simulation, $B_{\mathrm{nov}}$0 m, $B_{\mathrm{nov}}$1 m, and $B_{\mathrm{nov}}$2; in the real world, forward motion is further limited to $B_{\mathrm{nov}}$3 m by platform constraints (Xu et al., 26 Sep 2025). The paper explicitly notes that extreme changes may break reprojection and inverse-kinematics feasibility (Xu et al., 26 Sep 2025). The method also relies on accurate RGB-D alignment, masks, and URDF rendering; failures in segmentation or depth lead to artifacts (Xu et al., 26 Sep 2025). Severe occlusions or rapid motion can create large holes, and hole filling plus the generator may hallucinate (Xu et al., 26 Sep 2025). Wrist cameras are held unchanged in the reported setup, and larger wrist-view changes would require additional handling (Xu et al., 26 Sep 2025).

These limitations reflect the assumptions built into the framework. EgoDemoGen presumes that the dominant shift lies in the head egocentric camera induced by base motion, that robot geometry is known well enough to render masks and robot-only videos, and that inverse kinematics can retarget the motion into the new base frame without loss of task semantics (Xu et al., 26 Sep 2025). Within those assumptions, it provides a practical route to robustness: its end-to-end outline consists of loading source demos and calibration, sampling a novel base motion, retargeting actions through FK and IK, removing and reprojecting the scene, rendering the robot-only novel video, generating $B_{\mathrm{nov}}$4 with EgoViewTransfer, and training policies on the union of original and generated pairs (Xu et al., 26 Sep 2025).

The framework’s broader significance lies in its demonstration that egocentric robustness can be improved not by generic augmentation alone but by respecting the kinematic and geometric coupling between camera and robot. In that sense, EgoDemoGen articulates a design principle that appears increasingly central across egocentric embodied AI: when the sensor moves with the agent, valid synthetic data must preserve the joint structure of observation, embodiment, and action (Xu et al., 26 Sep 2025).