Egocentric Motion Generation: A Survey

• Egocentric motion generation is a family of first-person conditioned generative problems that synthesize motion, video, and camera trajectories from wearable inputs.
• It leverages multimodal signals—including egocentric RGB, head trajectories, wrist cues, and text prompts—to tackle challenges like partial observability, self-occlusion, and strict causality.
• Modern approaches employ diffusion models, flow matching, autoregressive frameworks, and reinforcement learning to achieve smooth, context-aware, and geometrically consistent motion generation.

Egocentric motion generation denotes a family of first-person conditioned generative problems in which motion, video, or camera trajectories are synthesized from signals available in the wearer’s frame, including egocentric RGB observations, head trajectories, wrist cues, text prompts, and structured pose controls. Across recent work, the central technical difficulty is the simultaneous requirement of scene grounding, partial observability, self-occlusion handling, head-camera/body coupling, and, in online settings, strict causality. The resulting literature spans full-body 3D motion generation and reconstruction, pose-controllable egocentric video synthesis, active camera-motion generation, hand-object interaction generation, and egocentric 6DoF object-motion prediction (Patel et al., 2 Aug 2025, Hwang et al., 13 May 2026, Ge et al., 7 Jun 2026).

1. Task family and problem formulations

Recent work does not treat egocentric motion generation as a single narrowly defined task. Instead, it includes at least three recurrent formulations for full-body human motion: reconstruction from egocentric video and device trajectory, forecasting from past egocentric observations, and generation from a single egocentric image or from egocentric observations plus language. UniEgoMotion makes this distinction explicit through the conditionals $p(X_{1:N}\mid I_1)$, $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$, and $p(X_{1:N}\mid I_{1:N},T_{1:N})$, corresponding respectively to generation, forecasting, and reconstruction (Patel et al., 2 Aug 2025).

A second line of work conditions on language and scene perception rather than only passive visual history. EgoMotion formulates Egocentric Vision-Language motion generation as $p(\mathcal{X}\mid I_0,\mathcal{L},X_0)$, where an egocentric RGB image, a natural-language instruction, and an initial pose specify a future 3D motion sequence. EgoPriMo similarly takes egocentric observations and a text prompt and outputs an SMPL-based full-body motion sequence, while treating language as a high-level control signal rather than a complete motion specification (Hou et al., 21 Apr 2026, Ge et al., 7 Jun 2026).

A third formulation moves from motion generation to egocentric video generation under motion control. EgoControl predicts future first-person frames from observed context frames and a target 3D full-body pose sequence. EgoTwin generalizes this further to joint generation of synchronized egocentric video and human motion from an initial pose, an initial egocentric observation, and a text prompt, explicitly modeling the closed-loop coupling between body motion and viewpoint change (Pallotta et al., 22 Nov 2025, Xiu et al., 18 Aug 2025).

Other subfields specialize the target variable. LIME treats camera motion itself as the action, mapping a current RGB observation and a free-form intent to an observation-gain description and a relative target camera pose in $SE(3)$. EgoFlow predicts object trajectories, representing each future state as $$\mathbf{x}_t=[\mathbf{p}_t;\mathbf{r}_t]\in\mathbb{R}^9$$. Earlier work in sports planning used a more restricted egocentric trajectory formulation: “Egocentric Basketball Motion Planning from a Single First-Person Image” generates a 12D camera-configuration trajectory from one initial image (Sun et al., 2 Jul 2026, Saroha et al., 1 Apr 2026, Bertasius et al., 2018).

A recurring misconception is to equate egocentric motion generation with camera control alone. The literature shows a broader picture: some methods generate body motion from first-person evidence, some generate egocentric video from future body pose, some generate camera motion as an active-perception primitive, and some generate object or hand trajectories conditioned by egocentric context. This suggests that “egocentric motion generation” is best understood as a family of first-person-conditioned generative problems rather than a single benchmark.

2. Representations and conditioning signals

A dominant design choice is the move from pelvis-centric to head-centric representations. UniEgoMotion transforms SMPL-X motion into a canonical head-aligned frame so that motion is represented by the canonical head trajectory together with pose relative to that frame. EgoTwin makes the same motivation explicit and replaces root-centric motion with the head-centric tuple $(h^r,\dot{h}^r,h^p,\dot{h}^p,j^p,j^v,j^r)$, where head rotation and translation are represented both absolutely and relatively, and other joints are expressed in head space. The stated purpose in both cases is to align motion representation with wearable devices and with camera-relevant dynamics (Patel et al., 2 Aug 2025, Xiu et al., 18 Aug 2025).

Pose-controlled video work uses more explicit decompositions of camera and body dynamics. EgoControl represents future control as $$\mathbf{P}=[\mathbf{\Delta h},\mathbf{\Delta r},\mathbf{J}]\in\mathbb{R}^{M\times23\times6}$$, where $\mathbf{\Delta h}$ encodes relative head motion, $\mathbf{\Delta r}$ encodes pelvis motion, and $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$0 encodes pelvis-relative articulated joints. PlayerOne similarly separates body and feet, hands, and head, and further converts head rotation into camera-control features via Rodrigues’ formula and Plücker rays. E$p(X_{n+1:N}\mid I_{1:n},T_{1:n})$1C uses 3D body joints and 6DoF wrist poses for ego control, with an ego motion encoder that converts these signals into persistent cross-attention tokens so that body motion remains available even when hands or arms leave the image (Pallotta et al., 22 Nov 2025, Tu et al., 11 Jun 2025, Gu et al., 25 May 2026).

Sparse and unreliable observations are another recurring theme. HaMoS conditions on a head trajectory in world coordinates and on sparse 6D hand or wrist observations with binary visibility, replacing invisible hands with a learnable null embedding rather than zeros. EgoForce uses a kinematic control signal $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$2, together with a binary visibility mask over motion dimensions; observed coordinates are pinned while unobserved coordinates remain stochastic latent variables. These methods treat missingness itself as informative (Cho et al., 22 Dec 2025, Hwang et al., 13 May 2026).

Hand-centric egocentric generation introduces yet another representational layer. MEgoHand represents a MANO hand state as $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$3, combining finger rotations, hand shape, wrist rotation, and wrist translation, and conditions generation on RGB, text, metric depth, and the initial hand pose. Hand2World renders projected 3D MANO hand meshes into silhouette-and-wireframe control videos, while “Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints” uses sparse 3D hand joints as embodiment-agnostic control signals with explicit semantic identity and depth-aware occlusion handling (Zhou et al., 22 May 2025, Wang et al., 10 Feb 2026, Zhang et al., 12 Mar 2026).

This body of work suggests a common principle: egocentric generation benefits when conditioning variables preserve the geometry that first-person sensing actually observes or implies. Head-centric motion, wrist-aware control, metric depth, Plücker-ray camera features, and visibility masks all serve that role.

3. Model families and training objectives

Diffusion-based generative modeling is the dominant backbone for full-body egocentric motion. UniEgoMotion uses a transformer-based conditional diffusion model in which noisy motion tokens are denoised under trajectory and image conditioning. HMD$p(X_{n+1:N}\mid I_{1:n},T_{1:n})$4 also uses a conditional DDPM-style diffusion model with a Transformer backbone, then adapts fixed-window denoising to streaming use through autoregressive inpainting over overlapping windows (Patel et al., 2 Aug 2025, Guzov et al., 2024).

Several works modify diffusion specifically for causal or multimodal egocentric settings. EgoForce introduces diffusion forcing with a temporally asymmetric noise schedule, assigning separate diffusion indices to frames in a sliding buffer and setting $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$5, with past and current frames denoised more than the future. The method reuses partially denoised latent state as a warm start, shifts the temporal window forward, injects new observations causally, and performs only a small refinement step. EgoTwin uses a triple-branch diffusion transformer with separate text, video, and motion streams, and trains them with asynchronous diffusion timesteps so that video and motion can denoise at different rates while still interacting (Hwang et al., 13 May 2026, Xiu et al., 18 Aug 2025).

Flow matching has become a second major family. EgoPriMo trains a Triple-stream DiT with flow matching on interpolants $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$6 and $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$7, while using task-conditioning masks to unify reconstruction, generation, and forecasting within one checkpoint. EgoMotion uses a two-stage hierarchy: a VLM first predicts structured motion primitives with an RVQ-VAE tokenizer, then a latent diffusion or latent flow-matching generator synthesizes final motion from frozen multimodal representations. EgoFlow applies flow matching to 6DoF object trajectories and refines generated motion at inference with differentiable penalties for collision avoidance and smoothness (Ge et al., 7 Jun 2026, Hou et al., 21 Apr 2026, Saroha et al., 1 Apr 2026).

Autoregressive and reinforcement-learning formulations remain important where strict causality or closed-loop interaction is central. EgoReAct, described as the first autoregressive framework for generating 3D-aligned human reaction motions from egocentric video streams in real time, compresses reaction motion with a Vector Quantised-Variational AutoEncoder and predicts discrete motion structure with a Generative Pre-trained Transformer, while incorporating metric depth and head dynamics for spatial grounding. EgoGen instead learns a PPO policy over the latent action space of a conditional VAE motion-primitive model, allowing a virtual human to perceive through an egocentric sensing proxy, choose a motion primitive, execute it, and update what it sees. “Moving by Looking” similarly decomposes the problem into a motion prior $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$8 and a Q-network $p(X_{n+1:N}\mid I_{1:n},T_{1:n})$9, so that $p(X_{1:N}\mid I_{1:N},T_{1:N})$0 maps egocentric observations to motion chunks through high-level control (Zhang et al., 28 Dec 2025, Li et al., 2024, Diomataris et al., 23 Sep 2025).

The architectural diversity reflects a genuine modeling split. Diffusion and flow matching dominate where multimodal conditioning and stochasticity are primary; autoregressive and RL-driven policies dominate where causal rollout, low latency, or active control are the central constraints.

4. Data construction, benchmarks, and supervision

A defining problem in this area is the scarcity of paired egocentric observations and 3D motion. Many papers devote substantial effort to dataset construction. UniEgoMotion introduces EE4D-Motion from EgoExo4D with pseudo-ground-truth SMPL-X motion. The paper reports an initial scale of about 208 hours of synchronized egocentric video-motion pairs, reduced after filtering to about 110 hours of high-quality 3D-accurate motion data, and uses 8-second clips at 10 fps with about 143K training samples (Patel et al., 2 Aug 2025).

Nymeria is a recurring anchor dataset for egocentric human motion. HMD$p(X_{1:N}\mid I_{1:N},T_{1:N})$1 trains and evaluates on Nymeria at approximately 300 hours, 1200 sequences, 264 participants, and 50 locations, using 1040 filtered sequences. EgoTwin clips Nymeria into 5-second segments and reports about 170K samples after filtering. EgoPriMo uses Nymeria together with HumanML3D and EgoExo4D in a final training mixture of 70% Nymeria, 15% HumanML3D, and 15% EgoExo4D, with clips of $p(X_{1:N}\mid I_{1:N},T_{1:N})$2 frames (Guzov et al., 2024, Xiu et al., 18 Aug 2025, Ge et al., 7 Jun 2026).

Several works construct new supervision by augmentation or automated annotation rather than direct capture. EgoReAct introduces the Human Reaction Dataset to address data scarcity and spatial misalignment, explicitly criticizing datasets in which dynamically moving motions are paired with fixed-camera videos. HaMoS trains primarily on AMASS but adds a hand-visibility augmentation with spatial field-of-view simulation and a two-stage temporal dropout process to mimic real wearable-camera conditions. PlayerOne automatically extracts synchronized motion-video pairs from Ego-Exo4D, Nymeria, FT-HID, and EgoExo-Fitness, while using EgoVid-5M for large-scale egocentric text-video pretraining (Zhang et al., 28 Dec 2025, Cho et al., 22 Dec 2025, Tu et al., 11 Jun 2025).

Large-scale automatic annotation is especially prominent in hand-centric and intent-centric work. MEgoHand curates a unified corpus of 3.35M RGB-D frames, 24K interactions, and 1.2K objects using an Inverse MANO Retargeting Network and a Virtual RGB-D Renderer. The sparse-3D-hand-joint video generator builds approximately 1 million high-quality Ego4D clips with synchronized hand trajectories. Hand2World derives hand geometry and camera trajectories from raw monocular egocentric video. LIME mines roughly 3 million intent-conditioned examples from RoomTour3D and Nymeria by pairing plausible intents and observation-gain descriptions with relative target camera poses (Zhou et al., 22 May 2025, Zhang et al., 12 Mar 2026, Wang et al., 10 Feb 2026, Sun et al., 2 Jul 2026).

Synthetic generation remains a parallel solution to the supervision bottleneck. EgoGen builds a closed-loop synthetic data generator in which motion synthesis directly drives an embodied egocentric camera and exports RGB, depth, normals, segmentation masks, optical flow, camera trajectories, and SMPL-X annotations (Li et al., 2024).

5. Evaluation protocols and empirical trends

Evaluation in egocentric motion generation is highly task-dependent, but several metric families recur. Full-body motion work typically reports MPJPE, PA-MPJPE or MPJPE-PA, hand errors, foot sliding, foot contact or air ratio, semantic similarity, and motion-distribution FID. UniEgoMotion reports reconstruction results including MPJPE 0.100, MPJPE-PA 0.053, MPJPE-H 0.180, Foot Slide 3.62, Semantic Similarity 0.918, and FID 0.027. EgoPriMo reports that, on Nymeria generation, MPJPE improves from 1.040 to 0.477, foot sliding from 4.35 to 1.26, and SemSim from 0.807 to 0.855 relative to UniEgoMotion, while UniEgoMotion is only better on M-FID there (Patel et al., 2 Aug 2025, Ge et al., 7 Jun 2026).

Joint video-motion generation has produced consistency metrics that are specific to first-person settings. EgoTwin introduces View Consistency by estimating camera pose from generated video and comparing it to head pose from generated motion after alignment, giving TransErr and RotErr, and introduces HandScore as an average F-score over visible hands. Against VidMLD, EgoTwin reports I-FID 98.17 versus 157.86, FVD 1033.52 versus 1547.28, TransErr 0.67 versus 1.28, RotErr 0.46 versus 1.53, and HandScore 0.81 versus 0.36 (Xiu et al., 18 Aug 2025).

Online and real-time methods add latency to the evaluation space. HMD$p(X_{1:N}\mid I_{1:N},T_{1:N})$3 reports roughly 0.17 s latency in the low-latency online setting with stride $p(X_{1:N}\mid I_{1:N},T_{1:N})$4. EgoForce reports inference time of about 0.08 s per frame, versus 84.9 s for EgoAllo and 2.05 s for UniEgoMotion, while remaining strictly online. HaMoS with optimization reports about 5.78 s, compared with about 99 s for EgoAllo with optimization (Guzov et al., 2024, Hwang et al., 13 May 2026, Cho et al., 22 Dec 2025).

Language-conditioned and active-perception variants use different metrics but reveal related trends. EgoMotion reports FID 0.0018, R@Top1 69.4, MM-Dist 0.872, FS 1.05, FC 12.3, Acce 0.0015, and Jerk 0.0015 on Nymeria, outperforming adapted text-to-motion baselines. LIME evaluates success under a shared motion budget rather than exact pose error and reports that, in LIBERO-Goal manipulation, using LIME as a preconditioning camera-motion step improves success from 26% to 74% overall in the sampled-start experiments. EgoFlow reports ADE 0.279, FDE 0.102, Geodesic 1.141, Collision rate 2.5%, and runtime 0.480s, with collision reduction by up to 79% relative to competing methods (Hou et al., 21 Apr 2026, Sun et al., 2 Jul 2026, Saroha et al., 1 Apr 2026).

A plausible implication is that evaluation is converging around three distinct questions: whether a sample is kinematically correct, whether it is semantically grounded in first-person context, and whether viewpoint change remains geometrically consistent with the generated motion.

6. Control, applications, and unresolved issues

The application space is broad. In humanoid robotics, EgoPriMo generates SMPL motion references that can be tracked by an existing Unitree whole-body controller, and the paper demonstrates prompts such as “Kick the ball with your left foot” and “Put up your hands and dance.” In embodied active perception, LIME predicts relative 6-DoF camera motions to obtain intent-relevant future views. In world simulation, PlayerOne generates egocentric videos aligned with real scene human motion captured by an exocentric camera, while E$p(X_{1:N}\mid I_{1:N},T_{1:N})$5C uses 3D environmental memory and ego/exo human control to produce physically grounded future egocentric video (Ge et al., 7 Jun 2026, Sun et al., 2 Jul 2026, Tu et al., 11 Jun 2025, Gu et al., 25 May 2026).

AR/VR and telepresence are a second major application domain. UniEgoMotion explicitly motivates AR/VR, human-robot interaction, assistive technologies, and adaptive healthcare solutions. MEgoHand targets immersive AR/VR and robotic imitation through hand-object interaction forecasting without explicit object CAD priors. EgoAnimate converts a single top-down egocentric image into a frontal avatar representation and then feeds it into downstream animation systems such as UniAnimate or MagicMan plus ExAvatar (Patel et al., 2 Aug 2025, Zhou et al., 22 May 2025, Türkoglu et al., 12 Jul 2025).

The literature is also notable for how often generation is paired with control disentanglement. PlayerOne separates body and feet, hands, and head. E$p(X_{1:N}\mid I_{1:N},T_{1:N})$6C separates persistent 3D scene structure from human-driven dynamics. EgoControl uses both AdaLN-style modulation and cross-attention for pose control. Hand2World isolates hand control from camera control by combining occlusion-invariant hand conditioning with Plücker-ray embeddings for viewpoint stabilization. This suggests a general design tendency: egocentric systems work better when scene memory, camera motion, and articulated body motion are not collapsed into a single undifferentiated control stream (Tu et al., 11 Jun 2025, Gu et al., 25 May 2026, Pallotta et al., 22 Nov 2025, Wang et al., 10 Feb 2026).

Several limitations recur across papers. Head motion alone remains ambiguous for full-body inference; HaMoS explicitly argues that head trajectory alone is highly ambiguous, and HMD$p(X_{1:N}\mid I_{1:N},T_{1:N})$7 makes the same point for single-HMD sensing. Foot sliding remains a persistent artifact; EgoPriMo identifies it as the main physical artifact, and HaMoS and EgoMotion both include explicit smoothness or skating-related losses. Some systems remain local rather than fully deliberative: “Moving by Looking” notes that the policy has no memory and therefore cannot systematically search a room, while LIME states that its model predicts local camera motions under a limited budget rather than a full long-horizon plan. Several papers also rely on upstream perception whose failures propagate forward, including SLAM, hand tracking, depth prediction, or monocular pose reconstruction (Cho et al., 22 Dec 2025, Guzov et al., 2024, Ge et al., 7 Jun 2026, Diomataris et al., 23 Sep 2025, Sun et al., 2 Jul 2026).

More fundamentally, the field contains an unresolved boundary between reconstruction and generation. Some methods aim to recover a specific motion instance from sparse first-person evidence; others synthesize plausible motions compatible with scene context and language; others generate first-person video conditioned on motion. The recent literature does not collapse these into one objective. Instead, it increasingly treats egocentric motion generation as a layered stack: multimodal reasoning over first-person context, structured motion or camera control, and a generative prior capable of maintaining temporal coherence, geometric consistency, and causal validity under severe partial observability.