Egocentric Motion Forecasting

• Egocentric motion forecasting is a prediction task that infers diverse future states—from body pose to object trajectories—from first-person sensor data.
• It employs various modeling paradigms like retrieval methods, recurrent encoder-decoders, and diffusion-based techniques using ego-aligned coordinates.
• Empirical studies show that leveraging ego-motion, scene context, and structural priors leads to improved forecasting accuracy even under partial observability.

Searching arXiv for papers on egocentric motion forecasting and closely related formulations. Egocentric motion forecasting is the family of prediction problems in which future motion is inferred from first-person sensing rather than from an external camera or a bird’s-eye view. In this literature, the forecast target is not uniform: it can be the future locomotion of the camera wearer on the ground plane (Park et al., 2015), a future sequence of 12D egocentric camera configurations in one-on-one basketball (Bertasius et al., 2018), future image-plane localization of surrounding vehicles or pedestrians relative to a moving ego platform (Yao et al., 2018), future full-body pose of the wearer (Escobar et al., 2024), future hand trajectories and articulated hand pose (Hatano et al., 11 Apr 2025), future object motion in first-person interaction scenes (Saroha et al., 1 Apr 2026), or even future global Maps of Dynamics inferred from local egocentric video (Catalano et al., 26 Feb 2026). More recent work also treats future ego/camera trajectory as a latent control variable for predicting actions, plans, and outcomes rather than as an end in itself (Jun et al., 19 May 2026). This breadth is central to the field: “egocentric motion forecasting” names a viewpoint-constrained prediction setting, not a single canonical output variable.

1. Historical development and scope

Early work framed the problem as future localization from geometry alone. “Future Localization from an Egocentric Depth Image” predicted a set of plausible future ego-motion trajectories from a single first-person depth image, represented on the ground plane and evaluated up to 15 seconds ahead (Park et al., 2015). “Egocentric Basketball Motion Planning from a Single First-Person Image” instead forecast a sequence of 12D camera configurations encoding 3D location and 3D head orientation from a single opening frame in a one-on-one basketball possession (Bertasius et al., 2018). In driving, “Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems” forecast the future bounding boxes of a target vehicle, including center and scale, from 1 second of egocentric observations to 1 second of future boxes (Yao et al., 2018).

Subsequent work broadened the scope from path extrapolation to scene-aware embodied prediction. “HOIMotion” forecast future full-body pose during human-object interaction from past body pose, head orientation, and egocentric 3D object bounding boxes (Hu et al., 2024). “EgoNav” modeled a distribution of future wearer trajectories from egocentric RGB-D, semantics, and a visual-memory panorama tailored to wearable navigation (Wang et al., 2024). “EgoCast” studied forecasting the wearer’s future 3D full-body pose from egocentric RGB history plus headset translation and rotation, explicitly without requiring past ground-truth body poses at inference time (Escobar et al., 2024). “UniEgoMotion” unified reconstruction, forecasting, and generation of egocentric full-body motion from first-person images and ego-device trajectory within one conditional diffusion model (Patel et al., 2 Aug 2025).

A parallel strand moved toward finer manipulation-centric prediction. “The Invisible EgoHand” forecast future 3D hand trajectories and articulated hand pose for both visible and invisible hands by jointly denoising body and hand joints (Hatano et al., 11 Apr 2025), while “EggHand” forecast future 3D hand pose sequences in a normalized egocentric canonical frame using an egocentric video-text encoder and a VLA action decoder (Choi et al., 8 May 2026). Another branch focused on objects rather than bodies: “EgoFlow” generated long-horizon, physically plausible 6DoF object trajectories from egocentric observations, scene geometry, task text, and target end pose (Saroha et al., 1 Apr 2026). This suggests that the field has expanded from first-person trajectory continuation to a broader program of forecasting embodied state, interaction dynamics, and scene-mediated future behavior.

2. Problem formulations and motion representations

The most immediate distinction across formulations is the predicted state space. In HOI forecasting, body pose is defined as $p \in R^{3 \times n}$, scene objects as $o \in R^{3 \times 8 \times m}$, and head orientation as $h \in R^3$, with the task written as predicting $P_{t+1:T}$ from historical sequences $P_{1:t}, H_{1:t}, O_{1:t}$ (Hu et al., 2024). In UniEgoMotion, the forecasting task is $p(X_{n+1:N} \mid I_{1:n}, T_{1:n})$, where each motion frame is parameterized by SMPL-X variables $(R_i^g, t_i^g, \theta_i, \beta_i)$ (Patel et al., 2 Aug 2025). EggHand instead predicts $$Y = \mathbf{P}_{1:T_{\mathrm{fut}}} \in \mathbb{R}^{T_{\mathrm{fut}}\times J\times 3}$$, with $J=42$ on EgoExo4D and all 3D hand poses represented in a normalized egocentric canonical frame (Choi et al., 8 May 2026).

Other formulations are image-plane rather than 3D-body centric. Future vehicle localization represents each target box as $X_t=[c_t^x,c_t^y,w_t,h_t]$ and predicts the future sequence $o \in R^{3 \times 8 \times m}$0 in the ego camera frame (Yao et al., 2018). TrajMamba uses an 8D pedestrian state $o \in R^{3 \times 8 \times m}$1 and predicts future bounding boxes in the image plane from an egocentric viewpoint (Peng et al., 16 Mar 2026). NEMO separates future ego-motion $o \in R^{3 \times 8 \times m}$2 from future target boxes $o \in R^{3 \times 8 \times m}$3, explicitly propagating uncertainty from the ego stream into the target-localization stream (Malla et al., 2019).

Some work uses trajectory representations that are neither full-body joint sequences nor image-plane boxes. TrajPilot represents future ego-motion as per-segment relative 6-DoF controls $o \in R^{3 \times 8 \times m}$4, with $o \in R^{3 \times 8 \times m}$5 in rotation-vector form (Jun et al., 19 May 2026). EgoFlow represents manipulated-object motion as $o \in R^{3 \times 8 \times m}$6, where each frame is $o \in R^{3 \times 8 \times m}$7 with translation and continuous 6D rotation (Saroha et al., 1 Apr 2026). EgoMoD does not predict trajectories at all; it forecasts future global Maps of Dynamics through cellwise flow magnitude, dominant direction, and directional entropy on a 2D allocentric grid (Catalano et al., 26 Feb 2026).

A consistent representational theme is the use of ego-aligned coordinates. The EgoSpace map places future locomotion in a gaze-normalized ground-plane frame (Park et al., 2015). EggHand removes global translation and rotation using the camera pose at the first observation step (Choi et al., 8 May 2026). UniEgoMotion introduces a head-centric motion representation based on a canonical head frame projected onto the floor (Patel et al., 2 Aug 2025). This suggests that egocentric forecasting often benefits from choosing a coordinate system that matches the sensing device rather than inheriting a third-person root convention.

3. Conditioning signals: scene context, ego-motion, and intent

A central empirical lesson across the literature is that past kinematics alone are often underconstrained in first-person settings. HOIMotion makes this explicit by conditioning future body motion on egocentric 3D object bounding boxes and head orientation, selecting the two closest dynamic and two closest static objects to the viewport center and encoding them jointly with pose in a learned pose-object graph (Hu et al., 2024). EgoNav similarly argues that a single forward-facing frame is insufficient for navigation forecasting and therefore constructs a panoramic visual-memory representation from recent RGB-D and semantic observations (Wang et al., 2024).

Several papers elevate ego-motion from nuisance variable to primary conditioning signal. EMAG models ego-motion as a sequence of homography matrices between consecutive frames, predicts future ego-motion with an auxiliary decoder, and shows that adding the ego-motion loss improves hand forecasting from ADE/FDE $o \in R^{3 \times 8 \times m}$8 to $o \in R^{3 \times 8 \times m}$9 in the reported cross-dataset setting (Hatano et al., 2024). In NEMO, future ego velocity and yaw rate are forecast as a distribution, and sampled “noisy ego priors” condition future object localization (Malla et al., 2019). TrajMamba separates pedestrian motion and ego-motion into distinct Mamba encoders and uses an ego-motion-guided decoder rather than latent post-fusion, improving the PIE 1.0 s setting from $h \in R^3$0 to $h \in R^3$1 in ADE/FDE/ARB/FRB (Peng et al., 16 Mar 2026).

A stronger claim appears in trajectory-conditioned future understanding. TrajPilot argues that future camera trajectory “lets the model commit to one of those futures” and reports validation $h \in R^3$2 of $h \in R^3$3 with trajectory conditioning versus $h \in R^3$4 with text, while shuffling trajectory degrades performance to $h \in R^3$5 (Jun et al., 19 May 2026). This does not establish a direct geometric forecasting benchmark, but it suggests that future ego-motion can be a finer-grained carrier of intent than language.

Manipulation-centric methods add still richer context. EgoH4 conditions on egocentric video, camera pose, visible 2D hand coordinates, and full-body structure so that body joints constrain hidden-hand motion (Hatano et al., 11 Apr 2025). EggHand adds text descriptions and an egocentric video-text encoder, while explicitly relying on an egocentric-pretrained visual backbone for robustness under severe ego-motion (Choi et al., 8 May 2026). EgoFlow conditions future object motion on point clouds, nearby fixtures, object category, task prompt, observed history, and a target end pose (Saroha et al., 1 Apr 2026). A plausible implication is that “scene context” in egocentric forecasting is not a single modality; it ranges from raw appearance and motion to object geometry, semantic prompts, head trajectory, and structured pose cues.

4. Modeling paradigms

The field contains several distinct modeling traditions. One is retrieval-plus-optimization. The EgoSpace formulation retrieves future trajectories from a dataset using nearest-neighbor EgoSpace maps and refines them with an occlusion-aware objective, thereby handling multimodality as a set of plausible candidate futures rather than a single path (Park et al., 2015). TrajPilot likewise handles multimodality through candidate trajectory retrieval and learned gate-and-rank selection rather than continuous trajectory regression (Jun et al., 19 May 2026).

A second tradition uses recurrent probabilistic encoder-decoders. The driving FOL model employs a multi-stream GRU encoder-decoder over box history, ROI-pooled optical flow, and future ego-motion (Yao et al., 2018). NEMO extends this family with explicit Gaussian output distributions, MC-dropout-based epistemic uncertainty, and a two-stage factorization in which ego-motion samples condition target localization (Malla et al., 2019).

Graph models appear in full-body HOI forecasting. HOIMotion uses an encoder-residual-decoder GCN architecture, extracts pose features with temporal and spatial GCNs, encodes head and object information with MLPs, and fuses them in a fully connected pose-object graph with $h \in R^3$6 spatial nodes (Hu et al., 2024). The resulting forecast is a future sequence of 3D skeletons over 30 future frames at 30 Hz.

Diffusion and flow-based generative models have become prominent. EgoNav uses a UNet diffusion model conditioned on past trajectory and a 64D visual-memory embedding, with stochastic sampling over 15 futures and a hybrid DDIM+DDPM generation schedule for speed (Wang et al., 2024). UniEgoMotion formulates forecasting as conditional motion diffusion plus inpainting over the future interval (Patel et al., 2 Aug 2025). EgoH4 uses a diffusion-based transformer to jointly denoise body and hand joints (Hatano et al., 11 Apr 2025). EgoForce adopts a temporally asymmetric diffusion schedule over a streaming motion buffer, progressively denoising current and future states under strict causal constraints (Hwang et al., 13 May 2026). EgoFlow replaces diffusion with flow matching for object trajectory generation and adds gradient-guided inference for collision avoidance and smoothness (Saroha et al., 1 Apr 2026).

Recent models also import long-sequence and foundation-model machinery. TrajMamba uses separate Mamba encoders for pedestrian and ego-motion streams and an ego-motion-guided Mamba decoder (Peng et al., 16 Mar 2026). Uni-Hand uses dual-branch diffusion over egomotion and hand-motion latents with a hybrid Mamba-Transformer module (Ma et al., 17 Nov 2025). EggHand combines an egocentric video-text encoder with the action decoder from GR00T (Choi et al., 8 May 2026). These architectures suggest that first-person forecasting increasingly borrows from general sequence modeling, but adapts it around ego-specific structure such as head alignment, missing observations, and hand-head coupling.

5. Datasets, protocols, and evaluation

The datasets used in egocentric motion forecasting vary sharply by target and sensing regime. Ego-motion path prediction from depth used the EgoMotion dataset with 21 scenes, 55,933 frames, and 7.7 hours of wearable stereo video (Park et al., 2015). Driving localization work introduced HEV-I with 230 videos and 2477 vehicles, using 1 second observation and 1 second forecast at 10 fps (Yao et al., 2018), while NEMO supplemented HEV-I with IMU/CAN odometry and evaluated 1 second of observation against 2 seconds of prediction (Malla et al., 2019). HOI full-body forecasting relies on ADT and MoGaze because they provide both full-body motion and 3D object context (Hu et al., 2024). EgoNav introduced an Egocentric Navigation Dataset with 34 recording sessions, 198 minutes of synchronized wearable data at 20 Hz (Wang et al., 2024). Full-body, hand, and unified motion models increasingly rely on EgoExo4D-derived data, including EE4D-Motion with 143K training clips and 4400 validation clips after filtering (Patel et al., 2 Aug 2025), and EgoH4’s 156K training and 34K test sequences for 3D hand forecasting (Hatano et al., 11 Apr 2025).

Because the outputs differ, the metrics do as well. Full-body and hand-pose papers commonly report MPJPE and horizon-specific errors (Hu et al., 2024). Path and localization papers use ADE, FDE, and sometimes FIOU or Hausdorff-style criteria (Yao et al., 2018). EgoFlow reports ADE, FDE, Fréchet Distance, geodesic distance, and collision rate for object motion (Saroha et al., 1 Apr 2026). EgoNav emphasizes Collision-Free Score, smoothness, and Best-of-$h \in R^3$7 over sampled futures (Wang et al., 2024). EgoMoD evaluates map quality with MSE, MAE, SSIM, Accuracy, and IoU over flow, entropy, and dominant direction (Catalano et al., 26 Feb 2026). TrajPilot evaluates predicted future trajectories only indirectly through downstream action/planning/outcome metrics rather than ADE/FDE (Jun et al., 19 May 2026).

This heterogeneity complicates direct comparison. A lower MPJPE for full-body pose, a higher FIOU for future boxes, a lower collision rate for object motion, and better downstream action retrieval under trajectory conditioning do not measure the same property. This suggests that egocentric motion forecasting is still a collection of related subproblems rather than a single benchmarked task with a stable evaluation protocol.

6. Empirical regularities, misconceptions, and open problems

One repeated result is that ego-centric context helps. HOIMotion reports $h \in R^3$8 mm average MPJPE on MoGaze-all, compared with $h \in R^3$9 mm when static objects, dynamic objects, and head direction are removed, and shows that selecting two dynamic plus two static objects works best (Hu et al., 2024). EgoNav reports that removing visual input degrades Collision-Free Score from $P_{t+1:T}$0 to $P_{t+1:T}$1 and Best of 1 from $P_{t+1:T}$2 to $P_{t+1:T}$3 on the full-dataset model (Wang et al., 2024). EMAG, NEMO, and TrajMamba all show gains from explicit ego-motion modeling rather than treating camera motion as an implicit nuisance (Hatano et al., 2024).

A second regularity is that richer structural priors improve forecasting under partial observability. EgoH4 improves over EgoEgoForecast by $P_{t+1:T}$4 cm in overall hand-trajectory ADE and $P_{t+1:T}$5 cm in overall hand-pose MPJPE by jointly modeling body and hands, visibility, and reprojection consistency (Hatano et al., 11 Apr 2025). EggHand improves final-horizon and articulated metrics over EgoH4, with $P_{t+1:T}$6 FDE versus $P_{t+1:T}$7 and $P_{t+1:T}$8 MPJPE versus $P_{t+1:T}$9, while also improving the high-egomotion subset (Choi et al., 8 May 2026). EgoFlow reduces collision from $P_{1:t}, H_{1:t}, O_{1:t}$0 to $P_{1:t}, H_{1:t}, O_{1:t}$1 when collision guidance is added during flow-matching inference (Saroha et al., 1 Apr 2026). These results suggest that kinematic structure, scene structure, and physically informed inference matter most when the first-person view is sparse, noisy, or occluded.

A common misconception is that every paper labeled “egocentric motion forecasting” predicts the same thing. In practice, the label covers at least five distinct problem classes: forecasting the wearer’s own body motion (Escobar et al., 2024), forecasting hand or object interaction motion (Hatano et al., 11 Apr 2025), forecasting surrounding-agent localization relative to the ego camera (Yao et al., 2018), forecasting latent future head/camera trajectory for downstream understanding (Jun et al., 19 May 2026), and forecasting environment-wide dynamic maps (Catalano et al., 26 Feb 2026). Another misconception is that future trajectory accuracy is always measured directly. TrajPilot is explicit that its contribution is “trajectory-conditioned future understanding” rather than classical geometric ego-motion benchmarking (Jun et al., 19 May 2026). EgoForce is framed as online reconstruction but maintains a future motion buffer with temporally asymmetric uncertainty, placing it between filtering and forecasting (Hwang et al., 13 May 2026).

Several limitations recur. Dataset availability remains narrow for methods requiring aligned full-body motion and rich egocentric context (Hu et al., 2024). Pseudo-ground-truth motion and exocentric fitting introduce annotation noise in large-scale datasets (Patel et al., 2 Aug 2025). Many methods are deterministic despite acknowledging multimodal futures (Choi et al., 8 May 2026). Some models assume static or known environments (Saroha et al., 1 Apr 2026) or degrade when the ego-motion source changes distribution (Jun et al., 19 May 2026). Long-horizon evaluation is also unresolved: UniEgoMotion forecasts 6 seconds ahead but evaluates MPJPE only over the first 2 predicted seconds because farther futures may diverge while remaining plausible (Patel et al., 2 Aug 2025).

Overall, the literature supports a clear technical thesis: egocentric motion forecasting is not merely motion extrapolation from first-person video. The strongest methods condition future prediction on ego-aligned structure—head trajectory, scene geometry, visual memory, object layout, body constraints, or task signals—and increasingly treat uncertainty, missing observations, and multimodality as first-class aspects of the problem. What remains unsettled is how to unify these diverse targets and protocols into a common forecasting framework without erasing the viewpoint-specific structure that makes the egocentric setting distinct.