---
title: Egocentric Motion Forecasting
url: https://www.emergentmind.com/topics/egocentric-motion-forecasting
type: topic
---

# Egocentric Motion Forecasting

Searching arXiv for recent papers on egocentric motion forecasting and closely related formulations.
Egocentric motion forecasting is the family of prediction problems in which future motion is inferred from first-person sensing rather than from an external camera or a bird’s-eye view. In this literature, the forecast target is not uniform: it can be the future locomotion of the camera wearer on the ground plane [1509.02094], a future sequence of 12D egocentric camera configurations in one-on-one basketball [1803.01413], future image-plane localization of surrounding vehicles or pedestrians relative to a moving ego platform [1809.07408], future full-body pose of the wearer [2412.02903], future hand trajectories and articulated hand pose [2504.08654], future object motion in first-person interaction scenes [2604.01421], or even future global Maps of Dynamics inferred from local egocentric video [2603.00167]. More recent work also treats future ego/camera trajectory as a latent control variable for predicting actions, plans, and outcomes rather than as an end in itself [2605.20388]. This breadth is central to the field: “egocentric motion forecasting” names a viewpoint-constrained prediction setting, not a single canonical output variable.

## 1. Historical development and scope

Early work framed the problem as future localization from geometry alone. “Future Localization from an Egocentric Depth Image” predicted a set of plausible future ego-motion trajectories from a single first-person depth image, represented on the ground plane and evaluated up to 15 seconds ahead [1509.02094]. “Egocentric Basketball Motion Planning from a Single First-Person Image” instead forecast a sequence of 12D camera configurations encoding 3D location and 3D head orientation from a single opening frame in a one-on-one basketball possession [1803.01413]. In driving, “Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems” forecast the future bounding boxes of a target vehicle, including center and scale, from 1 second of egocentric observations to 1 second of future boxes [1809.07408].

Subsequent work broadened the scope from path extrapolation to scene-aware embodied prediction. “HOIMotion” forecast future full-body pose during human-object interaction from past body pose, head orientation, and egocentric 3D object bounding boxes [2407.02633]. “EgoNav” modeled a distribution of future wearer trajectories from egocentric RGB-D, semantics, and a visual-memory panorama tailored to wearable navigation [2403.19026]. “EgoCast” studied forecasting the wearer’s future 3D full-body pose from egocentric RGB history plus headset translation and rotation, explicitly without requiring past ground-truth body poses at inference time [2412.02903]. “UniEgoMotion” unified reconstruction, forecasting, and generation of egocentric full-body motion from first-person images and ego-device trajectory within one conditional diffusion model [2508.01126].

A parallel strand moved toward finer manipulation-centric prediction. “The Invisible EgoHand” forecast future 3D hand trajectories and articulated hand pose for both visible and invisible hands by jointly denoising body and hand joints [2504.08654], while “EggHand” forecast future 3D hand pose sequences in a normalized egocentric canonical frame using an egocentric video-text encoder and a VLA action decoder [2605.07642]. Another branch focused on objects rather than bodies: “EgoFlow” generated long-horizon, physically plausible 6DoF object trajectories from egocentric observations, scene geometry, task text, and target end pose [2604.01421]. This suggests that the field has expanded from first-person trajectory continuation to a broader program of forecasting embodied state, interaction dynamics, and scene-mediated future behavior.

## 2. Problem formulations and motion representations

The most immediate distinction across formulations is the predicted state space. In HOI forecasting, body pose is defined as \(p \in R^{3 \times n}\), scene objects as \(o \in R^{3 \times 8 \times m}\), and head orientation as \(h \in R^3\), with the task written as predicting \(P_{t+1:T}\) from historical sequences \(P_{1:t}, H_{1:t}, O_{1:t}\) [2407.02633]. In UniEgoMotion, the forecasting task is \(p(X_{n+1:N} \mid I_{1:n}, T_{1:n})\), where each motion frame is parameterized by SMPL-X variables \((R_i^g, t_i^g, \theta_i, \beta_i)\) [2508.01126]. EggHand instead predicts \(Y = \mathbf{P}_{1:T_{\mathrm{fut}}} \in \mathbb{R}^{T_{\mathrm{fut}}\times J\times 3}\), with \(J=42\) on EgoExo4D and all 3D hand poses represented in a normalized egocentric canonical frame [2605.07642].

Other formulations are image-plane rather than 3D-body centric. Future vehicle localization represents each target box as \(X_t=[c_t^x,c_t^y,w_t,h_t]\) and predicts the future sequence \(\mathbf{Y}\) in the ego camera frame [1809.07408]. TrajMamba uses an 8D pedestrian state \(M_t=(x_t,y_t,w_t,h_t,v_t^x,v_t^y,\Delta w,\Delta h)\) and predicts future bounding boxes in the image plane from an egocentric viewpoint [2603.14739]. NEMO separates future ego-motion \(E=\{v,\dot\theta\}\) from future target boxes \(B_t=\{c_{x,t},c_{y,t},w_t,h_t\}\), explicitly propagating uncertainty from the ego stream into the target-localization stream [1909.08150].

Some work uses trajectory representations that are neither full-body joint sequences nor image-plane boxes. TrajPilot represents future ego-motion as per-segment relative 6-DoF controls \(u \in \mathbb{R}^{16 \times 6}\), with \([\Delta x,\Delta y,\Delta z,\Delta r_x,\Delta r_y,\Delta r_z]\) in rotation-vector form [2605.20388]. EgoFlow represents manipulated-object motion as \(\mathbf{x}_{H+1:T}\in\mathbb{R}^{(T-H)\times 9}\), where each frame is \([\mathbf{p}_t;\mathbf{r}_t]\in\mathbb{R}^9\) with translation and continuous 6D rotation [2604.01421]. EgoMoD does not predict trajectories at all; it forecasts future global Maps of Dynamics through cellwise flow magnitude, dominant direction, and directional entropy on a 2D allocentric grid [2603.00167].

A consistent representational theme is the use of ego-aligned coordinates. The EgoSpace map places future locomotion in a gaze-normalized ground-plane frame [1509.02094]. EggHand removes global translation and rotation using the camera pose at the first observation step [2605.07642]. UniEgoMotion introduces a head-centric motion representation based on a canonical head frame projected onto the floor [2508.01126]. This suggests that egocentric forecasting often benefits from choosing a coordinate system that matches the sensing device rather than inheriting a third-person root convention.

## 3. Conditioning signals: scene context, ego-motion, and intent

A central empirical lesson across the literature is that past kinematics alone are often underconstrained in first-person settings. HOIMotion makes this explicit by conditioning future body motion on egocentric 3D object bounding boxes and head orientation, selecting the two closest dynamic and two closest static objects to the viewport center and encoding them jointly with pose in a learned pose-object graph [2407.02633]. EgoNav similarly argues that a single forward-facing frame is insufficient for navigation forecasting and therefore constructs a panoramic visual-memory representation from recent RGB-D and semantic observations [2403.19026].

Several papers elevate ego-motion from nuisance variable to primary conditioning signal. EMAG models ego-motion as a sequence of homography matrices between consecutive frames, predicts future ego-motion with an auxiliary decoder, and shows that adding the ego-motion loss improves hand forecasting from ADE/FDE \(52.84/57.08\) to \(52.35/56.57\) in the reported cross-dataset setting [2405.20030]. In NEMO, future ego velocity and yaw rate are forecast as a distribution, and sampled “noisy ego priors” condition future object localization [1909.08150]. TrajMamba separates pedestrian motion and ego-motion into distinct Mamba encoders and uses an ego-motion-guided decoder rather than latent post-fusion, improving the PIE 1.0 s setting from \(7.91/17.87/11.99/23.19\) to \(7.72/16.99/11.60/22.00\) in ADE/FDE/ARB/FRB [2603.14739].

A stronger claim appears in trajectory-conditioned future understanding. TrajPilot argues that future camera trajectory “lets the model commit to one of those futures” and reports validation \(\ell_1\) of \(0.4572\) with trajectory conditioning versus \(0.4761\) with text, while shuffling trajectory degrades performance to \(0.4964\) [2605.20388]. This does not establish a direct geometric forecasting benchmark, but it suggests that future ego-motion can be a finer-grained carrier of intent than language.

Manipulation-centric methods add still richer context. EgoH4 conditions on egocentric video, camera pose, visible 2D hand coordinates, and full-body structure so that body joints constrain hidden-hand motion [2504.08654]. EggHand adds text descriptions and an egocentric video-text encoder, while explicitly relying on an egocentric-pretrained visual backbone for robustness under severe ego-motion [2605.07642]. EgoFlow conditions future object motion on point clouds, nearby fixtures, object category, task prompt, observed history, and a target end pose [2604.01421]. A plausible implication is that “scene context” in egocentric forecasting is not a single modality; it ranges from raw appearance and motion to object geometry, semantic prompts, head trajectory, and structured pose cues.

## 4. Modeling paradigms

The field contains several distinct modeling traditions. One is retrieval-plus-optimization. The EgoSpace formulation retrieves future trajectories from a dataset using nearest-neighbor EgoSpace maps and refines them with an occlusion-aware objective, thereby handling multimodality as a set of plausible candidate futures rather than a single path [1509.02094]. TrajPilot likewise handles multimodality through candidate trajectory retrieval and learned gate-and-rank selection rather than continuous trajectory regression [2605.20388].

A second tradition uses recurrent probabilistic encoder-decoders. The driving FOL model employs a multi-stream GRU encoder-decoder over box history, ROI-pooled optical flow, and future ego-motion [1809.07408]. NEMO extends this family with explicit Gaussian output distributions, MC-dropout-based epistemic uncertainty, and a two-stage factorization in which ego-motion samples condition target localization [1909.08150].

Graph models appear in full-body HOI forecasting. HOIMotion uses an encoder-residual-decoder GCN architecture, extracts pose features with temporal and spatial GCNs, encodes head and object information with MLPs, and fuses them in a fully connected pose-object graph with \(n+15\) spatial nodes [2407.02633]. The resulting forecast is a future sequence of 3D skeletons over 30 future frames at 30 Hz.

Diffusion and flow-based generative models have become prominent. EgoNav uses a UNet diffusion model conditioned on past trajectory and a 64D visual-memory embedding, with stochastic sampling over 15 futures and a hybrid DDIM+DDPM generation schedule for speed [2403.19026]. UniEgoMotion formulates forecasting as conditional motion diffusion plus inpainting over the future interval [2508.01126]. EgoH4 uses a diffusion-based transformer to jointly denoise body and hand joints [2504.08654]. EgoForce adopts a temporally asymmetric diffusion schedule over a streaming motion buffer, progressively denoising current and future states under strict causal constraints [2605.13041]. EgoFlow replaces diffusion with flow matching for object trajectory generation and adds gradient-guided inference for collision avoidance and smoothness [2604.01421].

Recent models also import long-sequence and foundation-model machinery. TrajMamba uses separate Mamba encoders for pedestrian and ego-motion streams and an ego-motion-guided Mamba decoder [2603.14739]. Uni-Hand uses dual-branch diffusion over egomotion and hand-motion latents with a hybrid Mamba-Transformer module [2511.12878]. EggHand combines an egocentric video-text encoder with the action decoder from GR00T [2605.07642]. These architectures suggest that first-person forecasting increasingly borrows from general sequence modeling, but adapts it around ego-specific structure such as head alignment, missing observations, and hand-head coupling.

## 5. Datasets, protocols, and evaluation

The datasets used in egocentric motion forecasting vary sharply by target and sensing regime. Ego-motion path prediction from depth used the EgoMotion dataset with 21 scenes, 55,933 frames, and 7.7 hours of wearable stereo video [1509.02094]. Driving localization work introduced HEV-I with 230 videos and 2477 vehicles, using 1 second observation and 1 second forecast at 10 fps [1809.07408], while NEMO supplemented HEV-I with IMU/CAN odometry and evaluated 1 second of observation against 2 seconds of prediction [1909.08150]. HOI full-body forecasting relies on ADT and MoGaze because they provide both full-body motion and 3D object context [2407.02633]. EgoNav introduced an Egocentric Navigation Dataset with 34 recording sessions, 198 minutes of synchronized wearable data at 20 Hz [2403.19026]. Full-body, hand, and unified motion models increasingly rely on EgoExo4D-derived data, including EE4D-Motion with 143K training clips and 4400 validation clips after filtering [2508.01126], and EgoH4’s 156K training and 34K test sequences for 3D hand forecasting [2504.08654].

Because the outputs differ, the metrics do as well. Full-body and hand-pose papers commonly report MPJPE and horizon-specific errors [2407.02633]. Path and localization papers use ADE, FDE, and sometimes FIOU or Hausdorff-style criteria [1809.07408]. EgoFlow reports ADE, FDE, Fréchet Distance, geodesic distance, and collision rate for object motion [2604.01421]. EgoNav emphasizes Collision-Free Score, smoothness, and Best-of-\(N\) over sampled futures [2403.19026]. EgoMoD evaluates map quality with MSE, MAE, SSIM, Accuracy, and IoU over flow, entropy, and dominant direction [2603.00167]. TrajPilot evaluates predicted future trajectories only indirectly through downstream action/planning/outcome metrics rather than ADE/FDE [2605.20388].

This heterogeneity complicates direct comparison. A lower MPJPE for full-body pose, a higher FIOU for future boxes, a lower collision rate for object motion, and better downstream action retrieval under trajectory conditioning do not measure the same property. This suggests that egocentric motion forecasting is still a collection of related subproblems rather than a single benchmarked task with a stable evaluation protocol.

## 6. Empirical regularities, misconceptions, and open problems

One repeated result is that ego-centric context helps. HOIMotion reports \(76.1\) mm average MPJPE on MoGaze-all, compared with \(79.0\) mm when static objects, dynamic objects, and head direction are removed, and shows that selecting two dynamic plus two static objects works best [2407.02633]. EgoNav reports that removing visual input degrades Collision-Free Score from \(90.6\) to \(82.5\) and Best of 1 from \(0.81\) to \(1.19\) on the full-dataset model [2403.19026]. EMAG, NEMO, and TrajMamba all show gains from explicit ego-motion modeling rather than treating camera motion as an implicit nuisance [2405.20030].

A second regularity is that richer structural priors improve forecasting under partial observability. EgoH4 improves over EgoEgoForecast by \(3.4\) cm in overall hand-trajectory ADE and \(5.1\) cm in overall hand-pose MPJPE by jointly modeling body and hands, visibility, and reprojection consistency [2504.08654]. EggHand improves final-horizon and articulated metrics over EgoH4, with \(0.271\) FDE versus \(0.333\) and \(0.076\) MPJPE versus \(0.116\), while also improving the high-egomotion subset [2605.07642]. EgoFlow reduces collision from \(11.6\%\) to \(2.5\%\) when collision guidance is added during flow-matching inference [2604.01421]. These results suggest that kinematic structure, scene structure, and physically informed inference matter most when the first-person view is sparse, noisy, or occluded.

A common misconception is that every paper labeled “egocentric motion forecasting” predicts the same thing. In practice, the label covers at least five distinct problem classes: forecasting the wearer’s own body motion [2412.02903], forecasting hand or object interaction motion [2504.08654], forecasting surrounding-agent localization relative to the ego camera [1809.07408], forecasting latent future head/camera trajectory for downstream understanding [2605.20388], and forecasting environment-wide dynamic maps [2603.00167]. Another misconception is that future trajectory accuracy is always measured directly. TrajPilot is explicit that its contribution is “trajectory-conditioned future understanding” rather than classical geometric ego-motion benchmarking [2605.20388]. EgoForce is framed as online reconstruction but maintains a future motion buffer with temporally asymmetric uncertainty, placing it between filtering and forecasting [2605.13041].

Several limitations recur. Dataset availability remains narrow for methods requiring aligned full-body motion and rich egocentric context [2407.02633]. Pseudo-ground-truth motion and exocentric fitting introduce annotation noise in large-scale datasets [2508.01126]. Many methods are deterministic despite acknowledging multimodal futures [2605.07642]. Some models assume static or known environments [2604.01421] or degrade when the ego-motion source changes distribution [2605.20388]. Long-horizon evaluation is also unresolved: UniEgoMotion forecasts 6 seconds ahead but evaluates MPJPE only over the first 2 predicted seconds because farther futures may diverge while remaining plausible [2508.01126].

Overall, the literature supports a clear technical thesis: egocentric motion forecasting is not merely motion extrapolation from first-person video. The strongest methods condition future prediction on ego-aligned structure—head trajectory, scene geometry, visual memory, object layout, body constraints, or task signals—and increasingly treat uncertainty, missing observations, and multimodality as first-class aspects of the problem. What remains unsettled is how to unify these diverse targets and protocols into a common forecasting framework without erasing the viewpoint-specific structure that makes the egocentric setting distinct.

Source: https://www.emergentmind.com/topics/egocentric-motion-forecasting