EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction

Abstract: Accurately forecasting human trajectories from an egocentric perspective plays a central role in applications such as humanoid robotics, wearable sensing systems, and assistive navigation. However, progress in this direction remains limited due to the scarcity of egocentric trajectory datasets collected in real-world environments. Addressing this need, we introduce EgoTraj, an egocentric multimodal open dataset recorded using Meta Quest Pro (MQPro). EgoTraj contains 75 sequences of human navigation collected from multiple MQPro wearers in real-world urban environments. Each recording provides synchronized RGB video along with ground-truth data, including continuous time-synchronized 6-degree-of-freedom head poses, per-frame 3D eye gaze vectors, scene annotations. To the best of our knowledge, EgoTraj differs from typical egocentric trajectory datasets by capturing long-horizon, self-directed navigation across diverse urban routes with broad participant diversity. To demonstrate the potential of the dataset, we benchmark several state-of-the-art methods for egocentric trajectory prediction and conduct ablation studies to analyze the contributions of gaze, scene, and motion cues. The results highlight the utility of EgoTraj for AR-based perception, navigation, and assistive systems. The EgoTraj dataset, code, and EgoViz Dashboard are publicly available at https://github.com/yehiahmad/EgoTraj.

• The paper introduces EgoTraj, a privacy-compliant dataset of 75 participants, 10.7 hours, 1.15 million frames, synchronized 6DoF pose, 3D gaze, RGB video, and scene annotations from real urban navigation.
• Multimodal prediction combining ego-motion, scene context, social cues, and gaze achieves 0.12 m ADE, 0.23 m FDE, and 0.58 head-rotation error, reducing ADE by 37% versus ego-motion alone.
• The dataset generalizes across held-out waypoint pairs and unfamiliar participants with ADE increasing only from 0.12 m to 0.14 m, but abrupt turns, small evaluation splits, VLM label bias, and single-city collection remain limitations.

EgoTraj is a multimodal egocentric dataset for human trajectory prediction, recorded with Meta Quest Pro (MQPro) headsets by 75 participants navigating real urban environments (2605.19004). The paper addresses a concrete gap: existing trajectory benchmarks are dominated by bird's-eye or static-camera recordings that capture only externally observable motion, while prior egocentric datasets either lack synchronized gaze and ground-truth 6DoF pose, restrict collection to single participants or indoor settings, or target action recognition rather than navigation forecasting.

Dataset design and collection

Each participant wore an MQPro operating in full-color passthrough mode, whose sensor suite comprises a passthrough RGB camera, two infrared eye-tracking cameras, four inside-out tracking cameras, and a 6-axis IMU. A custom Unity application interfaces with the headset's visual–inertial SLAM system to log time-synchronized data at 30 Hz: 6DoF head pose (position, orientation quaternion, linear and angular velocity), binocular 3D gaze origin and direction vectors, and H.264-encoded RGB video at 1024×1024 resolution. Raw telemetry is resampled onto the 30 Hz video timeline using linear interpolation for positions, SLERP for quaternions, and normalized interpolation for gaze vectors; the released format is one HDF5 file per session with pose, gaze, and video groups.

The recording protocol is deliberately naturalistic rather than scripted. Participants navigated between pairs of seven predefined outdoor waypoints (21 origin–destination pairs) across sidewalks, crosswalks, and busy streets, choosing their own routes. To verify route diversity, the authors computed pairwise Dynamic Time Warping distances within each waypoint pair ($n=103$ comparisons), obtaining a median of 122.8 m with a range of $[19.3, 379.4]$ m — evidence that participants did not retrace templated paths. Sessions were capped at 15 minutes (8 minutes on average). The cohort spans ages 18–38, 14 nationalities, balanced gender, and includes 31 participants unfamiliar with the recording area. Each participant contributed exactly one session, so all splits are subject-disjoint by construction.

Privacy handling follows an IRB-approved protocol: written informed consent, withdrawal and redaction rights, EgoBlur-based de-identification of faces and license plates, anonymized session identifiers, and no public distribution of raw unprocessed video.

Scale and comparison to prior work

EgoTraj totals 10.7 hours, 1.15M RGB frames, and 46.73 km of cumulative walking across 75 sessions. The dataset-wide mean walking speed of 1.25 m/s matches typical pedestrian speeds, and 7% of frames exhibit near-stationary motion (<0.3 m/s), capturing stop-and-go dynamics. Relative to comparable egocentric trajectory datasets, EgoTraj's distinguishing combination is scale plus full modality coverage:

Dataset	Hours	Frames	Subjects	Gaze	6DoF	Scene ann.
KrishnaCam	70.0	7.6M	1	✗	✗	✗
Nymeria	300	32.4M	264	✓	✓	✗
LookOut	4.0	288K	N/P	✗	✓	✗
EgoCogNav (CEN)	6.0	432K	17	✓	✓	✗
EgoTraj	10.7	1.15M	75	✓	✓	✓

Nymeria is larger but captures activity-level trajectories in collaborative multi-actor settings rather than pedestrian navigation; KrishnaCam relies on GPS-only localization from a single wearer; LookOut lacks gaze; CEN targets cognition-aware modeling and is not exclusively outdoor. The claim that EgoTraj is the first large-scale egocentric trajectory dataset jointly providing synchronized 6DoF pose, per-frame 3D gaze, RGB video, and scene annotations in real traffic scenarios is plausible given this comparison, though it rests on the authors' own survey of related work.

Scene annotation via VLM pipeline

From 253 privacy-blurred segments, frames were sampled at 1 fps, yielding 38,606 annotated frames processed by Qwen2.5-VL-7B-Instruct with chain-of-thought prompting. Annotations cover environmental context, dynamic agents, traffic signals, gaze fixation targets, and inferred short-term intent. Quality was assessed on a 100-frame stratified sample: structural compliance reached 96% after up to two prompt retries, inter-annotator Cohen's $\kappa$ ranged from 0.83 (short-term intent) to 0.96 (traffic signals), and VLM accuracy against the human reference tracked annotator agreement closely (e.g., 0.98 on gaze target, 0.84 on intent). A 50-frame failure audit found residual error below 10%, concentrated on visually ambiguous scenes such as distant pedestrians and occluded traffic lights. These numbers indicate usable but not perfect annotations; the inferential "intent" field is the weakest link, which matters because downstream models may inherit VLM biases if trained on these labels.

Gaze calibration

A per-session quadratic model maps gaze yaw–pitch angles to pixel coordinates $(u,v)$, enabling projection of fixations into the egocentric image plane. This calibration operationalizes the visual-neuroscience finding that gaze anticipates locomotor actions by 1–2 seconds, which underpins the paper's central hypothesis about intention-aware prediction.

Benchmarking results

All baselines use $T_\text{obs}=1.5$ s and $T_\text{pred}=3.5$ s with an 80/10/10 session split. Because most egocentric prediction methods lack released code, the authors adapted five baselines: Constant Velocity, Linear Extrapolation, a Multimodal Transformer, CXA-Transformer, and EgoCast.

On the test split, adapted EgoCast achieves the best translation errors (ADE 0.16 m, FDE 0.28 m), while CXA-Transformer achieves the lowest head rotation error (0.69). Kinematic baselines degrade sharply on rotation — Linear Extrapolation reaches 1.39 — indicating that per-axis regression cannot capture nonlinear head motion.

The ablation study, using CXA-Transformer as the base architecture, quantifies modality contributions. Starting from ego-motion alone (ADE 0.19), scene segmentation reduces ADE to 0.16 and gaze to 0.15; among social representations, body pose outperforms bounding boxes and center points. The full configuration $\mathcal{Y}+\mathcal{P}+\mathcal{S}+\mathcal{G}$ achieves the strongest overall result: ADE 0.12 m, FDE 0.23 m, head rotation error 0.58 — a 37% ADE reduction over ego-motion alone. Notably, gaze improves performance even when combined with rich scene and social context, supporting the claim that visual attention carries predictive signal beyond kinematic and environmental cues.

Generalization was tested on two stricter splits beyond random participants: a waypoint-pair held-out split (3 of 21 OD pairs reserved, $n=10$ sessions) and an unfamiliar-participant split ($n=8$ sessions), both with 95% bootstrap confidence intervals from 1000 resamples. The full multimodal model degrades only modestly (ADE 0.12 → 0.14 → 0.14), suggesting the learned cues transfer across held-out landmark pairs and novel-to-area participants rather than overfitting route templates. However, these held-out splits contain only 8–10 sessions, so the confidence intervals are wide relative to the reported gaps, and conclusions about generalization rest on small evaluation sets.

Qualitative analysis and failure modes

Qualitative examples show multimodal models producing smoother, more socially compliant trajectories than kinematic baselines, which overshoot turns. A representative failure case involves a sharp ~90° intersection turn triggered by a traffic light change: all baselines deviate, learned models partially anticipate but underestimate curvature. The authors attribute this to limited predictive signal in short-horizon ego-motion history and insufficient representation of abrupt transitions in the data — a candid limitation that also reflects the inherent difficulty of deterministic forecasting under multimodal intent.

To isolate the anticipation regime, the authors identified "active transition" windows where turning begins within the final 0.5 s of the observation window. On this subset, the full multimodal CXA-Transformer achieves ADE $0.23 \pm 0.022$ and FDE $[19.3, 379.4]$0, and transformer-based predictors track the turn while motion-only baselines drift along the pre-turn heading — consistent with gaze leading motion by 1–2 s. This is the paper's most direct evidence for the value of gaze, though it remains qualitative/observational rather than a controlled causal test.

Limitations and open questions

Several constraints bear directly on the results. First, the benchmark omits direct comparison with EgoNav, LookOut, and EgoCogNav, since none had released code at the time; the baseline set consists of adaptations, so absolute numbers may not reflect the state of the art. Second, abrupt near-instantaneous sharp turns remain unsolved due to their scarcity in the data, and the authors explicitly leave this regime open. Third, the generalization splits are small ($[19.3, 379.4]$1–10 sessions), limiting statistical power. Fourth, scene annotations are VLM-generated; while human agreement is high, the short-term-intent field has the lowest reliability ($[19.3, 379.4]$2), and training on inferred intent risks propagating systematic VLM errors. Fifth, collection is restricted to one urban area and adults aged 18–38 with normal or corrected-to-normal vision, leaving cross-city transfer and broader demographic coverage untested. Finally, the active-transition analysis supports but does not isolate causally whether gaze drives the improvement versus correlated scene context near turns.

Conclusion

EgoTraj provides a subject-disjoint, privacy-compliant, publicly released corpus of 75 real-world egocentric navigation sessions with synchronized 6DoF pose, calibrated 3D gaze, RGB video, and VLM-generated scene annotations, together with a standardized benchmark and the EgoViz inspection dashboard. Its ablations establish quantitatively that gaze and scene context yield complementary predictive gains — culminating in ADE 0.12 m for the full multimodal configuration — and its generalization analysis suggests these cues transfer across routes and participants. The dataset's principal open problems are abrupt-turn prediction under multimodal intent, rigorous comparison against unreleased competing methods, and validation of VLM-derived intent annotations as supervision signals.