AND: Real-World Egocentric Navigation Dataset

• AND is a real-world egocentric navigation dataset capturing synchronized video and 6D head-pose data for future trajectory prediction.
• It comprises 274k RGB frames and 36,000 clips from diverse indoor and outdoor settings with dynamic obstacles.
• The dataset supports forecasting both translations and rotations, emphasizing collision-free and natural navigation behaviors.

Searching arXiv for the specified paper and closely related references to ground the article. arxiv_search(query="(Pan et al., 20 Aug 2025)", max_results=5) The Aria Navigation Dataset (AND) is a real-world egocentric navigation dataset introduced with “LookOut: Real-World Humanoid Egocentric Navigation” (Pan et al., 20 Aug 2025). It was collected using a single pair of Project Aria glasses and is organized around the task of predicting a sequence of future 6D head poses from egocentric video. AND consists of 4 hours of recording of users navigating in real-world scenarios, with 274,000 RGB frames, SLAM-processed 6D head-pose trajectories aligned to those frames, and clip-level packaging for forecasting with $T_1=T_2=8$. The dataset is explicitly designed to capture active information-gathering behavior, including head-turning events during natural navigation, and includes indoor and outdoor scenes with substantial pedestrian and vehicular activity (Pan et al., 20 Aug 2025).

1. Scope and problem setting

AND is defined by the future head-pose prediction problem. Given past video $$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$ and poses $H_{1:T_1}$, the task is to predict future poses $H_{T_1+1:T_1+T_2}$, with the LookOut instantiation fixing $T_1=T_2=8$ (Pan et al., 20 Aug 2025). Operationally, the benchmark input is a sequence of 8 posed RGB frames together with the corresponding 8 past 6D head poses, and the required output is 8 future 6D head poses.

The emphasis on predicting both translations and rotations is central. The formulation is not restricted to path continuation in the planar sense; it includes head orientation as part of the behavioral signal. This is directly tied to active information gathering, as the stated motivation includes learning behavior expressed through head-turning events. A plausible implication is that AND is intended not merely for locomotion forecasting, but for modeling the sensing strategy embedded in natural movement.

The abstract situates the problem in humanoid robotics, VR / AR, and assistive navigation (Pan et al., 20 Aug 2025). Within that framing, AND serves as a dataset for learning egocentric navigation policies from synchronized video and pose traces rather than from exocentric tracking or scripted demonstrations.

2. Acquisition pipeline and sensing stack

The acquisition pipeline uses a single pair of Project Aria glasses as the data capture device (Pan et al., 20 Aug 2025). The on-board sensors, all timestamped in the VRS recording format, are two fisheye RGB cameras at 20 fps, two monochrome SLAM cameras at 20 fps, eye-tracking cameras, an Inertial Measurement Unit (IMU), and a barometer and GPS. The stated advantages are that the setup is lightweight, non-intrusive, requires minimal per-session setup of less than 10 seconds, and supports naturalistic recording.

Raw data are stored in Meta’s VRS container, which preserves hardware timestamps. Post-processing via Aria Machine Perception Service (MPS) yields undistorted RGB frames, a SLAM-reconstructed 6D head-pose trajectory, a dense static point-cloud of the environment, and eye gaze estimates, although the eye gaze estimates are unused in LookOut (Pan et al., 20 Aug 2025). All modalities are time-synchronized by virtue of the VRS timestamps and MPS pipeline.

The collection protocol spans 18 densely trafficked locations, including indoor hallways, urban sidewalks, campus quads, and parks (Pan et al., 20 Aug 2025). Sessions were scheduled during peak pedestrian or vehicular traffic and at varied times of day. Subjects were instructed to “navigate naturally,” with emphasis on active information gathering such as looking both ways before crossing streets and looking down when stepping off curbs. No scripted trajectories were used. The resulting raw footage is reported as 274k RGB frames at 20 fps, corresponding to approximately 3.8 hours. This suggests that the dataset prioritizes behavioral realism over task regularization through prescribed routes.

3. Dataset composition and organization

At aggregate scale, AND contains 274,000 RGB frames and a total SLAM-processed 6D head-pose sequence aligned to the RGB frames (Pan et al., 20 Aug 2025). Clip segmentation is performed with a sliding window of length $T_1+T_2=16$ frames and stride 6 frames, corresponding to approximately 4.5 seconds per clip and yielding 36,000 clips.

The environment types include indoor corridors and lobbies with high ceilings and mixed lighting, outdoor sidewalks and plazas with direct sunlight and shadows, and road crossings with moving cars and cyclists (Pan et al., 20 Aug 2025). Dynamic obstacles listed in the dataset description include pedestrians, skateboarders, and rolling luggage. The quantitative summary further states that the average clip has approximately a 1.2 m translation span, that typical head-turn angles span up to $\pm 45^\circ$ in 4.5 seconds, and that approximately 70% of clips contain at least one pedestrian crossing.

The folder organization reflects both raw and processed representations. The raw folder contains .vrs files, one per session. The processed folder, organized per session, contains rgb/ for undistorted RGB frames as PNG at 20 fps, poses.npy as an $[N\times 9]$ array of 6D head poses, pointcloud.ply as the static environment point cloud, and clips/ as .npz files containing frames: [16\times H\times W\times 3] together with h_input: [8\times 9] and h_target: [8\times 9] (Pan et al., 20 Aug 2025).

A common misunderstanding would be to treat AND as a conventional egocentric video collection. The released structure indicates that it is a synchronized multimodal navigation dataset in which the video stream, pose stream, and static geometry are co-registered and directly consumable for forecasting and collision-oriented evaluation.

4. Ground truth, pose representation, and supervision

At time step $t$, the head pose is represented as $h_t = [t_t \mid r_t] \in \mathbb{R}^9$ (Pan et al., 20 Aug 2025). Here, $$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$0 is camera (head) translation in a head-centered canonical frame with Y-axis up, X forward, and Z right, while $$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$1 is the continuous 6D rotation representation, later converted to $$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$2. This parameterization makes translation and orientation jointly first-class prediction targets.

Ground-truth 6D head poses are obtained via Project Aria SLAM through MPS, with sub-centimeter translational accuracy and sub-degree rotational drift over short windows (Pan et al., 20 Aug 2025). The environment point cloud is fused from stereo SLAM cameras and filtered to remove dynamic points and noise. Additional labels include the static point cloud used for collision-distance metrics and dynamic-obstacle “distance” masks. The latter are not part of the ground-truth release and are computed from monocular depth estimation with Depth Pro plus semantic segmentation with DINOv2 and Mask2Former, taking the closest “person” pixel depth.

The training loss is defined as an average over the future horizon of translation $$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$3 error and rotation-matrix error:

$$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$4

with $$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$5 (Pan et al., 20 Aug 2025).

Two clarifications follow from the specification. First, the canonical frame is explicitly head-centered rather than world-centered. Second, eye gaze is available from MPS but is unused in LookOut; therefore, AND should not be conflated with an eye-gaze-supervised benchmark, even though gaze-related sensor data exist in the acquisition stack.

5. Benchmark protocol and evaluation metrics

The primary benchmark task is future 6D head-pose prediction over an 8-step horizon (Pan et al., 20 Aug 2025). Evaluation uses both direct pose error and collision-aware criteria. The primary pose metrics are

$$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$6

and

$$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$7

Collision-oriented evaluation is separated into static and dynamic non-collision scores at thresholds $$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$8 cm (Pan et al., 20 Aug 2025). For static geometry,

$$\mathcal{X}\in\mathbb{R}^{(T_1\times H\times W\times 3)}$$9

and

$H_{1:T_1}$0

The dataset description analogously defines $H_{1:T_1}$1 and $H_{1:T_1}$2 as fractions of predicted translations at or above threshold distance from the nearest static-cloud point or nearest dynamic-object estimate, respectively. Average non-collision is reported as

$H_{1:T_1}$3

This metric design indicates that benchmark performance is not reducible to low pose error alone. A plausible implication is that a model can be penalized even when its forecast is geometrically close to the reference trajectory if it places predictions near static structure or estimated dynamic obstacles. That emphasis is consistent with the motivating requirement of collision-free future trajectory prediction from egocentric observations.

6. Behavioral content, examples, and access conditions

The reported examples emphasize behaviors such as waiting, looking both ways, and rerouting (Pan et al., 20 Aug 2025). Figure 1 is described as an overlay of past black camera frustums versus future green frustums on an RGB frame; Figure 2 presents bird’s-eye trajectories overlaid on a height-map of the static cloud; Figure 3 shows five sample clips exhibiting waiting, looking-both-ways, and rerouting behaviors. In the abstract, the accompanying model is said to learn human-like navigation behaviors such as waiting or slowing down, rerouting, and looking around for traffic while generalizing to unseen environments (Pan et al., 20 Aug 2025). This suggests that AND was constructed to preserve behaviorally consequential context rather than only kinematic continuity.

Access is provided through the dataset project page at https://sites.google.com/stanford.edu/lookout. The dataset is open upon request under Meta Project Aria research guidelines, and the usage license is to be governed by a Stanford/Meta research agreement, according to the webpage (Pan et al., 20 Aug 2025). The project materials specify the citation as “Pan et al., LookOut: Real-World Humanoid Egocentric Navigation, CVPR 2025 (arXiv (Wang et al., 2024)).”

From an archival and reproducibility standpoint, two constraints are explicit. Dynamic-obstacle distance masks are not part of the ground-truth release, and access is request-based rather than unrestricted public download. Those conditions are relevant when interpreting benchmark portability, especially for evaluations that depend on the dynamic non-collision signal.