Nymeria: Multimodal Egocentric Dataset

• Nymeria dataset is a comprehensive multimodal collection capturing 300 hours of unconstrained human activities with synchronized sensor data in natural settings.
• It integrates high-resolution egocentric videos, full-body motion capture, inertial measurements, and detailed scene annotations via a hierarchical protocol.
• Nymeria supports embodied AI research by enabling benchmark tasks in motion tracking, generative motion priors, and motion-to-language grounding.

Nymeria is a large-scale, multimodal dataset of egocentric human motion captured in natural daily settings, designed to support research in embodied AI, egocentric perception, action recognition, motion-language grounding, and related domains. The dataset features synchronized full-body motion ground truth, multiple egocentric video modalities (including RGB, grayscale, and eye-tracking), inertial and environmental sensors, and observer-perspective recordings, all precisely localized and temporally interaligned in a unified metric 3D frame. Nymeria also includes a comprehensive hierarchical annotation protocol with fine-grained motion narration, atomic actions, and high-level activity summarization in natural language, making it the world's largest motion-language dataset. The subsequent NymeriaPlus extension further enhances the resource with improved parametric motion models, dense 2D/3D scene object labels, instance-level mesh reconstruction, and new multimodal data streams (Ma et al., 2024, DeTone et al., 19 Mar 2026).

1. Dataset Scale, Demographics, and Recording Protocol

Nymeria comprises 1,200 sequences (∼15 minutes each), totaling 300 hours of recorded activity, with 264 participants balanced for gender (48.5% female/51.5% male) and spanning seven ethnicities, ages 18–50+, heights 150–200 cm, and weights 40–100 kg. Recordings were performed across 50 diverse real-world locations (homes, cafeteria, office, campus), covering 20 indoor/outdoor scenarios (e.g., cooking, hiking), 15% outdoor, and 15% two-participant sessions. Device trajectory statistics include 399 km traveled by the head-camera and 1,053 km by both wrists. All data capture occurred in natural, unscripted conditions: participants wore motion-capture suits (Xsens MVN Link, 17 IMUs) and head-mounted Project Aria glasses, performing unconstrained everyday tasks without scripts. Data collection emphasizes ecological validity and diversity—activities range from manipulation to locomotion and social interaction (Ma et al., 2024, Bai et al., 26 Jun 2025).

2. Modalities, Synchronization, and Data Formats

Nymeria provides the following synchronized modalities per session:

• Egocentric Video:
  • Project Aria RGB (54M frames, 30 Hz, 1,408×1,408) and grayscale (151.2M frames at 640×480), with additional observer-perspective video when available.
  • Eye-tracking: 10.8M gaze samples (320×240, 10 Hz).
• Full-body Xsens MVN Motion Capture: 260M frames at 240 Hz; outputs 23 segment orientations; for PEVA, only upper body (15 joints + 3-DoF root) is used and downsampled to 4 Hz, joints as ΔEuler angles in [–π,π].
• IMU and Environmental Sensors: Barometer, magnetometer, device-recorded audio (7x 48 kHz, participant and observer), and wristbands with 1,000 Hz IMU sampling.
• Device Trajectories: Time-stamped SE(3) extrinsics (head, wrists), localized via SLAM/BA to a shared metric 3D world.
• Scene Structure: NymeriaPlus adds semi-dense 3D point clouds (0.5–3M points/scene), dense 2D/3D bounding boxes, and instance-level object reconstructions (ShapeR pipeline).
• Wristband Cameras: Additional RGB and grayscale video streams at 30 Hz (NymeriaPlus).

All sensors are time-synchronized to sub-millisecond accuracy, with global calibration aligning Project Aria and Xsens coordinate frames into a single gravity-upright, metric world space. File formats include .npz/JSON for parametric motion (MHR, SMPL), .ply for 3D shapes, WAV for audio, and custom metadata/pose formats (Ma et al., 2024, DeTone et al., 19 Mar 2026, Bai et al., 26 Jun 2025).

3. Hierarchical Annotation Protocols and Preprocessing

Annotation in Nymeria proceeds along a three-tiered schema:

• Motion Narration: Fine-grained (∼3–5 s) clips with frame-accurate posture-focused language prompts (full-body, arms, lower-body, intent), totaling 38.6 h and 117.2K segments.
• Atomic Actions: Verb-centric (∼3–5 s) segments, 207 h and 170.6K annotated actions covering "describe the action, posture, direction, and interactions."
• Activity Summarization: Coarse (15–30 s) segments, 196 h and 22.6K summaries.
• Language Data: 301.5K sentences, 8.64M words, vocabulary size 6,545 (average 27.8 words/sentence).

Annotation uses custom UIs presenting time-synchronized egocentric and third-person video with 3D rendered motion. Training for annotators and quality control via manual review yield 92% usable XSens data, <1% frame drop (head), <6.3% wrist dropout, and robust action granularity. Atomic action extraction uses thresholded joint-delta vectors (Ma et al., 2024, Bai et al., 26 Jun 2025).

Raw IMU and pose data are filtered, downsampled, and normalized: translations to [–1,1], rotations to [–π,π]. Control representations are computed as relative pose deltas, $\mathbf{a}_t = \mathbf{p}_t - \mathbf{p}_{t-1}$. PEVA and related methods reshape the sequence as $\mathcal{D} = \{(x_0, a_0), ..., (x_T, a_T)\}$ for Markovian video prediction (Bai et al., 26 Jun 2025).

4. Object, Scene, and Environment Annotations (NymeriaPlus)

NymeriaPlus extends Nymeria with improved motion representations (MHR, SMPL) and comprehensive 2D/3D scene understanding:

• Dense 2D Bounding Boxes: Projected from 3D OBBs; 19 closed-set classes (e.g., bed, chair, wall), >10,000 unique instances; >50,000 visible boxes over sequences.
• Dense 3D Bounding Boxes: Each box $B$ is defined by center $c\in\mathbb{R}^3$, extents $(l_x, l_y, l_z)$, rotation $R\in SO(3)$; open-set "Anything" objects labeled ($\sim$12 per recording, 12,000 total, 2,896 dense in 5 venues). Intersection-over-Union (IoU$_3$) metrics and convex polytope algorithms for evaluation.
• Instance-Level Reconstructions: Automated via ShapeR, producing metric .ply meshes per object instance, with manual QC (>3 rating).
• Basemap Scans: 47 indoor venues, 5–15 min per scan, for static reference and annotation transfer.
• Additional Modalities: Wristband videos, extended audio (600+ hours), synchronized IMU, and environmental sensors.

Metadata includes global calibration (camera intrinsics/extrinsics, hand-eye), per-frame device extrinsics, participant anthropometrics, and action/scene descriptors. The dataset is structured for scalability, with directory organization by modality and sequence (DeTone et al., 19 Mar 2026).

5. Benchmark Tasks, Metrics, and Baseline Performance

Nymeria enables benchmarking for a broad suite of embodied and multimodal AI tasks:

• Egocentric Body Tracking: Diffusion and regression models evaluated via mean per-joint position error (MPJPE), hand pose error (Hand PE), velocity, and self-penetration. For example, BoDiffusion (3-point) achieves mean MPJPE 7.98 mm, EgoEgo (1-point) 13.22 mm.
• Generative Motion Priors: VQ-VAE ablations: best config (PQ=2, CB=16384, dim=64) yields MPJPE 34.5 mm, PA-MPJPE 26.8 mm, acceleration 0.67 m/s².
• Motion-to-Text: TM2T and MotionGPT: BERT, BLEU, CIDEr, and ROUGE-L metrics (e.g., MotionGPT: BERT 14.1, BLEU@1 42.2, CIDEr 37.3).
• Device Trajectory Estimation: Absolute Trajectory Error (ATE) per Equation:

$$\mathrm{ATE} = \sqrt{\frac{1}{N}\sum_{i=1}^N \|\mathbf{t}_i^{\text{est}} - \mathbf{t}_i^{\text{gt}}\|^2}$$

• 3D Object Detection: Evaluated via mAP@IoU thresholds (e.g., Cube R-CNN, EFM3D protocol).
• Human Motion Quality: Wrist error reduced from 14.32 cm (Nymeria) to 5.07 cm (NymeriaPlus), self-penetration from