---
title: 'Nymeria: Multimodal Egocentric Dataset'
url: https://www.emergentmind.com/topics/nymeria-dataset
type: topic
---

# Nymeria: Multimodal Egocentric Dataset

Nymeria is a large-scale, multimodal dataset of egocentric human motion captured in natural daily settings, designed to support research in embodied AI, egocentric perception, action recognition, motion-language grounding, and related domains. The dataset features synchronized full-body motion ground truth, multiple egocentric video modalities (including RGB, grayscale, and eye-tracking), inertial and environmental sensors, and observer-perspective recordings, all precisely localized and temporally interaligned in a unified metric 3D frame. Nymeria also includes a comprehensive hierarchical annotation protocol with fine-grained motion narration, atomic actions, and high-level activity summarization in natural language, making it the world's largest motion-language dataset. The subsequent NymeriaPlus extension further enhances the resource with improved parametric motion models, dense 2D/3D scene object labels, instance-level mesh reconstruction, and new multimodal data streams [2406.09905][2603.18496].

## 1. Dataset Scale, Demographics, and Recording Protocol

Nymeria comprises 1,200 sequences (∼15 minutes each), totaling 300 hours of recorded activity, with 264 participants balanced for gender (48.5% female/51.5% male) and spanning seven ethnicities, ages 18–50+, heights 150–200 cm, and weights 40–100 kg. Recordings were performed across 50 diverse real-world locations (homes, cafeteria, office, campus), covering 20 indoor/outdoor scenarios (e.g., cooking, hiking), 15% outdoor, and 15% two-participant sessions. Device trajectory statistics include 399 km traveled by the head-camera and 1,053 km by both wrists. All data capture occurred in natural, unscripted conditions: participants wore motion-capture suits (Xsens MVN Link, 17 IMUs) and head-mounted Project Aria glasses, performing unconstrained everyday tasks without scripts. Data collection emphasizes ecological validity and diversity—activities range from manipulation to locomotion and social interaction [2406.09905][2506.21552].

## 2. Modalities, Synchronization, and Data Formats

Nymeria provides the following synchronized modalities per session:

- **Egocentric Video**: 
  - Project Aria RGB (54M frames, 30 Hz, 1,408×1,408) and grayscale (151.2M frames at 640×480), with additional observer-perspective video when available.
  - Eye-tracking: 10.8M gaze samples (320×240, 10 Hz).
- **Full-body Xsens MVN Motion Capture**: 260M frames at 240 Hz; outputs 23 segment orientations; for PEVA, only upper body (15 joints + 3-DoF root) is used and downsampled to 4 Hz, joints as ΔEuler angles in [–π,π].
- **IMU and Environmental Sensors**: Barometer, magnetometer, device-recorded audio (7x 48 kHz, participant and observer), and wristbands with 1,000 Hz IMU sampling.
- **Device Trajectories**: Time-stamped SE(3) extrinsics (head, wrists), localized via SLAM/BA to a shared metric 3D world.
- **Scene Structure**: NymeriaPlus adds semi-dense 3D point clouds (0.5–3M points/scene), dense 2D/3D bounding boxes, and instance-level object reconstructions (ShapeR pipeline).
- **Wristband Cameras**: Additional RGB and grayscale video streams at 30 Hz (NymeriaPlus).

All sensors are time-synchronized to sub-millisecond accuracy, with global calibration aligning Project Aria and Xsens coordinate frames into a single gravity-upright, metric world space. File formats include .npz/JSON for parametric motion (MHR, SMPL), .ply for 3D shapes, WAV for audio, and custom metadata/pose formats [2406.09905][2603.18496][2506.21552].

## 3. Hierarchical Annotation Protocols and Preprocessing

Annotation in Nymeria proceeds along a three-tiered schema:

- **Motion Narration**: Fine-grained (∼3–5 s) clips with frame-accurate posture-focused language prompts (full-body, arms, lower-body, intent), totaling 38.6 h and 117.2K segments.
- **Atomic Actions**: Verb-centric (∼3–5 s) segments, 207 h and 170.6K annotated actions covering "describe the action, posture, direction, and interactions."
- **Activity Summarization**: Coarse (15–30 s) segments, 196 h and 22.6K summaries.
- **Language Data**: 301.5K sentences, 8.64M words, vocabulary size 6,545 (average 27.8 words/sentence).

Annotation uses custom UIs presenting time-synchronized egocentric and third-person video with 3D rendered motion. Training for annotators and quality control via manual review yield 92% usable XSens data, <1% frame drop (head), <6.3% wrist dropout, and robust action granularity. Atomic action extraction uses thresholded joint-delta vectors [2406.09905][2506.21552].

Raw IMU and pose data are filtered, downsampled, and normalized: translations to [–1,1], rotations to [–π,π]. Control representations are computed as relative pose deltas, $\mathbf{a}_t = \mathbf{p}_t - \mathbf{p}_{t-1}$. PEVA and related methods reshape the sequence as $\mathcal{D} = \{(x_0, a_0), ..., (x_T, a_T)\}$ for Markovian video prediction [2506.21552].

## 4. Object, Scene, and Environment Annotations (NymeriaPlus)

NymeriaPlus extends Nymeria with improved motion representations (MHR, SMPL) and comprehensive 2D/3D scene understanding:

- **Dense 2D Bounding Boxes**: Projected from 3D OBBs; 19 closed-set classes (e.g., bed, chair, wall), >10,000 unique instances; >50,000 visible boxes over sequences.
- **Dense 3D Bounding Boxes**: Each box $B$ is defined by center $c\in\mathbb{R}^3$, extents $(l_x, l_y, l_z)$, rotation $R\in SO(3)$; open-set "Anything" objects labeled ($\sim$12 per recording, 12,000 total, 2,896 dense in 5 venues). Intersection-over-Union (IoU$_3$) metrics and convex polytope algorithms for evaluation.
- **Instance-Level Reconstructions**: Automated via ShapeR, producing metric .ply meshes per object instance, with manual QC (>3 rating).
- **Basemap Scans**: 47 indoor venues, 5–15 min per scan, for static reference and annotation transfer.
- **Additional Modalities**: Wristband videos, extended audio (600+ hours), synchronized IMU, and environmental sensors.

Metadata includes global calibration (camera intrinsics/extrinsics, hand-eye), per-frame device extrinsics, participant anthropometrics, and action/scene descriptors. The dataset is structured for scalability, with directory organization by modality and sequence [2603.18496].

## 5. Benchmark Tasks, Metrics, and Baseline Performance

Nymeria enables benchmarking for a broad suite of embodied and multimodal AI tasks:

- **Egocentric Body Tracking**: Diffusion and regression models evaluated via mean per-joint position error (MPJPE), hand pose error (Hand PE), velocity, and self-penetration. For example, BoDiffusion (3-point) achieves mean MPJPE 7.98 mm, EgoEgo (1-point) 13.22 mm.
- **Generative Motion Priors**: VQ-VAE ablations: best config (PQ=2, CB=16384, dim=64) yields MPJPE 34.5 mm, PA-MPJPE 26.8 mm, acceleration 0.67 m/s².
- **Motion-to-Text**: TM2T and MotionGPT: BERT, BLEU, CIDEr, and ROUGE-L metrics (e.g., MotionGPT: BERT 14.1, BLEU@1 42.2, CIDEr 37.3).
- **Device Trajectory Estimation**: Absolute Trajectory Error (ATE) per Equation:
  $$
  \mathrm{ATE} = \sqrt{\frac{1}{N}\sum_{i=1}^N \|\mathbf{t}_i^{\text{est}} - \mathbf{t}_i^{\text{gt}}\|^2}
  $$
- **3D Object Detection**: Evaluated via mAP@IoU thresholds (e.g., Cube R-CNN, EFM3D protocol).
- **Human Motion Quality**: Wrist error reduced from 14.32 cm (Nymeria) to 5.07 cm (NymeriaPlus), self-penetration from

Source: https://www.emergentmind.com/topics/nymeria-dataset