Nymeria: Multimodal Egocentric Human Motion Dataset

• Nymeria is a comprehensive egocentric dataset capturing realistic human motion with multimodal sensor data including inertial measurements, RGB video, and full-body kinematics.
• It integrates synchronized streams from head, wrist, and full-body devices to provide precise spatial registration and temporal alignment across varied daily activity scenarios.
• Its rich annotations combine hierarchical motion-language descriptions with gaze tracking, enabling advanced benchmarks for egocentric body tracking, motion synthesis, odometry, and behavior interpretation.

Searching arXiv for the cited Nymeria papers to ground the article in current preprints. Nymeria is a large-scale multimodal egocentric dataset for in-the-wild human motion, designed to couple first-person sensing with full-body kinematic ground truth, synchronized device trajectories, scene geometry, and hierarchical language descriptions. Introduced by Ma et al. in 2024, it comprises 1,200 sessions totaling 300 hours across 50 locations and roughly 399 km of head translation, and it has subsequently become a benchmark for egocentric body tracking, motion synthesis, motion-language modeling, inertial odometry, controllable video generation, and gaze-aware behavior interpretation (Ma et al., 2024, Li et al., 2 Jun 2026, Gu et al., 25 May 2026, Chang et al., 20 Nov 2025).

1. Corpus scope and dataset identity

Nymeria was created to capture daily human activity under realistic first-person dynamics rather than laboratory-constrained motion capture. The original release reports 1,200 sessions, each approximately 15 minutes long, from 264 unique subjects, spanning 50 locations and 20 scenarios, with recordings covering cooking, hiking, sports, party decorating, and other daily activities. It also reports demographic breadth across sex, self-reported ethnicity, age, height, and weight, together with over 1,053 km of wrist trajectories and a motion-language corpus exceeding 8.64 million words (Ma et al., 2024).

Subsequent papers preserve this overall characterization while differing in some bookkeeping details. The NymeriaPlus overview describes the original Nymeria collection as over 300 hours from 256 unique participants across 50 distinct locations and 20 unscripted activity scenarios, while MARIO describes Nymeria as 5x larger than datasets used in prior inertial-odometry work and summarizes 264 unique human subjects, 1,200 continuous sequences, and approximately 400 km of translation. Source papers also alternate between describing the activity taxonomy as “20 scripted scenarios,” “20 unscripted activity scenarios,” and “20 activity categories.” This suggests that the corpus identity is stable at the level of scale, sensing, and task coverage, while some later summaries compress or reframe participant and scenario counts for downstream benchmarking (DeTone et al., 19 Mar 2026, Li et al., 2 Jun 2026).

The original release characterizes Nymeria as the world’s largest collection of human motion in the wild, the first of its kind to provide synchronized and localized multi-device multimodal egocentric data, and the world’s largest motion-language dataset. Within the literature that uses it, Nymeria is therefore treated less as a single-task benchmark than as a general-purpose infrastructure for human-centric AI (Ma et al., 2024).

2. Wearable capture stack and sensing modalities

Nymeria’s acquisition protocol combines head-mounted, wrist-mounted, full-body, and observer-view sensing in a shared metric 3D world. The core platforms are Project Aria smart glasses, two miniAria wristbands, an Xsens MVN-Link inertial motion-capture suit, and an additional observer wearing Aria glasses. Later Nymeria-based work, especially MARIO, exploits the fact that the head-worn platform already exposes multiple lightweight sensors beyond the primary IMU, including a secondary IMU, a barometer, and a magnetometer (Ma et al., 2024, Li et al., 2 Jun 2026).

Platform	Modalities	Representative specs
Project Aria headset	RGB, grayscale, eye-tracking video, IMUs, magnetometer, barometer, audio	30 fps RGB $1408 \times 1408$; 30 fps grayscale $640 \times 480$; 10 fps eye-tracking $320 \times 240$; 1 kHz and 800 Hz IMUs; 10 Hz magnetometer; 50 Hz barometer; 7-channel 48 kHz audio
miniAria wristbands	Cameras and IMUs	Same cameras and IMUs as Aria except no microphones, barometer, magnetometer, or ET cameras; IMU ranges doubled for fast wrist motion
Xsens MVN-Link suit	Full-body inertial MoCap	17 inertial trackers plus magnetometer; 240 Hz full-body pose
Observer Aria	Third-person perspective	Moving external view synchronized with egocentric streams

This sensor stack is unusually dense for an egocentric corpus. In the original release it supports multimodal streams from head and wrists, full-body pose, gaze, synchronized third-person video, and SLAM-derived geometry. In MARIO it additionally serves as a platform for camera-less inertial odometry, where Project Aria’s primary IMU, secondary IMU, barometer, and magnetometer are fused with an IMU-inferred body-pose prior. In E$^3$C it underlies egocentric video generation with context RGB frames, headset trajectories, wrist motion, ego-body joints, and semi-dense point-cloud memory (Ma et al., 2024, Li et al., 2 Jun 2026, Gu et al., 25 May 2026).

3. Synchronization, localization, and motion representation

A defining property of Nymeria is that all wearable streams are jointly time-synchronized and spatially registered into a single metric world. In the original dataset, a hardware time-sync device feeds a common clock to Aria glasses, miniAria wristbands, and the Xsens suit, with drift reported as less than 4.2 ms per session; all streams are time-stamped and hardware-synchronized to sub-millisecond accuracy. Project Aria Machine Perception Service then runs visual-inertial odometry and SLAM on each device and jointly optimizes all sessions at a location into a gravity-aligned metric world frame at 1 kHz (Ma et al., 2024).

The alignment between the Xsens odometry frame $O$ and the Aria world frame $W$ is posed as a hand-eye calibration problem. Let $T_{OH}^t$ denote the Xsens head-to-odometry pose at time $t$, and $T_{WD}^t$ the Aria device-to-world pose. Nymeria solves for a constant rigid transform $T_{HD}$ by minimizing

$640 \times 480$0

which is described as a closed-form hand-eye problem solved via SVD. Full-body poses are then retargeted to an anatomically inspired human model through the inverse-kinematics objective

$640 \times 480$1

where $640 \times 480$2 are global shape parameters, $640 \times 480$3 are joint angles, $640 \times 480$4 are local landmark offsets, and $640 \times 480$5 are Xsens landmarks in world coordinates (Ma et al., 2024).

NymeriaPlus strengthens this representation layer by introducing improved human motion in Momentum Human Rig and SMPL formats. Its joint XSens-Aria optimization refines an XSens-retargeted initialization $640 \times 480$6 by minimizing

$640 \times 480$7

with $640 \times 480$8 for robust reprojection of head and wrist joints to Aria trajectories, $640 \times 480$9 for deviation from the initial XSens pose, $320 \times 240$0 for parameter limits, $320 \times 240$1 for temporal smoothness of the global root, $320 \times 240$2 for gravity-axis alignment, and $320 \times 240$3 for constant foot position on contact. The solver is Gauss-Newton over 2,000-frame batches, and the reported improvements are wrist distance error from 14.32 cm to 5.07 cm, body self-penetration error from 18.67 units to 2.44 units, and foot-sliding frames from 35% to 9.81% (DeTone et al., 19 Mar 2026).

4. Hierarchical semantic annotation and multimodal supervision

Nymeria is not limited to kinematic supervision. Its annotation protocol introduces a hierarchy of motion-language descriptions aligned to synchronized sensor streams and scene context. Trained annotators view egocentric video, third-person video, and 3D scene renderings, then produce three semantic levels: fine-grained motion narration over 3–5 s clips, atomic actions over 3–5 s clips, and high-level activity summaries over 15–30 s clips. The original release reports 38.6 hours of motion narration, 207 hours of atomic-action annotation, and 196 hours of activity summaries (Ma et al., 2024).

The corresponding text corpus is large. The original paper reports 310,500 sentences, 8.64 million words, and a vocabulary size of 6,545. By task, it reports 117,200 motion-narration sentences, 170,600 atomic-action sentences, and 22,600 activity-summary sentences. Example atomic actions include “C lifts his right leg to prepare for a kick” and “C kicks the soccer ball with his left foot,” while activity summaries include “C bikes on the road with a colleague.” These labels make Nymeria simultaneously a motion dataset and a language-grounding resource (Ma et al., 2024).

The dataset also exposes gaze streams from eye tracking, which later work elevates from raw sensor modality to semantic signal. GazeInterpreter uses Nymeria as a benchmark with 236 manually annotated segments, continuous eye-gaze signals sampled at up to 120 Hz, and body-motion prompts. It divides these segments into approximately 120 low-level sequences of 2–10 s and approximately 116 high-level sequences of 8–30 s, and studies whether symbolic gaze parsing can improve eye-body-coordinated narrations for downstream text-driven motion generation, action anticipation, and behavior summarization. A plausible implication is that Nymeria’s design supports not only multimodal observation, but also multimodal semantic recomposition, in which gaze, motion, and language are jointly modeled (Chang et al., 20 Nov 2025).

5. Benchmark structure and representative quantitative results

Nymeria’s importance is closely tied to the diversity of evaluation protocols it supports. The original release reports baselines for egocentric body tracking, generative motion priors, and motion understanding; later papers add camera-less inertial odometry, controllable egocentric video generation, and gaze-informed narration. These benchmarks are heterogeneous, but they share Nymeria’s central premise: accurate modeling of embodied behavior requires synchronized access to motion, sensor, and scene signals rather than isolated RGB clips (Ma et al., 2024).

For inertial odometry, MARIO defines three standard error families. With ground-truth position $320 \times 240$4 and estimate $320 \times 240$5 after time and rigid alignment in a common world frame,

$320 \times 240$6

with horizontal and vertical ATE obtained by projection into the $320 \times 240$7 and $320 \times 240$8 axes. Relative translational error over a window $320 \times 240$9 is

$^3$0

typically for $^3$1 s or $^3$2 s, and drift is

$^3$3

On the Nymeria test split, MARIO evaluates AirIO, TLIO, EqNIO, and RoNIN-LSTM in Base, +Pose, and +All variants. Averaged across the four backbones, the full +All configuration yields approximately 36% drift reduction over Base, with up to 44% in TLIO and ATE reduction up to 41%. The TLIO ablation reports ATE decreasing from 10.19 m to 7.97 m with PoseNet alone and to 5.73 m with full fusion, while drift decreases from 6.46% to 4.94% and then to 3.64% (Li et al., 2 Jun 2026).

For controllable video generation, E$^3$4C frames each example as a rollout simulator: from $^3$5 context frames and $^3$6 target frames, with future camera poses and ego/exo pose controls, the model must synthesize the next 8 s of first-person video. The reported metrics include visual fidelity (FVD, LPIPS, PSNR, SSIM), camera-motion adherence via translation and rotation error after Sim(3) alignment, object consistency (Obj-F1, Obj-mIoU), exo-human control (Exo-F1, PCK@10%), and ego-hand control (Hand-F1, Hand-mIoU). On 100 held-out Nymeria snippets, E$^3$7C improves over a VACE baseline from FVD 328 to 249, LPIPS 0.440 to 0.415, PSNR 17.3 to 18.6, SSIM 0.604 to 0.629, translation error 5.41 cm to 2.40 cm, rotation error 24.5° to 23.3°, Obj-F1 47.95 to 52.85, Exo-F1 36.26 to 39.59, and Hand-mIoU 18.17 to 24.76 (Gu et al., 25 May 2026).

For text-conditioned motion and behavior understanding, the original Nymeria paper reports egocentric body-tracking baselines such as AvatarPoser at MPJPE 7.97 mm on real Aria+miniAria data, BoDiffusion at MPJPE 7.98 mm and FID 2.32, and EgoEgo at MPJPE 13.22 mm and FID 5.14. It also reports a generative motion-prior configuration with product quantization $^3$8, codebook size $^3$9, and latent dimension $O$0, giving MPJPE 34.5 mm, PA-MPJPE 26.8 mm, and acceleration error 0.67 mm/sec$O$1. For motion-to-text, TM2T yields BERTScore 11.1 and CIDEr 20.9, while MotionGPT yields BERTScore 14.1 and CIDEr 37.3 (Ma et al., 2024).

GazeInterpreter repurposes Nymeria’s