Embody 3D: Multimodal 3D Motion Dataset
- Embody 3D is a large-scale, multimodal 3D motion and behavior dataset featuring 500 hours of SMPL-X tracked motion, hand tracking, per-participant audio, and detailed text annotations.
- The dataset unifies both prompted motion capture and multi-person social interactions, enabling comprehensive modeling of diverse human behaviors in a consistent 3D world space.
- Its advanced capture system uses 80 high-resolution cameras with precise calibration and synchronization, ensuring high-accuracy tracking and multimodal data alignment.
Searching arXiv for the specified paper to ground the article in the latest indexed record. Embody 3D is a large-scale, high-quality 3D motion and behavior dataset created by the Codec Avatars Lab at Meta. It comprises 500 individual hours of 3D motion data from 439 participants, amounting to over 54 million frames of tracked 3D motion, and it provides tightly synchronized motion, per-participant audio, and text annotations in a common 3D world space (McLean et al., 17 Oct 2025). The dataset was designed to address a persistent trade-off in human motion corpora: large 2D video collections offer scale but lack accurate 3D tracking and consistency, whereas existing 3D datasets provide precision but are typically small and often omit hand tracking, body shape, audio, text, or multi-person interactions. Embody 3D therefore targets broad behavior coverage across single-person motions and multi-person social settings while maintaining full-body SMPL-X tracking, hand tracking, and person-specific body shape.
1. Dataset scope and design objectives
Embody 3D was introduced as a multimodal corpus of human motion and behavior rather than as a narrowly scoped action dataset. Its stated goals are to combine scale with completeness, to cover broad behavior domains, and to provide multimodal data tightly synchronized in a common 3D world space. In practical terms, this means full-body motion including hands, person-specific body shape, per-participant speech-separated audio, and text annotations spanning prompts, scene-level descriptions, per-person pose and motion descriptions, and emotion labels (McLean et al., 17 Oct 2025).
The dataset is organized around two broad behavioral regimes. The first is prompted motion capture, including charades, hand-focused interactions, and locomotion. The second is social and behavioral capture in multi-person settings, including dyadic conversations, multi-person conversations, collaborative and competitive scenarios, and co-living situations in an apartment-like space. This structure is central to the dataset’s positioning: it unifies domains that are often siloed, notably locomotion, conversational gesture, and socially situated activity.
A common misconception is to treat Embody 3D as merely a larger motion-capture collection. The paper instead frames it as a behavior dataset with multimodal alignment. The floor-centric world coordinate system consistent across sessions, the per-participant audio separation, and the scenario-level annotation regime indicate that the dataset is intended for problems in social interaction modeling and multimodal generation, not only for isolated pose estimation.
2. Scale, composition, and behavioral coverage
The corpus totals 500 individual hours of 3D motion data. The paper explicitly notes that individual hours count person-hours, so one hour of two-person conversation equals two individual hours. The participant pool contains 439 individuals, and the release covers over 54 million 3D motion frames (McLean et al., 17 Oct 2025).
The subcategory breakdown is as follows:
| Subcategory | Hours / participants | Notes |
|---|---|---|
| Charades | 88.9 h / 221 | 15-second prompted motion segments; text prompts; no audio |
| Hand Interactions | 111.3 h / 137 | Hand and arm motions with self-contact; no audio |
| Locomotion | 21.0 h / 46 | Walking, running, jumping; high-level text; no audio |
| Dyadic Conversations | 59.4 h / 86 | Guided topics and emotions plus free-form conversation; audio |
| Multi-person Conversations | 125.2 h / 210 | Furniture interactions included; audio; no fine-grained text |
| Scenarios | 49.2 h / 77 | Collaborative/competitive activities; audio and fine-grained text |
| Day in the Life | 46.4 h / 77 | 3–4 participants in a small apartment-like setup; audio and fine-grained text |
The behavioral coverage is unusually heterogeneous for a 3D dataset. Charades contributes 15-second prompted segments with explicit text prompts. Hand Interactions emphasizes hand and arm motions with self-contact. Locomotion provides instructed movement styles such as walking, running, and jumping. Dyadic Conversations combines guided topics and emotional states with free-form discussion. Multi-person Conversations includes furniture interactions with chairs, a high-table, and a couch. Scenarios adds collaborative and competitive activities such as games and assembling furniture. “Day in the Life” extends this further to co-living, hosting, and group activity goals in a small apartment-like setup.
This composition suggests that Embody 3D is not optimized for a single benchmark task. A plausible implication is that the dataset is better understood as a substrate for cross-domain multimodal modeling, where prompted kinematics, hand-centric behavior, and socially grounded interaction are jointly represented in a consistent 3D frame.
3. Modalities and annotation schema
The motion representation is released in SMPL-X (Expressive Body Capture) format, with full body tracking including hands and person-specific body shape parameters estimated from calibration poses (McLean et al., 17 Oct 2025). The paper states that the SMPL-X skeleton is used, but it does not list joint counts and does not claim that face expression parameters are part of the release. During processing, 308 2D keypoints for face and body are obtained via Sapiens-1B, triangulated to 3D, and fit to SMPL-X.
The audio modality is captured with five custom MEMS microphone arrays, each containing 128 elements in a spherical arrangement, for a combined 640 channels that effectively record 10th-order ambisonics. The release includes a non-separated reference channel from a central microphone and a separated speech channel for each participant in each segment via beamforming. Timestamp encoding is embedded in audio files, audio is aligned to camera frame timestamps, and frame drops are replaced with zeros. The sampling rate and bit depth are not specified in the paper.
Text annotations vary substantially by subcategory. Charades provides per-segment prompts such as “jumping” or “shooting an arrow.” Dyadic conversations include prompt-driven emotions such as anger, happiness, and sadness, together with topics and free-form segments. Scenarios and Day in the Life provide fine-grained, human-generated annotations with scene-level descriptions, detailed per-person pose and motion notes, and per-participant emotion labels inferred from face, pose, and speech. Multi-person Conversations provides audio but not fine-grained text. Locomotion provides motion-type labels at a high level. The paper describes segment-level prompts and synchronized multimodal capture, but exact alignment schemas are not detailed beyond segment and per-person labeling.
The release scope is narrower than the acquisition apparatus might imply. Although the dataset is acquired with 80 high-resolution cameras, the paper specifies releasing tracked motion, audio, and text, and it does not state that raw images or depth streams are included. This is an important interpretive point because the dataset is a processed multimodal corpus rather than an unrestricted sensor dump.
4. Capture environment, calibration, and reconstruction pipeline
The primary capture environment is a multi-camera stage measuring 6 m × 6 m × 3.6 m high, with a capture area of 3.6 m × 3.6 m (McLean et al., 17 Oct 2025). The visual system comprises 80 global-shutter machine vision cameras at 24.47 MP resolution (5320 × 4600) and 30 fps. Of these, 64 are body-tracking cameras with 8–15 mm F4 EF lenses and 4 ms exposure, and 16 are face-tracking cameras with 35 mm F1.4 EF lenses and 2 ms exposure. Illumination is provided by 14 LED panels evenly distributed across the setup, with average illuminance of approximately 650 lux.
Calibration and synchronization are central technical features of the dataset. Custom fiducial rigs are used for intrinsic and extrinsic calibration, and floor fiducials are used to estimate the floor plane. Reported calibration quality is p50 reprojection error below 0.2 px and p99 around 0.8 px. All cameras are co-triggered, post-hoc timestamp clustering is used to build a global frame list, and out-of-sync frames are filtered. Microphone arrays are geometrically aligned to the camera system using fiducials. The resulting coordinate frame is floor-centric and consistent across sessions.
The processing pipeline proceeds from global camera-audio synchronization through geometric calibration, participant shape estimation, keypoint detection, identity matching, triangulation, pose tracking, beamforming, and QA. Participant shape estimation uses four calibration poses—A, T, C, and T-Rex—with dense 2D keypoints to optimize SMPL-X shape coefficients. Multi-person keypoint detection begins with person bounding boxes per camera, after which Sapiens-1B produces 308 2D keypoints for face and body. Identity matching uses spatio-temporal clustering with ray bundles and face embeddings matched to reference face images via the Hungarian algorithm. Keypoint triangulation is performed with RANSAC across camera pairs with reprojection-error refinement, temporal smoothness, and bone-length constraints. Pose tracking uses a pose encoder that maps shape and 3D keypoints to joint rotations, aligns the torso via Procrustes, is regularized by a pose prior, and is trained to minimize joint-keypoint distances. Beamforming then separates per-participant speech, with hyperparameters tuned using human annotations for noise, bleed, and distortion.
Quality assurance is performed by overlaying tracked motion on camera views and assigning 1–5 Likert ratings. Segments with average scores below 2.5 are discarded. The paper does not report inter-annotator agreement.
5. Research uses and comparative position
The paper identifies several research uses enabled by the dataset: speech-to-gesture and conversational motion synthesis with per-participant audio, multi-person interaction modeling in a consistent 3D world frame, motion understanding across hands, body, and social behaviors, hand-object and furniture interaction modeling, and motion reconstruction and retargeting in SMPL-X with body shape (McLean et al., 17 Oct 2025). Because audio, motion, and text are synchronized and aligned in a floor-centric world frame, the dataset is also suited to studies of audio-motion alignment and social behavior recognition.
Embody 3D is positioned against both large 2D motion datasets and prior 3D motion datasets. Relative to large 2D collections such as MotionMillion, Motion-X, and Seamless Interaction, it provides high-fidelity 3D tracking in a consistent world frame, thereby avoiding depth ambiguity and occlusion issues inherent to monocular reconstruction. Relative to existing 3D datasets such as KIT-ML, BABEL, BEAT, Talking with Hands, and AIST++, it is presented as substantially larger and more complete, with 500 individual hours, 439 participants, SMPL-X full-body tracking including hands, person-specific body shape, per-person audio via beamforming, and diverse multi-person social scenarios.
The dataset’s comparative significance lies in its integration strategy. It does not merely scale one axis of annotation or one motion domain. Instead, it combines prompted motion, conversational interaction, furniture-rich social settings, and co-living scenarios within a single capture and coordinate regime. This suggests a shift from narrowly task-specific corpora toward multimodal behavioral corpora that can support unified models of motion, speech, and interaction.
6. Limitations, access, and ethical considerations
Several constraints are explicit. Modalities and annotations vary by subcategory: Charades, Hand Interactions, and Locomotion do not provide audio; fine-grained text annotations are provided only for Scenarios and Day in the Life; Dyadic Conversations and Locomotion provide high-level labels; and Multi-person Conversations lacks fine-grained text (McLean et al., 17 Oct 2025). The paper does not claim face expression parameters in the release, even though face cameras and keypoints are used during processing and identity matching.
The paper also omits several items that are often expected in benchmark-oriented datasets. It does not define train, validation, or test splits, and it does not provide recommended partitions. It does not present baseline models or quantitative benchmarks. It does not define evaluation metric formulas, beyond noting that tracking quality is assessed through human Likert ratings with a discard threshold at 2.5. Specific file formats such as FBX, BVH, or NPZ, dataset units such as meters, and directory structures are not detailed in the paper. Licensing and usage restrictions are not specified, and the application process or prerequisites for access are not described.
The ethical information is similarly bounded. Participants were informed about research use and signed consent, sessions were supervised by research assistants, and QA procedures were conducted. However, participant demographics, licensing terms, and broader ethical review details beyond consent are not reported. The capture environment is an indoor stage with controlled lighting and acoustics, and generalization to outdoor or unconstrained settings is not discussed.
Access information is limited to the dataset landing page and bibliographic identification. The paper lists the website at https://www.meta.com/emerging-tech/codec-avatars/embody-3d and attributes the resource to Claire McLean et al. at the Codec Avatars Lab, Meta. In this form, Embody 3D stands as a large-scale multimodal research dataset whose technical contribution lies in synchronized SMPL-X motion, per-participant audio, and structured text across single-person and socially interactive settings, while leaving benchmarking conventions, release formalities, and some documentation details open.