---
title: OmniMoCap-X Unified MoCap Dataset
url: https://www.emergentmind.com/topics/omnimocap-x
type: topic
---

# OmniMoCap-X Unified MoCap Dataset

OmniMoCap-X is a unified, large-scale multimodal motion-capture (MoCap) dataset, constructed to enable high-quality training and evaluation of generative models for whole-body human motion, supporting ten diverse conditioned and unconditioned motion tasks. It integrates and standardizes 28 publicly available MoCap sources spanning six modality groups, employs the SMPL-X mesh as a representation backbone, and augments all sequences with hierarchical captions via automated GPT-4o video annotation. OmniMoCap-X provides the data and annotation basis for state-of-the-art multimodal frameworks such as OmniMotion-X, enabling research in text-to-motion, music-to-dance, speech-driven gesture, and context-conditional synthesis tasks [2510.19789].

## 1. Dataset Integration and Composition

OmniMoCap-X consolidates 64.3 million frames (amounting to 286.2 hours at 30 fps) from 28 datasets, systematically grouped by primary modality:

- **Text-to-Motion (T2M; text-conditioned):** Mixamo, KIT, OMOMO, IDEA400, 100Style, HumanML3D
- **Music-to-Dance (M2D; music-conditioned):** Choreomaster, FineDance, PhantomDance, AIST++, Motorica, AIOZ
- **Speech-to-Gesture (S2G; speech-conditioned):** BEAT2
- **Human-Human Interaction (HHI; paired motion):** HumanSC3D, InterHuman, Inter-x
- **Human-Object Interaction (HOI; trajectory/object-conditioned):** Arctic, TACO, Fit3D, Behave, Chairs, HOI-M3, Oaklnkv2, NeuralDome
- **Human-Scene Interaction (HSI; scene/contact-conditioned):** EMDB, Rich, LaFAN1, Trumans, Circle

Each MoCap source is documented with framecount, estimated capture hours, and capture modality. Motion data is segmented into uniform 5-second (150-frame) clips, yielding approximately 428,667 motion sequences. Original modalities encompass marker-based systems, Vicon (optical MoCap), IMU, multi-view RGB, and single-view RGB; all are retargeted to a standardized skeletal representation.

### Motion Source Summary

| Modality Group | Example Sources    | Total Frames (M) | Hour Count  |
|:--------------:|:------------------|:----------------:|:------------|
| T2M            | HumanML3D, KIT    | 36.8             | 114.3       |
| M2D            | AIOZ, Motorica    | 12.3             | 96.8        |
| S2G            | BEAT2             | 6.9              | 64.2        |
| HHI            | Inter-x           | 20.9             | 59.4        |
| HOI            | Oaklnkv2, HOI-M3  | 10.1             | 75.9+       |
| HSI            | Trumans, Circle   | 6.7              | 26.9        |

## 2. Data Standardization and Motion Representation

Each motion sequence is represented in the unified SMPL-X format, which defines a skeleton topology of 127 joints—including body, hands, and facial landmarks—extracted from the SMPL-X mesh.

- **Parametrization:**
  - 53 joints use 6D rotation encodings.
  - 74 "free" joints are described by positions and velocities.
  - Mesh parameters: global root orientation, body pose (63-DoF), left/right hand pose (15 DoF each), jaw (3 DoF), facial expression embedding (100D FLAME code), and 10 shape coefficients.

Each frame $i$ is described by a pose vector
$$
p_i = (\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, j^p, j^r, j^v, c^f, f)
$$
where
- $\dot{r}^a$ (scalar): root angular velocity (Y-axis)
- $\dot{r}^x, \dot{r}^z$ (2D): root linear velocity (XZ-plane)
- $r^y$ (scalar): root height above ground
- $j^p \in \mathbb{R}^{3N-1}$: local joint positions ($N=127$)
- $j^r \in \mathbb{R}^{6N'}$: 6D joint rotations ($N'=53$)
- $j^v \in \mathbb{R}^{3N}$: joint velocities
- $c^f \in \{0,1\}^2$: binary foot-contact (heels/toes)
- $f \in \mathbb{R}^{100}$: FLAME facial code

**Format conversion** is performed via MotionBuilder for BVH/FBX (Euler-to-axis-angle-to-6D), and direct parameter mapping from SMPL/SMPL-H to SMPL-X. All data is spatially scaled to a common metric (meters) and temporally resampled to 30 fps after rotational/translation Gaussian smoothing. Initial root orientation is aligned to face +Z, +Y up, and grounding ensures feet touch the ground at $t=0$.

## 3. Hierarchical Video-Based Annotation and Captioning

Motion sequences are rendered as SMPL-X meshes using Blender, with a fixed 3/4 viewpoint, neutral lighting, and minimal motion blur to facilitate downstream annotation. Videos are generated at 30 fps to preserve fine kinematic details.

Annotation employs hierarchical captioning via GPT-4o:

- **Inputs:** Rendered motion video, legacy action labels, and task tags where present.
- **Prompt:** Two-stage: (1) Frame-indexed, low-level explicit action descriptions (e.g., joint angles, displacements); (2) high-level semantic summaries (e.g., object interactions).
- **Joint Encoding:** Video frames and prompt are jointly processed by GPT-4o for multimodal understanding.
- **Output:** Structured JSON with timestamps, sub-actions, and global activity categories.

Metrics for annotation quality include an average caption length of 276.8 words and Type-Token Ratio spanning 0.139–1.000, reflecting diverse lexical coverage. The verb distribution (top-10 subset tracking) is monitored for modality relevance, and 5% of captions are manually spot-checked for motion fidelity.

## 4. Task Taxonomy and Evaluation Protocols

OmniMoCap-X supports ten canonical and composable motion-generation and prediction tasks:

1. Text-to-Motion (T2M)
2. Music-to-Dance (M2D)
3. Speech-to-Gesture (S2G)
4. Human-Object Interaction (HOI)
5. Human-Scene Interaction (HSI)
6. Human-Human Interaction (HHI)
7. Motion Prediction (temporal extrapolation)
8. In-Betweening (temporal interpolation)
9. Joint-Guided Synthesis (sparse/dense joint constraint)
10. Trajectory-Guided Synthesis (path or root-trajectory constraint)

**Dataset splits:** Each task is defined over 150-frame clips. Training comprises approximately 95% of the dataset; validation is a 5% random hold-out per source. Test splits use uniform sampling (e.g., 10 sequences per source for T2M and GSTC, or official splits of constituent datasets for M2D and S2G). For multi-condition protocols, specific control segments may be masked.

| Task Group    | Example Test Protocol         | Details         |
|---------------|------------------------------|-----------------|
| T2M & GSTC    | 10 sequences per source      | ≈280 samples    |
| M2D           | Official splits (AIST++, …)  | Source-specific |
| S2G           | Official BEAT2 test set      | Source-specific |

A plausible implication is that the broad task coverage enables multi-task and cross-modal benchmarking previously infeasible with individual datasets.

## 5. Licensing and Access

OmniMoCap-X is publicly released, including both SMPL-X formatted motion data and GPT-4o-generated captions, at [https://github.com/GuoweiXu368/OmniMocap-X](https://github.com/GuoweiXu368/OmniMocap-X). Redistributed data are governed by the original licensing conditions of each constituent MoCap dataset, including possible academic-use restrictions for some proprietary sources. No further limitations are imposed beyond those of the providers.

## 6. Research Impact and Context

OmniMoCap-X constitutes the largest unified multimodal motion dataset available to date, explicitly designed for the training and evaluation of models such as OmniMotion-X, which adopt autoregressive diffusion transformer architectures in sequence-to-sequence mapping for realistic motion synthesis [2510.19789]. Its scale, heterogeneity, and standardized representation facilitate cross-modal, conditional, and controllable animation generation. The GPT-4o-based hierarchical annotation pipeline offers rich, structured supervision for tasks ranging from low-level actuation to high-level semantic interpretation.

This enables state-of-the-art performance in interactive, controllable, and coherent long-duration motion synthesis tasks, and supports emerging research on sequence modeling, joint/trajectory-guided synthesis, and multimodal conditional animation. A plausible implication is that OmniMoCap-X will accelerate advances in both generative modeling and benchmark evaluation within human motion research, as well as support applications in animation, robotics, and virtual environment simulation.

Source: https://www.emergentmind.com/topics/omnimocap-x