OmniMoCap-X Unified MoCap Dataset

• OmniMoCap-X is a unified multimodal motion-capture dataset that standardizes data from 28 sources and six modality groups for entire-body motion analysis.
• It segments 64.3 million frames into uniform 150-frame clips using the SMPL-X mesh, ensuring consistent representation and evaluation for generative models.
• The dataset features hierarchical GPT-4o video annotations and supports ten motion tasks, enabling research in text-to-motion, music-to-dance, and more.

OmniMoCap-X is a unified, large-scale multimodal motion-capture (MoCap) dataset, constructed to enable high-quality training and evaluation of generative models for whole-body human motion, supporting ten diverse conditioned and unconditioned motion tasks. It integrates and standardizes 28 publicly available MoCap sources spanning six modality groups, employs the SMPL-X mesh as a representation backbone, and augments all sequences with hierarchical captions via automated GPT-4o video annotation. OmniMoCap-X provides the data and annotation basis for state-of-the-art multimodal frameworks such as OmniMotion-X, enabling research in text-to-motion, music-to-dance, speech-driven gesture, and context-conditional synthesis tasks (Xu et al., 22 Oct 2025).

1. Dataset Integration and Composition

OmniMoCap-X consolidates 64.3 million frames (amounting to 286.2 hours at 30 fps) from 28 datasets, systematically grouped by primary modality:

• Text-to-Motion (T2M; text-conditioned): Mixamo, KIT, OMOMO, IDEA400, 100Style, HumanML3D
• Music-to-Dance (M2D; music-conditioned): Choreomaster, FineDance, PhantomDance, AIST++, Motorica, AIOZ
• Speech-to-Gesture (S2G; speech-conditioned): BEAT2
• Human-Human Interaction (HHI; paired motion): HumanSC3D, InterHuman, Inter-x
• Human-Object Interaction (HOI; trajectory/object-conditioned): Arctic, TACO, Fit3D, Behave, Chairs, HOI-M3, Oaklnkv2, NeuralDome
• Human-Scene Interaction (HSI; scene/contact-conditioned): EMDB, Rich, LaFAN1, Trumans, Circle

Each MoCap source is documented with framecount, estimated capture hours, and capture modality. Motion data is segmented into uniform 5-second (150-frame) clips, yielding approximately 428,667 motion sequences. Original modalities encompass marker-based systems, Vicon (optical MoCap), IMU, multi-view RGB, and single-view RGB; all are retargeted to a standardized skeletal representation.

Motion Source Summary

Modality Group	Example Sources	Total Frames (M)	Hour Count
T2M	HumanML3D, KIT	36.8	114.3
M2D	AIOZ, Motorica	12.3	96.8
S2G	BEAT2	6.9	64.2
HHI	Inter-x	20.9	59.4
HOI	Oaklnkv2, HOI-M3	10.1	75.9+
HSI	Trumans, Circle	6.7	26.9

2. Data Standardization and Motion Representation

Each motion sequence is represented in the unified SMPL-X format, which defines a skeleton topology of 127 joints—including body, hands, and facial landmarks—extracted from the SMPL-X mesh.

• Parametrization:
  • 53 joints use 6D rotation encodings.
  • 74 "free" joints are described by positions and velocities.
  • Mesh parameters: global root orientation, body pose (63-DoF), left/right hand pose (15 DoF each), jaw (3 DoF), facial expression embedding (100D FLAME code), and 10 shape coefficients.

Each frame $i$ is described by a pose vector

$$p_i = (\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, j^p, j^r, j^v, c^f, f)$$

where

• $\dot{r}^a$ (scalar): root angular velocity (Y-axis)
• $\dot{r}^x, \dot{r}^z$ (2D): root linear velocity (XZ-plane)
• $r^y$ (scalar): root height above ground
• $j^p \in \mathbb{R}^{3N-1}$: local joint positions ($N=127$)
• $j^r \in \mathbb{R}^{6N'}$: 6D joint rotations ($N'=53$)
• $j^v \in \mathbb{R}^{3N}$: joint velocities
• $$p_i = (\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, j^p, j^r, j^v, c^f, f)$$0: binary foot-contact (heels/toes)
• $$p_i = (\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, j^p, j^r, j^v, c^f, f)$$1: FLAME facial code

Format conversion is performed via MotionBuilder for BVH/FBX (Euler-to-axis-angle-to-6D), and direct parameter mapping from SMPL/SMPL-H to SMPL-X. All data is spatially scaled to a common metric (meters) and temporally resampled to 30 fps after rotational/translation Gaussian smoothing. Initial root orientation is aligned to face +Z, +Y up, and grounding ensures feet touch the ground at $$p_i = (\dot{r}^a, \dot{r}^x, \dot{r}^z, r^y, j^p, j^r, j^v, c^f, f)$$2.

3. Hierarchical Video-Based Annotation and Captioning

Motion sequences are rendered as SMPL-X meshes using Blender, with a fixed 3/4 viewpoint, neutral lighting, and minimal motion blur to facilitate downstream annotation. Videos are generated at 30 fps to preserve fine kinematic details.

Annotation employs hierarchical captioning via GPT-4o:

• Inputs: Rendered motion video, legacy action labels, and task tags where present.
• Prompt: Two-stage: (1) Frame-indexed, low-level explicit action descriptions (e.g., joint angles, displacements); (2) high-level semantic summaries (e.g., object interactions).
• Joint Encoding: Video frames and prompt are jointly processed by GPT-4o for multimodal understanding.
• Output: Structured JSON with timestamps, sub-actions, and global activity categories.

Metrics for annotation quality include an average caption length of 276.8 words and Type-Token Ratio spanning 0.139–1.000, reflecting diverse lexical coverage. The verb distribution (top-10 subset tracking) is monitored for modality relevance, and 5% of captions are manually spot-checked for motion fidelity.

4. Task Taxonomy and Evaluation Protocols

OmniMoCap-X supports ten canonical and composable motion-generation and prediction tasks:

1. Text-to-Motion (T2M)
2. Music-to-Dance (M2D)
3. Speech-to-Gesture (S2G)
4. Human-Object Interaction (HOI)
5. Human-Scene Interaction (HSI)
6. Human-Human Interaction (HHI)
7. Motion Prediction (temporal extrapolation)
8. In-Betweening (temporal interpolation)
9. Joint-Guided Synthesis (sparse/dense joint constraint)
10. Trajectory-Guided Synthesis (path or root-trajectory constraint)

Dataset splits: Each task is defined over 150-frame clips. Training comprises approximately 95% of the dataset; validation is a 5% random hold-out per source. Test splits use uniform sampling (e.g., 10 sequences per source for T2M and GSTC, or official splits of constituent datasets for M2D and S2G). For multi-condition protocols, specific control segments may be masked.

Task Group	Example Test Protocol	Details
T2M & GSTC	10 sequences per source	≈280 samples
M2D	Official splits (AIST++, …)	Source-specific
S2G	Official BEAT2 test set	Source-specific

A plausible implication is that the broad task coverage enables multi-task and cross-modal benchmarking previously infeasible with individual datasets.

5. Licensing and Access

OmniMoCap-X is publicly released, including both SMPL-X formatted motion data and GPT-4o-generated captions, at https://github.com/GuoweiXu368/OmniMocap-X. Redistributed data are governed by the original licensing conditions of each constituent MoCap dataset, including possible academic-use restrictions for some proprietary sources. No further limitations are imposed beyond those of the providers.

6. Research Impact and Context

OmniMoCap-X constitutes the largest unified multimodal motion dataset available to date, explicitly designed for the training and evaluation of models such as OmniMotion-X, which adopt autoregressive diffusion transformer architectures in sequence-to-sequence mapping for realistic motion synthesis (Xu et al., 22 Oct 2025). Its scale, heterogeneity, and standardized representation facilitate cross-modal, conditional, and controllable animation generation. The GPT-4o-based hierarchical annotation pipeline offers rich, structured supervision for tasks ranging from low-level actuation to high-level semantic interpretation.

This enables state-of-the-art performance in interactive, controllable, and coherent long-duration motion synthesis tasks, and supports emerging research on sequence modeling, joint/trajectory-guided synthesis, and multimodal conditional animation. A plausible implication is that OmniMoCap-X will accelerate advances in both generative modeling and benchmark evaluation within human motion research, as well as support applications in animation, robotics, and virtual environment simulation.