HuMo100M: Multimodal Motion Dataset

• HuMo100M is a large-scale multimodal human motion dataset combining 5M 3D motion sequences with 100M instruction-motion pairs to enhance vision-language-motion research.
• It offers detailed part-level annotations that enable fine-grained control in motion synthesis and support natural language directive execution.
• The dataset tackles challenges such as noisy data and short-clip bias through robust filtering, 3D SMPL recovery, and reinforcement learning-based motion refinement.

HuMo100M is a large-scale, multimodal human motion dataset developed to address critical data limitations in vision-language-motion modeling for human motion generation and understanding. Comprising over 5 million 3D motion sequences and 100 million instruction–motion pairs sourced and curated from diverse, unconstrained video data, HuMo100M integrates extensive part-level annotations and serves as the foundational resource for the controllable vision-language-motion model Being-M0.5. Its comprehensive design enables control over five core problem dimensions: natural language instruction following, arbitrary pose initialization, long-term motion synthesis, generalization to novel actions, and fine-grained part-level body control (Cao et al., 11 Aug 2025).

1. Objectives and Scope

HuMo100M addresses the scarcity of sufficiently diverse, annotated, and large-scale motion data in the development of vision-language-motion models. The dataset is particularly constructed to support:

• Natural-language directed motion generation
• Arbitrary pose initialization
• Robust long-term sequence generation
• Unseen scenario generalization
• Fine-grained, anatomically meaningful part control

It combines self-collected and web-extracted motion with established public datasets, covering a wide spectrum of real-world activities from everyday locomotion to complex sports and object interactions. The inclusion of detailed part-level supervision supplies a crucial annotation layer missing from earlier resources, facilitating fine-grained, controllable motion synthesis.

2. Dataset Composition and Statistics

HuMo100M comprises the following:

• Motion sequences: Over 5 million 3D human motion sequences
• Instructional instances: 100 million text–motion or instruction–motion pairs
• Source videos: Derived from over 20 million input videos

Key integrated subsets include Motion-X, 3DPW, MSCOCO pose snapshots, PoseTrack, PROX, BEHAVE, WebVid, and custom web-scraped clips. Sequence lengths range from 16 frames (PoseTrack) to approximately 995 frames (PROX, BEHAVE), covering both short and extended temporal contexts. Scenario diversity encompasses daily activities, sports, public space interactions, and object manipulation.

A summary of illustrative subset counts is provided below:

Subset	Sequence Count
PROX	27
BEHAVE	975
WebVid	2,376,376

(Table is for illustration; the full distribution is presented on a logarithmic scale in the original appendix.)

3. Data Collection and Annotation Workflow

The HuMo100M pipeline features multiple sequential stages:

1. Initial Filtering: Video metadata are filtered via keywords related to human activity.
2. Human Tracking and Segmentation: YOLO-based detection locates and tracks subjects, segmenting continuous trajectories for motion extraction.
3. Quality Assessment:
   • Occlusion Filtering: 2D keypoint confidence checks remove clips with insufficient high-confidence detections.
   • Length Filtering: Clips below a minimum frame threshold are discarded for temporal context adequacy.
4. 3D Motion Recovery: SMPL parameter regression is performed using WHAM.
5. Motion Refinement: A reinforcement learning-based policy (PHC) smooths trajectories and increases physical plausibility.
6. Instruction Generation:
   • LMM-based: Gemini-1.5-Pro generates body- and part-level captions using prompt templates.
   • Rule-based: PoseScript extracts structured joint-relation statements (e.g., “left hand is below right shoulder”).

4. Part-Level Supervision and Instructional Benchmarks

Five anatomically defined regions receive explicit supervision: left arm, right arm, left leg, right leg, and torso, with each comprising relevant skeletal joints (e.g., spine₁–₃, collars, limb segments). Every motion sequence includes:

• A body-level caption (e.g., “a person walks forward”)
• Part-level captions per limb (e.g., “the right arm swings forward”)
• Rule-based pose relationships (e.g., “the left knee is bent more than the right”)

Derived benchmarks and splits follow standard protocols:

Benchmark	Pairs/Instances	Split (Train/Val/Test)
HuMo-T2M	200,000	80% / 5% / 15%
HuMo-I2M	1,000,000	80% / 5% / 15%
HuMo-I2LM	500,000	80% / 5% / 15%
HuMo-I2PM	200,000	25% / 75% (val/test)

For HuMo-I2PM, no training split is separately designated since part-level instructions are always sampled from the general 100M instruction pool.

5. Motion Tokenization and Evaluation Protocols

A novel part-aware residual quantization (PRQ) approach is introduced for tokenizing motion, enabling control at the body-part level. The quantization process per part proceeds iteratively as:

$$b^k = \mathcal{Q}(r^k),\quad r^{k+1} = r^k - b^k,\quad \mathrm{PRQ}(\tilde b)= [\,b^0,\dots,b^K\,]$$

The loss objective for motion reconstruction and embedding is:

$$\mathcal{L} = \sum_{j=1}^p\|m_j-\tilde m_j\|_1 +\|m-\tilde m\|_1 +\beta\sum_{k=1}^K\sum_{j=1}^p\|r^k_j-\mathrm{sg}[b^k_j]\|_2^2$$

Model evaluation leverages several standard metrics:

• Fréchet Inception Distance (FID): assesses motion sequence quality
• R-Precision (R@1/3): measures retrieval congruence between text and motion
• Multimodal Distance (MM-Dist): computes separation in joint embedding space
• Mean Per-Joint Position Error (MPJPE): quantifies 3D pose accuracy

The token-prediction learning objective is:

$$\mathcal{L}(\Theta) = -\sum_{j=1}^{L}\log P_{\Theta}\bigl(y_j\mid \mathcal{X}_Q,\hat y_{1:j-1}\bigr)$$

6. Addressed Challenges, Limitations, and Prospects

HuMo100M directly confronts several technical challenges:

• Noisy and occluded sources: Ameliorated by keypoint filtering and RL-based motion refinement.
• Short-clip bias: Mitigated through interpolation-based and learning-based concatenation for long-form data.
• Lack of part-level supervision: Resolved via hierarchical, multi-level (body/part/rule) instruction annotation.

However, some limitations persist:

• Residual errors in SMPL parameterization may remain, especially in cases of heavy occlusion.
• The web-source composition is inherently unbalanced, with sports actions relatively overrepresented vis-à-vis daily activities.

Planned and prospective extensions include the addition of object and scene-context labels (e.g., “sitting on a chair”), taxonomy expansion to encompass finer structures (fingers, facial expressions), and, potentially, end-to-end vision-to-motion modeling using raw video (Cao et al., 11 Aug 2025).

7. Research Impact and Relevance

HuMo100M underpins advances in real-time, controllable motion generation, exemplified by its foundational role in Being-M0.5. The dataset’s scale, anatomical granularity, and instruction-to-motion design close critical gaps previously hampering generalization, controllability, and practical application in vision-language-motion agents. Its benchmarking splits, comprehensive annotation, and detailed computational protocols set a new standard for reproducible research in multimodal human motion synthesis and control.