MotionMillion-Eval Benchmark
- MotionMillion-Eval is a human-verified benchmark for evaluating text-to-motion generation models, emphasizing generalization and compositionality.
- It employs 126 curated prompts across seven thematic categories to systematically gauge text alignment, motion smoothness, and physical plausibility.
- Results indicate that increased model size enhances textual alignment, while gains in smoothness and plausibility plateau, especially for stylized motions.
MotionMillion-Eval is a human-verified benchmark designed to systematically evaluate the zero-shot performance of text-to-motion generation models. Addressing the limitations of prior benchmarks, which often fail to probe generalization and compositionality due to narrow scope and dataset size, MotionMillion-Eval introduces a challenging suite of natural-language prompts to assess alignment, physical plausibility, and temporal coherence of generated human motion, both in familiar and out-of-domain contexts (Fan et al., 9 Jul 2025).
1. Design Objectives and Scope
MotionMillion-Eval directly targets the need for robust evaluation of zero-shot generalization in text-to-motion models—a capability largely unassessable with previous datasets such as HumanML3D. Its design goals include:
- Exposing limitations in model generalization to both in-domain (well-represented actions) and out-of-domain or rare motions.
- Systematically testing alignment to complex, compositional, and multi-clause natural-language prompts derived from industrial design guidelines.
- Establishing a human-grounded comparative framework for both benchmarked baselines and new large-scale models.
Prompts are distributed across seven thematic categories: Daily Life, Work, Arts/Dance, Communication, Combat, Sports, and Non-human Behavior. These ensure coverage of common, composite, stylized, and rarely observed motions.
2. Dataset Construction and Splitting Protocol
The benchmark consists of 126 hand-curated textual prompts, structured to demand the demonstration of compositional reasoning in motion synthesis (e.g., sequential actions like “drinking from a mug and then wiping the table”). No motion ground-truth data is released for these scenarios. Instead, models must generate motion sequences ab initio for evaluation.
Crucially, all participating models are trained (and optionally validated) solely on the MotionMillion training dataset (~1.6M sequences) or other large-scale text-motion datasets, with MotionMillion-Eval serving as a fully held-out set. This isolates genuine zero-shot performance and precludes leakage of evaluation examples or action combinations into the training regime. Prompts frequently reference “unseen concepts” such as stylized or rare behaviors (e.g., zombie walk, martial arts forms) that are absent or underrepresented in existing datasets.
3. Evaluation Methodology and Metrics
Evaluation proceeds exclusively via human annotation along three axes, each quantified on a four-point scale (1 = poor; 4 = excellent):
- Text Alignment (TA): Degree to which the motion accurately reflects the content and ordering of the textual prompt.
- Motion Smoothness (MS): Assessment of temporal fluidity and lack of stutter or discontinuity.
- Physical Plausibility (PP): Extent to which the motion adheres to real-world physics, such as natural body mechanics and avoidance of artifacts (e.g., foot sliding, interpenetration).
The precise rubric is documented in supplementary material; each axis is scored independently, enabling the decoupling of semantic and physical evaluation.
| Metric | Score 4 (Best) | Score 1 (Worst) |
|---|---|---|
| Text Alignment | Accurate, complete, ordered | Unrelated to prompt |
| Smoothness | Fluid, no stutter | Highly unnatural |
| Plausibility | Full physical realism | Impossible/implausible |
4. Baseline Models and Generation Protocol
MotionMillion-Eval contrasts several state-of-the-art zero-shot models:
- MDM: Diffusion-based generation.
- MotionGPT: LLM-based motion synthesis.
- T2M-GPT: Discrete token autoregressive approach.
- ScaMo-3B: Scalable architecture on MotionUnion, 1 B→3 B parameters.
Additionally, the new MotionMillion models are evaluated at three parameter sizes: 1B, 3B, and 7B, all trained on the 2M-clip MotionMillion dataset. For each evaluation prompt, all models generate a 3D pose sequence without any prompt-specific fine-tuning. Rendered clips are then presented side-by-side for blind annotation.
5. Comparative Results and Category Analysis
Aggregate human scores reveal that Text Alignment improves markedly with increased model size, while physical plausibility and smoothness plateau at high quality across strong models. The 7B MotionMillion model sets a new benchmark for Text Alignment, outperforming all listed baselines.
Table of mean human scores across 126 prompts:
| Method | Text Alignment | Physical Plausibility | Motion Smoothness |
|---|---|---|---|
| MDM | 195.5 | 478.5 | 416.5 |
| MotionGPT | 170.0 | 497.0 | 501.5 |
| T2M-GPT | 207.0 | 495.5 | 500.0 |
| ScaMo-3B | 226.6 | 477.5 | 494.0 |
| Ours-1B | 170.3 | 497.0 | 501.0 |
| Ours-3B | 238.6 | 496.0 | 499.5 |
| Ours-7B | 261.0 | 495.5 | 501.0 |
The win/tie/lose counts between the MotionMillion-7B and ScaMo-3B models, as assessed by three annotators across every category, indicate that MotionMillion-7B usually prevails, except in Arts/Dance and Non-human categories—areas characterized by data scarcity and stylization. Compositional prompts in Communication and Combat yield the largest Text Alignment gains, underscoring the benefits of scale.
6. Strengths, Limitations, and Prospective Extensions
MotionMillion-Eval is the first benchmark of its scale to operationalize human-grounded, zero-shot evaluation for text-driven motion generation. Its use of detailed, compositional prompts and its three-axis scoring approach supply a nuanced, multi-faceted understanding of model capabilities.
Identified limitations include the cost and subjectivity of human evaluation, the modest prompt count (126), and limited fine-grained coverage of gesture (e.g., hand or facial motion). Extensions proposed by the original authors include: scaling prompt sets via user-generated queries, integrating automated scoring pipelines (e.g., motion–text embedding alignment), and broadening behavioral granularity to include retrieval or micro-gesture tasks.
A plausible implication is that continued scaling of both training data and model size yields diminishing returns on smoothness and plausibility but offers clear advances in compositional alignment, especially in zero-shot regimes. Emerging challenges persist for highly stylized motions, implying that specialized data collection remains an open problem.
7. Impact on Text-to-Motion Evaluation Paradigms
MotionMillion-Eval addresses a critical methodological gap by providing a standardized and rigorous zero-shot testbed, facilitating direct comparisons and exposing the strengths and weaknesses of current and future generative models. It introduces a reproducible path for benchmarking holistic text-to-motion abilities and informs priorities for future data curation and the development of automated, scalable evaluative proxies (Fan et al., 9 Jul 2025).