---
title: 'NextMotionQA: 3D Human Motion Benchmark'
url: https://www.emergentmind.com/topics/nextmotionqa
type: topic
---

# NextMotionQA: 3D Human Motion Benchmark

NextMotionQA is a benchmark for 3D human-motion understanding that evaluates vision-language models (VLMs) at a finer semantic and methodological granularity than earlier motion benchmarks. It is designed not merely to report a single aggregate score, but to diagnose where models fail across task format, motion semantics, and clip complexity. The benchmark is organized as a \(3 \times 3 \times 3\) structure spanning three task formats, three semantic axes, and three difficulty levels, and is built through a semi-automated, expert-verified pipeline over AMASS-derived motion clips with metadata from BABEL and HumanML3D [2606.04773].

## 1. Motivation and problem formulation

NextMotionQA was introduced in response to three structural deficiencies identified in prior motion benchmarks: coarse semantic granularity, lack of explicit difficulty stratification, and limited annotation quality with substantial answer ambiguity [2606.04773]. The benchmark’s premise is that broad prompts such as “What action is happening?” conceal clinically important failure modes, including inability to determine which body part is involved, which direction the body translates, or how compositional modifiers alter an action.

A central empirical motivation comes from a pilot study on HumanMotionQA. On 150 sampled test items, three motion experts achieved **per-annotator accuracy** of \(52.9 \pm 4.0\%\), **pooled accuracy** of \(51.8\%\), **all correct (unanimous)** at \(34.3\%\), **either correct** at \(74.3\%\), and **inter-annotator agreement** of Fleiss’ \(\kappa = 0.46\) [2606.04773]. These figures indicate that benchmark noise in earlier resources could obscure actual model capability.

Formally, each NextMotionQA instance is represented as
\[
(m_i, t_i, a_i, d_i, y_i), \quad i=1,\dots,N
\]
where \(m_i\) is a 3D motion clip from AMASS, \(t_i \in \{T1, T2, T3\}\) is the task format, \(a_i \in \{A1, A2, A3\}\) is the semantic axis, \(d_i \in \{\text{Easy}, \text{Medium}, \text{Hard}\}\) is the difficulty tier, and \(y_i\) is the gold answer [2606.04773]. This factorization is itself part of the benchmark’s contribution, because it localizes errors to specific regions of the evaluation space rather than collapsing them into a single score.

## 2. Benchmark architecture: tasks, semantic axes, and difficulty

NextMotionQA is explicitly structured as a **3 × 3 × 3 benchmark** with three task formats, three semantic axes, and three complexity levels [2606.04773]. This design enables diagnostic slicing by recognition versus generation versus correction, by body-part versus direction versus action semantics, and by clip compositionality.

The three task formats are:

- **T1: Multiple-choice question answering (MQA)**. This tests recognition. For **A2 direction** and **A3 action**, it is a single-correct multiple-choice task. For **A1 body-part**, it is a **select-all-that-apply** format because multiple body parts may legitimately be involved [2606.04773].
- **T2: Video captioning**. This tests whether a model can freely describe the relevant motion attribute in open vocabulary, without answer options [2606.04773].
- **T3: Fine-grained error correction**. This tests critique and repair: given a corrupted caption, the model must identify the wrong span and rewrite it correctly [2606.04773].

The three semantic axes are:

- **A1: Body-part involvement**. This asks which body parts are actually involved in the motion, with examples such as arms, legs, torso, and head [2606.04773].
- **A2: Translation direction**. This asks which way the body moves, including forward, backward, lateral, and in place [2606.04773].
- **A3: Action semantics**. This asks what action is being performed, with examples such as walk, run, jump, kick, wave, sit, stand, crawl, and dance [2606.04773].

Difficulty is assigned at the clip level rather than the question level, so all tasks applied to a clip inherit a common complexity label. The formal rule is
\[
d(m)=
\begin{cases}
\text{Easy}, & \text{if } |C(m)| = 1 \text{ and } M(m)=0 \\
\text{Medium}, & \text{if } |C(m)| = 2 \\
\text{Hard}, & \text{if } |C(m)| \ge 3 \text{ or } M(m)\ge 0
\end{cases}
\]
where \(C(m)\) is the set of BABEL action labels overlapping the clip and \(M(m)\) is the set of compositional modifiers extracted from HumanML3D captions, such as direction, speed, manner, or fine-grained body-part modifiers [2606.04773]. The appendix restates this operationally as: **Easy** for exactly one BABEL action label and no modifiers, **Medium** for two adjacent BABEL labels with a fast transition, and **Hard** for three or more BABEL labels or captions with modifiers such as direction, speed, or body-part detail [2606.04773].

This organization matters because it turns motion understanding into a decomposed evaluation problem. A model may recognize the action yet fail on direction, or identify an erroneous caption span yet fail to generate a correct repair. NextMotionQA is designed to expose those separations directly.

## 3. Construction pipeline and annotation protocol

NextMotionQA is built with a **two-pass semi-automatic pipeline** using a frontier VLM, followed by expert verification [2606.04773]. The drafting model is **Qwen3.6-Plus**.

In the first pass, the VLM receives only **BABEL action labels** and **HumanML3D captions**, and does **not** see the rendered video [2606.04773]. This metadata-conditioned drafting stage encourages outputs grounded in curated symbolic descriptions rather than in potentially noisy visual inference. In the second pass, the same VLM is shown the rendered video together with the draft item and revises mismatches between draft and visible motion [2606.04773]. The benchmark then enforces a hard quota over the \(3 \times 3 \times 3 = 27\) cells; over-generated cells are rejected down and underfilled cells are re-drafted [2606.04773].

Every filtered item is subsequently judged independently by **three motion-domain experts** as **accept**, **revise**, or **reject** [2606.04773]. Acceptance requires **unanimous accept** from all three experts. If an item is marked revise, it is jointly edited and re-judged. Reject is mandatory for any instance exhibiting one of four ambiguity patterns identified in the pilot study:

- **F1: Body-part granularity collisions**, such as hand versus arm.
- **F2: Ambiguous spatial frame of reference**, such as “move right” meaning viewer-right, actor-right, or world-right.
- **F3: Temporal scope mismatch**, such as unclear interpretation of “before action X” when segments overlap.
- **F4: Composite-action dominance**, such as clips with multiple simultaneous actions and no single dominant verb [2606.04773].

The reported benchmark statistics are **1,307 expert-verified instances**, derived from **992 unique SMPL-H clips**, drawn from **16 AMASS subsets**, rendered at **30 fps**, with metadata from **BABEL + HumanML3D**. The **average clip duration** is **10.2 s**, with durations spanning **0.8 to 119.9 s** [2606.04773]. A statistics table in the paper reports the following averages by difficulty tier: **Easy** with 7.6 examples, 8.2 videos, and 8.0 average duration; **Medium** with 9.4 examples, 9.8 videos, and 9.3 average duration; and **Hard** with 12.4 examples, 15.3 videos, and 13.5 average duration [2606.04773].

A plausible implication is that NextMotionQA trades raw dataset scale for annotation precision and diagnostic validity. That interpretation is consistent with the paper’s emphasis on human-solvability screening and expert verification [2606.04773].

## 4. Evaluation protocol and model performance

The benchmark evaluates **12 representative VLMs**, including open-source models such as **Qwen3.5-0.8B**, **Qwen3.5-4B**, **Qwen3.5-9B**, **Qwen3.5-27B**, **InternVL3.5-4B**, **InternVL3.5-8B**, **InternVL3.5-14B**, **LLaVA-OneVision-1.5-4B**, and **LLaVA-OneVision-1.5-8B**, as well as closed or frontier systems including **GPT-5.4-mini**, **Qwen3.6-Plus**, and **Gemini-3.1-Flash** [2606.04773].

The overall best model is **Gemini-3.1-Flash** with **58.44**, followed by **Qwen3.6-Plus** with **54.85**. The best open-source model is **Qwen3.5-27B** with **49.75**, leaving an approximately **8.69-point gap** between the strongest open and closed models [2606.04773].

For **T1 MQA**, the reported metrics are **Accuracy**, **Jaccard**, and **Precision**, with **Jaccard** treated as the headline metric because exact-match underestimates partial overlap in multi-select answers [2606.04773]. Representative scores include **64.07 Jaccard** for Gemini-3.1-Flash, **64.52 Jaccard** for Qwen3.6-Plus, and **55.78 Jaccard** for Qwen3.5-27B [2606.04773]. The consistent pattern is that Jaccard exceeds exact-match accuracy, revealing that models often recover part of the correct body-part set without fully specifying it.

For **T2 captioning**, the paper identifies a universal bottleneck. The top scores are **43.53** for Gemini-3.1-Flash, **35.23** for Qwen3.6-Plus, and **35.57** for Qwen3.5-27B [2606.04773]. No open-source system exceeds **35.57**. This indicates that open-vocabulary motion description is markedly harder than closed-set recognition.

For **T3 error correction**, the benchmark reports **Identify**, **Token Recall**, **Correct**, and their average [2606.04773]. The dominant phenomenon is a **locate-to-fix gap**: models can identify wrong spans much more reliably than they can rewrite them correctly. For Gemini-3.1-Flash, **Identify** is **76.8%** while **Correct** is **44.9%**; for Qwen3.6-Plus, **74.5%** versus **38.4%**; for Qwen3.5-27B, **70.6%** versus **39.9%** [2606.04773]. The paper summarizes this as
\[
\text{gap} = \text{Identify} - \text{Correct}
\]
with an average of about **26 points** [2606.04773].

Several capability gaps recur across models. **A2 direction** is the hardest axis in almost all settings, with the paper describing a V-shaped T1 pattern in which direction lags body-part and action [2606.04773]. **T2 captioning** is systematically harder than **T1 recognition**, and in **T3** the ability to localize an error consistently exceeds the ability to correct it. Scaling generally helps within families, but not monotonically: for example, **InternVL3.5-14B** underperforms its 4B and 8B siblings on **T1**, indicating that model size alone does not guarantee motion understanding [2606.04773].

## 5. VLMs as judges for text-to-motion evaluation

NextMotionQA also studies whether VLMs can function as automatic judges for generated motion, extending the benchmark from direct motion understanding to motion evaluation [2606.04773]. The setup evaluates generated motions from **nine T2M methods** across three increasing difficulty regimes:

- **V1**: single-action prompts on small clips,
- **V2**: composite, chained actions on larger clips,
- **V3**: fine-grained, temporally segmented motions [2606.04773].

Two judgment criteria are used: **Realism** and **Semantic consistency** [2606.04773]. Agreement between human expert judgments and VLM judgments is reported at instance, per-question, and system levels.

For **V1**, agreement is strong: **instance Pearson \(r = 0.774\)**, **Cohen’s \(\kappa = 0.701\)**, and **system Pearson \(r = 0.966\)** [2606.04773]. For **V2**, agreement drops to **instance Pearson \(r = 0.495\)** and **Cohen’s \(\kappa = 0.346\)**, although **system Pearson \(r = 1.000\)** remains perfect [2606.04773]. For **V3**, the judge essentially fails: **instance Pearson \(r = 0.116\)**, **Cohen’s \(\kappa = 0.104\)**, and **system Pearson \(r = -0.146\)** [2606.04773].

The paper’s headline interpretation is that VLM judges align strongly with experts on coarse criteria, with **\(\kappa = 0.70\)**, but break down on fine-grained, part-level judgment, with **\(\kappa = 0.10\)** [2606.04773]. This validates the VLM-as-a-judge paradigm only in a restricted regime: broad realism and consistency, especially at coarse or system level, not detailed part-aware evaluation.

This result is closely coupled to the benchmark’s broader diagnosis. If models already struggle on direction, fine-grained captioning, and correction, then degradation under part-level judgment is not anomalous; it is a predictable extension of the same representational weakness. The judge study makes that limitation explicit.

## 6. Position within motion-understanding research

NextMotionQA sits within a line of work that attempts to move motion understanding beyond coarse action recognition. **HumanMotionQA** introduced question answering over long human motion sequences and emphasized multi-step reasoning over motor cues, temporal relations, and motion attributes, together with the neuro-symbolic **NSPose** model [2305.08953]. Later work such as **IMoRe** argued that explicit program execution limits scalability and proposed implicit program-guided reasoning, achieving state-of-the-art performance on Babel-QA and generalization to HuMMan-QA [2508.01984]. NextMotionQA differs from both by focusing primarily on **evaluation design** rather than on a new reasoning architecture: its contribution is diagnostic granularity, expert verification, and a multi-task benchmark regime [2606.04773].

The benchmark also differs from more geometry-centered motion QA resources. **4DP-QA** targets 4D scene understanding and disentangling camera motion from object motion, introducing **True-Motion Tracking** and a **400K**-sample training set with a **2.2K** benchmark [2606.11568]. **HumanMoveVQA** focuses on exocentric reasoning about global human trajectory and orientation in a first-frame anchored world coordinate system, with over **10K** QA pairs across seven reasoning categories [2606.27999]. By contrast, NextMotionQA concentrates on **human motion semantics in 3D mocap-derived clips**, structured by task, axis, and complexity rather than by world-consistent trajectory reconstruction or large-scale auto-generated supervision [2606.04773].

The benchmark’s limitations are explicitly acknowledged. It is **AMASS-only**, so it inherits the distributional biases of that motion domain and does not cover **dyadic interaction**, **hand-object manipulation**, or **non-rigid deformation** [2606.04773]. Its VLM-as-judge protocol uses **single-view rendering**, which under-resolves back-of-body configurations; the paper identifies **multi-view rendering** as a natural next step [2606.04773]. Finally, expert verification constrains scale to roughly \(10^3\) items, and the paper suggests that scaling toward \(10^4\) instances will likely require **expert-in-the-loop active learning** [2606.04773].

In that sense, NextMotionQA functions less as a high-volume pretraining resource and more as a high-precision diagnostic instrument. It reframes motion understanding as a set of separable competencies—recognition, description, correction; body-part, direction, action; easy, medium, hard—and shows that contemporary VLMs remain strongest on coarse recognition, weaker on open-ended description, and weakest on fine-grained correction and directional grounding [2606.04773].

Source: https://www.emergentmind.com/topics/nextmotionqa