NextMotionQA: 3D Human Motion Benchmark

• NextMotionQA is a diagnostic benchmark for 3D human-motion understanding that segments evaluation by task format, semantic axis, and difficulty level.
• It employs a 3×3×3 design covering recognition, captioning, and error correction tasks across body-part, direction, and action semantics.
• A semi-automated, expert-verified pipeline ensures high annotation quality, enabling precise error analysis over aggregate scoring.

NextMotionQA is a benchmark for 3D human-motion understanding that evaluates vision-LLMs (VLMs) at a finer semantic and methodological granularity than earlier motion benchmarks. It is designed not merely to report a single aggregate score, but to diagnose where models fail across task format, motion semantics, and clip complexity. The benchmark is organized as a $3 \times 3 \times 3$ structure spanning three task formats, three semantic axes, and three difficulty levels, and is built through a semi-automated, expert-verified pipeline over AMASS-derived motion clips with metadata from BABEL and HumanML3D (Cao et al., 3 Jun 2026).

1. Motivation and problem formulation

NextMotionQA was introduced in response to three structural deficiencies identified in prior motion benchmarks: coarse semantic granularity, lack of explicit difficulty stratification, and limited annotation quality with substantial answer ambiguity (Cao et al., 3 Jun 2026). The benchmark’s premise is that broad prompts such as “What action is happening?” conceal clinically important failure modes, including inability to determine which body part is involved, which direction the body translates, or how compositional modifiers alter an action.

A central empirical motivation comes from a pilot study on HumanMotionQA. On 150 sampled test items, three motion experts achieved per-annotator accuracy of $52.9 \pm 4.0\%$, pooled accuracy of $51.8\%$, all correct (unanimous) at $34.3\%$, either correct at $74.3\%$, and inter-annotator agreement of Fleiss’ $\kappa = 0.46$ (Cao et al., 3 Jun 2026). These figures indicate that benchmark noise in earlier resources could obscure actual model capability.

Formally, each NextMotionQA instance is represented as

$(m_i, t_i, a_i, d_i, y_i), \quad i=1,\dots,N$

where $m_i$ is a 3D motion clip from AMASS, $t_i \in \{T1, T2, T3\}$ is the task format, $a_i \in \{A1, A2, A3\}$ is the semantic axis, $52.9 \pm 4.0\%$0 is the difficulty tier, and $52.9 \pm 4.0\%$1 is the gold answer (Cao et al., 3 Jun 2026). This factorization is itself part of the benchmark’s contribution, because it localizes errors to specific regions of the evaluation space rather than collapsing them into a single score.

2. Benchmark architecture: tasks, semantic axes, and difficulty

NextMotionQA is explicitly structured as a 3 × 3 × 3 benchmark with three task formats, three semantic axes, and three complexity levels (Cao et al., 3 Jun 2026). This design enables diagnostic slicing by recognition versus generation versus correction, by body-part versus direction versus action semantics, and by clip compositionality.

The three task formats are:

• T1: Multiple-choice question answering (MQA). This tests recognition. For A2 direction and A3 action, it is a single-correct multiple-choice task. For A1 body-part, it is a select-all-that-apply format because multiple body parts may legitimately be involved (Cao et al., 3 Jun 2026).
• T2: Video captioning. This tests whether a model can freely describe the relevant motion attribute in open vocabulary, without answer options (Cao et al., 3 Jun 2026).
• T3: Fine-grained error correction. This tests critique and repair: given a corrupted caption, the model must identify the wrong span and rewrite it correctly (Cao et al., 3 Jun 2026).

The three semantic axes are:

• A1: Body-part involvement. This asks which body parts are actually involved in the motion, with examples such as arms, legs, torso, and head (Cao et al., 3 Jun 2026).
• A2: Translation direction. This asks which way the body moves, including forward, backward, lateral, and in place (Cao et al., 3 Jun 2026).
• A3: Action semantics. This asks what action is being performed, with examples such as walk, run, jump, kick, wave, sit, stand, crawl, and dance (Cao et al., 3 Jun 2026).

Difficulty is assigned at the clip level rather than the question level, so all tasks applied to a clip inherit a common complexity label. The formal rule is

$52.9 \pm 4.0\%$2

where $52.9 \pm 4.0\%$3 is the set of BABEL action labels overlapping the clip and $52.9 \pm 4.0\%$4 is the set of compositional modifiers extracted from HumanML3D captions, such as direction, speed, manner, or fine-grained body-part modifiers (Cao et al., 3 Jun 2026). The appendix restates this operationally as: Easy for exactly one BABEL action label and no modifiers, Medium for two adjacent BABEL labels with a fast transition, and Hard for three or more BABEL labels or captions with modifiers such as direction, speed, or body-part detail (Cao et al., 3 Jun 2026).

This organization matters because it turns motion understanding into a decomposed evaluation problem. A model may recognize the action yet fail on direction, or identify an erroneous caption span yet fail to generate a correct repair. NextMotionQA is designed to expose those separations directly.

3. Construction pipeline and annotation protocol

NextMotionQA is built with a two-pass semi-automatic pipeline using a frontier VLM, followed by expert verification (Cao et al., 3 Jun 2026). The drafting model is Qwen3.6-Plus.

In the first pass, the VLM receives only BABEL action labels and HumanML3D captions, and does not see the rendered video (Cao et al., 3 Jun 2026). This metadata-conditioned drafting stage encourages outputs grounded in curated symbolic descriptions rather than in potentially noisy visual inference. In the second pass, the same VLM is shown the rendered video together with the draft item and revises mismatches between draft and visible motion (Cao et al., 3 Jun 2026). The benchmark then enforces a hard quota over the $52.9 \pm 4.0\%$5 cells; over-generated cells are rejected down and underfilled cells are re-drafted (Cao et al., 3 Jun 2026).

Every filtered item is subsequently judged independently by three motion-domain experts as accept, revise, or reject (Cao et al., 3 Jun 2026). Acceptance requires unanimous accept from all three experts. If an item is marked revise, it is jointly edited and re-judged. Reject is mandatory for any instance exhibiting one of four ambiguity patterns identified in the pilot study:

• F1: Body-part granularity collisions, such as hand versus arm.
• F2: Ambiguous spatial frame of reference, such as “move right” meaning viewer-right, actor-right, or world-right.
• F3: Temporal scope mismatch, such as unclear interpretation of “before action X” when segments overlap.
• F4: Composite-action dominance, such as clips with multiple simultaneous actions and no single dominant verb (Cao et al., 3 Jun 2026).

The reported benchmark statistics are 1,307 expert-verified instances, derived from 992 unique SMPL-H clips, drawn from 16 AMASS subsets, rendered at 30 fps, with metadata from BABEL + HumanML3D. The average clip duration is 10.2 s, with durations spanning 0.8 to 119.9 s (Cao et al., 3 Jun 2026). A statistics table in the paper reports the following averages by difficulty tier: Easy with 7.6 examples, 8.2 videos, and 8.0 average duration; Medium with 9.4 examples, 9.8 videos, and 9.3 average duration; and Hard with 12.4 examples, 15.3 videos, and 13.5 average duration (Cao et al., 3 Jun 2026).

A plausible implication is that NextMotionQA trades raw dataset scale for annotation precision and diagnostic validity. That interpretation is consistent with the paper’s emphasis on human-solvability screening and expert verification (Cao et al., 3 Jun 2026).

4. Evaluation protocol and model performance

The benchmark evaluates 12 representative VLMs, including open-source models such as Qwen3.5-0.8B, Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, InternVL3.5-4B, InternVL3.5-8B, InternVL3.5-14B, LLaVA-OneVision-1.5-4B, and LLaVA-OneVision-1.5-8B, as well as closed or frontier systems including GPT-5.4-mini, Qwen3.6-Plus, and Gemini-3.1-Flash (Cao et al., 3 Jun 2026).

The overall best model is Gemini-3.1-Flash with 58.44, followed by Qwen3.6-Plus with 54.85. The best open-source model is Qwen3.5-27B with 49.75, leaving an approximately 8.69-point gap between the strongest open and closed models (Cao et al., 3 Jun 2026).

For T1 MQA, the reported metrics are Accuracy, Jaccard, and Precision, with Jaccard treated as the headline metric because exact-match underestimates partial overlap in multi-select answers (Cao et al., 3 Jun 2026). Representative scores include 64.07 Jaccard for Gemini-3.1-Flash, 64.52 Jaccard for Qwen3.6-Plus, and 55.78 Jaccard for Qwen3.5-27B (Cao et al., 3 Jun 2026). The consistent pattern is that Jaccard exceeds exact-match accuracy, revealing that models often recover part of the correct body-part set without fully specifying it.

For T2 captioning, the paper identifies a universal bottleneck. The top scores are 43.53 for Gemini-3.1-Flash, 35.23 for Qwen3.6-Plus, and 35.57 for Qwen3.5-27B (Cao et al., 3 Jun 2026). No open-source system exceeds 35.57. This indicates that open-vocabulary motion description is markedly harder than closed-set recognition.

For T3 error correction, the benchmark reports Identify, Token Recall, Correct, and their average (Cao et al., 3 Jun 2026). The dominant phenomenon is a locate-to-fix gap: models can identify wrong spans much more reliably than they can rewrite them correctly. For Gemini-3.1-Flash, Identify is 76.8% while Correct is 44.9%; for Qwen3.6-Plus, 74.5% versus 38.4%; for Qwen3.5-27B, 70.6% versus 39.9% (Cao et al., 3 Jun 2026). The paper summarizes this as

$52.9 \pm 4.0\%$6

with an average of about 26 points (Cao et al., 3 Jun 2026).

Several capability gaps recur across models. A2 direction is the hardest axis in almost all settings, with the paper describing a V-shaped T1 pattern in which direction lags body-part and action (Cao et al., 3 Jun 2026). T2 captioning is systematically harder than T1 recognition, and in T3 the ability to localize an error consistently exceeds the ability to correct it. Scaling generally helps within families, but not monotonically: for example, InternVL3.5-14B underperforms its 4B and 8B siblings on T1, indicating that model size alone does not guarantee motion understanding (Cao et al., 3 Jun 2026).

5. VLMs as judges for text-to-motion evaluation

NextMotionQA also studies whether VLMs can function as automatic judges for generated motion, extending the benchmark from direct motion understanding to motion evaluation (Cao et al., 3 Jun 2026). The setup evaluates generated motions from nine T2M methods across three increasing difficulty regimes:

• V1: single-action prompts on small clips,
• V2: composite, chained actions on larger clips,
• V3: fine-grained, temporally segmented motions (Cao et al., 3 Jun 2026).

Two judgment criteria are used: Realism and Semantic consistency (Cao et al., 3 Jun 2026). Agreement between human expert judgments and VLM judgments is reported at instance, per-question, and system levels.

For V1, agreement is strong: instance Pearson $52.9 \pm 4.0\%$7, Cohen’s $52.9 \pm 4.0\%$8, and system Pearson $52.9 \pm 4.0\%$9 (Cao et al., 3 Jun 2026). For V2, agreement drops to instance Pearson $51.8\%$0 and Cohen’s $51.8\%$1, although system Pearson $51.8\%$2 remains perfect (Cao et al., 3 Jun 2026). For V3, the judge essentially fails: instance Pearson $51.8\%$3, Cohen’s $51.8\%$4, and system Pearson $51.8\%$5 (Cao et al., 3 Jun 2026).

The paper’s headline interpretation is that VLM judges align strongly with experts on coarse criteria, with $51.8\%$6, but break down on fine-grained, part-level judgment, with $51.8\%$7 (Cao et al., 3 Jun 2026). This validates the VLM-as-a-judge paradigm only in a restricted regime: broad realism and consistency, especially at coarse or system level, not detailed part-aware evaluation.

This result is closely coupled to the benchmark’s broader diagnosis. If models already struggle on direction, fine-grained captioning, and correction, then degradation under part-level judgment is not anomalous; it is a predictable extension of the same representational weakness. The judge study makes that limitation explicit.

6. Position within motion-understanding research

NextMotionQA sits within a line of work that attempts to move motion understanding beyond coarse action recognition. HumanMotionQA introduced question answering over long human motion sequences and emphasized multi-step reasoning over motor cues, temporal relations, and motion attributes, together with the neuro-symbolic NSPose model (Endo et al., 2023). Later work such as IMoRe argued that explicit program execution limits scalability and proposed implicit program-guided reasoning, achieving state-of-the-art performance on Babel-QA and generalization to HuMMan-QA (Li et al., 4 Aug 2025). NextMotionQA differs from both by focusing primarily on evaluation design rather than on a new reasoning architecture: its contribution is diagnostic granularity, expert verification, and a multi-task benchmark regime (Cao et al., 3 Jun 2026).

The benchmark also differs from more geometry-centered motion QA resources. 4DP-QA targets 4D scene understanding and disentangling camera motion from object motion, introducing True-Motion Tracking and a 400K-sample training set with a 2.2K benchmark (Cho et al., 10 Jun 2026). HumanMoveVQA focuses on exocentric reasoning about global human trajectory and orientation in a first-frame anchored world coordinate system, with over 10K QA pairs across seven reasoning categories (Gera et al., 26 Jun 2026). By contrast, NextMotionQA concentrates on human motion semantics in 3D mocap-derived clips, structured by task, axis, and complexity rather than by world-consistent trajectory reconstruction or large-scale auto-generated supervision (Cao et al., 3 Jun 2026).

The benchmark’s limitations are explicitly acknowledged. It is AMASS-only, so it inherits the distributional biases of that motion domain and does not cover dyadic interaction, hand-object manipulation, or non-rigid deformation (Cao et al., 3 Jun 2026). Its VLM-as-judge protocol uses single-view rendering, which under-resolves back-of-body configurations; the paper identifies multi-view rendering as a natural next step (Cao et al., 3 Jun 2026). Finally, expert verification constrains scale to roughly $51.8\%$8 items, and the paper suggests that scaling toward $51.8\%$9 instances will likely require expert-in-the-loop active learning (Cao et al., 3 Jun 2026).

In that sense, NextMotionQA functions less as a high-volume pretraining resource and more as a high-precision diagnostic instrument. It reframes motion understanding as a set of separable competencies—recognition, description, correction; body-part, direction, action; easy, medium, hard—and shows that contemporary VLMs remain strongest on coarse recognition, weaker on open-ended description, and weakest on fine-grained correction and directional grounding (Cao et al., 3 Jun 2026).