- The paper introduces HumanMoveVQA, a benchmark that evaluates Video MLLMs on global human motion reasoning using world-anchored 3D pose trajectories.
- It details a scalable pipeline that integrates world-space lifting, spatial discretization, and quality control across diverse datasets with over 10,000 QA pairs.
- Experimental results reveal that state-of-the-art MLLMs struggle with trajectory and orientation reasoning, while fine-tuning significantly improves performance.
HumanMove VQA: Benchmarking the Video MLLM Reasoning on Global Human Motion
Motivation and Problem Specification
Multimodal LLMs (MLLMs) have made marked strides in video understanding tasks, but remain fundamentally limited in capturing and reasoning about global human movement—specifically, 3D trajectory and body orientation—over extended temporal horizons. Existing benchmarks predominantly focus on short sequences, high-level event labels, or joint-level articulations, thereby ignoring the challenge of global spatial reasoning crucial to domains such as sports analytics, robotics, and surveillance. The paper introduces HumanMove VQA (2606.27999), a large-scale and rigorous benchmark for evaluating the capacity of Video MLLMs to reason about human movement in exocentric video, emphasizing displacement and orientation changes with respect to a fixed world reference.
Benchmark Construction and Methodology
Dataset Composition and Motion Tracking
HumanMove VQA integrates three high-fidelity human movement datasets: EMDB, RICH, and EgoBody, covering a diverse range of environments, camera motions (including strong ego-motion), and person appearances. A scalable pipeline is proposed to standardize sequence representation:
- World-Space Lifting: 2D video frames are transformed into 3D SMPL-X pose trajectories in a scene-anchored frame using PromptHMR, strictly decoupling subject movement from camera motion.
- Spatial Discretization: Continuous translation and rotation (displacement_x/y/z, rotation_roll/pitch/yaw) are discretized to spatial codes, providing categorical and noise-robust descriptors.
- Identity Grounding: In multi-person settings, BLIP-2-generated clothing descriptors resolve referential ambiguity, ensuring the spatial query is grounded to a specific subject.
- Quality Control: Both global (video-level) and local (event-level) filtering remove outliers, emphasizing long, high-integrity motion tracks.
Question-Answer Pair Generation
The benchmark consists of over 10,000 multiple-choice QA pairs with all options crafted to eliminate linguistic priors, demanding authentic visual-motion reasoning. There are seven task axes:
- Existence: Detection of specific movement or rotation events.
- Comparative: Pairwise comparison (e.g., more motions left/right).
- Dominant: Aggregation for identifying primary movement direction.
- Numerical: Counting events or measuring displacement.
- Ordering: Sequencing of movement events temporally.
- Temporal: Localization and event speed estimation.
- Trajectory Affordance: Long-range spatial reasoning (e.g., path length, trajectory evolution from start to end position).
All answers are mapped from deterministic logic over the discretized motion tracks, ensuring ground-truth derivability, and distractors are systematically balanced.
Experimental Study
Evaluation Protocol
The authors perform a comprehensive evaluation on zero-shot and fine-tuned settings using leading proprietary (e.g., Gemini-3-Flash, GPT-4o) and open-source (Qwen3-VL, VideoLLaVA, VideoGPT+, MotionLLM) MLLMs. They also introduce a supervised fine-tuning (SFT) regime on Qwen3-VL 8B using HumanMove VQA to test learnability of the task.
Key Results
- Zero-shot performance on trajectory and orientation reasoning is poor across all models: Gemini-3-Flash, while leading, approaches chance-level for comparative and aggregation tasks (e.g., chance-normalized score 14.3 for trajectory tasks), indicating MLLMs' failure to aggregate spatial dynamics over time.
- Qwen3-VL 8B fine-tuned with HumanMove VQA triples accuracy (score increases from ~13 to ~38–47 depending on split), especially in Numerical (+40.9 pp), Trajectory Affordance (+22.6 pp), and Ordering (+17.0 pp) categories, validating the necessity of direct, world-anchored supervision.
- Joint-level specialization (MotionLLM) is insufficient; the model fails especially on aggregate or trajectory-level axes, confirming that conventional pose supervision does not generalize to global movement reasoning.
- Ordering tasks are the toughest, requiring precise sequential integration of event streams, and show only modest gains with SFT.
- Cross-dataset generalization is asymmetric: training on complex, multi-person data (EgoBody) leads to better transfer, while training on temporally diverse (EMDB) or short-range sets (RICH) is less robust.
Critical Analysis and Implications
HumanMove VQA exposes a fundamental weakness in current MLLM architectures—the inability to reason over world-consistent, long-horizon human motion. This deficit is not intrinsic; instead, it reflects domain gaps in training supervision and the inadequacy of canonical-coordinate, joint-level or event-centric benchmarks. The benchmark’s design, focusing on first-frame anchored world coordinates and controlled distractors, prevents both language shortcutting and overfitting to dataset biases.
The successful SFT of Qwen3-VL 8B on this benchmark establishes that MLLMs can learn structured, geometric reasoning about human movement if—and only if—systematic world-referenced supervision is provided. However, such learning does not transfer to joint-level action understanding (as evaluated on ActionArt): trajectory- and articulation-level reasoning must be explicitly decoupled and targeted.
From a practical standpoint, models trained on HumanMove VQA have potential applicability in robotics (trajectory tracking), sports biomechanics, ergonomics, and safety analytics. The failure of large proprietary models on these axes indicates significant limitations for their deployment in any application requiring fine spatial-temporal understanding of human actors.
Limitations and Future Directions
While comprehensive, the benchmark’s reliance on pose-tracking from monocular videos imposes a noise floor, particularly for small, subtle rotations and short events. The current scope is limited to single-person reasoning in exocentric views, although the pipeline could be extended to social interaction and egocentric domains. Furthermore, reasoning trace supervision (as opposed to option-only QA) did not yield improvements, possibly due to the increased cognitive and representational burden on the model.
To bridge the reasoning gap observed, future research should consider the integration of:
- Graph- or sequence-based temporal aggregation modules,
- Explicit world-referencing architectures and loss functions,
- Multi-level supervision combining articulation and trajectory,
- Integration of self-supervised spatial priors learned from large-scale, world-anchored human motion datasets.
Conclusion
HumanMove VQA defines a new standard for evaluating and developing Video MLLMs with authentic global motion reasoning ability. The evidence demonstrates that current models—even at massive scales—rely heavily on event labels or local spatiotemporal patterns, and lack the geometric foundations to reason about physical trajectories. The benchmark and results underscore the necessity for more structured, physically grounded model supervision, and open new directions for the geometric and spatial grounding of multimodal intelligence (2606.27999).