MMHU: Multimodal Behavior Benchmark

• MMHU is a massive-scale benchmark that offers detailed human motion trajectories, hierarchical text captions, and safety-critical behavior labels for autonomous driving.
• The dataset employs a human-in-the-loop annotation scheme that combines automated vision-language modules with rigorous manual quality assurance for high annotation fidelity.
• MMHU supports key tasks like motion prediction, generation, behavior VQA, and intention prediction, with well-defined metrics to facilitate model benchmarking and transfer learning.

MMHU (Massive-Scale Multimodal Benchmark for Human Behavior Understanding) is a comprehensive and large-scale dataset designed to facilitate rigorous research in the analysis and benchmarking of human behavior within autonomous driving contexts. MMHU offers rich, multi-type annotations, including high-fidelity human motion trajectories, hierarchical text descriptions, critical behavior and intention labels, and is structured to support multiple core evaluation tasks in human-centered driving scenarios (Li et al., 16 Jul 2025).

1. Dataset Composition and Scale

MMHU comprises a diverse and substantial collection of monocular driving-scene video clips, prioritizing annotations relevant to safe autonomous navigation. The dataset includes:

• 57,000 unique human motion clips distributed over 1.73 million video frames, amounting to 48 hours of annotated video at 10 Hz (mean clip length: 3.01 s; median: 2 s).
• Data sources:
  • Autonomous driving datasets: e.g., Waymo Open Dataset (1.7 h, 73,000 frames).
  • In-the-wild videos: YouTube (CC-licensed, ~10 h, 318,000 frames).
  • Self-collected/paid recordings: 66.5 h, ∼2.39 million raw frames, filtered to contribute to the final 1.73 million.
• Splits:
  • MMHU-V: 47,000 VLM-labeled instances (used for pretraining).
  • MMHU-H: 9,500 human-labeled instances (approx. 10% of data; for fine-tuning).
  • MMHU-T: 840 held-out test instances.
  • Usage: MMHU-V and MMHU-H combined for training, MMHU-T for evaluation.

This large scale and comprehensive source diversity establish MMHU as a unified platform for studying human behavior in diverse traffic environments (Li et al., 16 Jul 2025).

2. Annotation Architecture

A human-in-the-loop annotation scheme is central to MMHU. The pipeline combines automated vision-language modules for scalable labeling and rigorous spot-checking by human annotators:

• Quality assurance: Randomly selected 10% of all clips are manually labeled. These are used to fine-tune the vision-LLM (VLM) annotator, which is then reapplied to the remaining data. If a VLM label violates format constraints, up to three automated roll-backs are attempted; persistent failures are escalated for human review.
• Annotation modalities:
  • Human motion and trajectory:
  • Represented via SMPL parameters ($S_t \in \mathbb{R}^{n \times m}$) per frame and global 3D joint positions ($P_t = \{p_k \in \mathbb{R}^3\}_{k=1}^n$), sampled at 10 Hz.
  • Temporal segmentation differentiates input (past) motion $S_p$ and prediction target (future) $S_f$.
  • Missing-frame completion utilizes spherical linear interpolation.
  • Hierarchical text captions:
  • Low-level: Short (~15 words), joint-wise descriptions generated via rule-based PoseScript extraction and aggregation.
  • High-level: Longer (~33 words), semantic summaries produced by VLMs over 8 sampled frames plus the low-level text.
  • Critical behavior and intention labels: Spanning a taxonomy of 13 binary behaviors relevant to safety (e.g., “crossing street,” “carrying items,” “riding bike”). VLM-based answers to yes/no QA prompts, with human fallback as needed.

3. Benchmarked Tasks and Associated Metrics

MMHU is designed to benchmark model performance on four distinct, high-impact tasks crucial for autonomous vehicle perception and reasoning:

Task	Input/Output	Main Metrics
Motion Prediction	Past poses $\rightarrow$ future poses	MPJPE, ACCL
Motion Generation	Text $\rightarrow$ motion	FID, Multi-modality
Behavior VQA	Images $\rightarrow$ behavior answers	Accuracy, F1
Intention Prediction	Images $\rightarrow$ intent (crossing)	Accuracy, F1, AuROC

• Motion prediction: MPJPE (mean per joint position error) and ACCL (average $\ell_2$ acceleration error) assess pose forecasting accuracy.
• Motion generation: FID measures the statistical similarity between real and generated motion, and multi-modality quantifies sample diversity for fixed prompts.
• Behavior visual question answering (VQA): Thirteen-class close-ended VQA is evaluated by accuracy and F1 score computed over binary answers.
• Intention prediction: Specialized version of Behavior VQA for “crossing street,” evaluated with accuracy, F1, and Area Under ROC (AuROC).

4. Empirical Benchmark Results

Results reported on MMHU-T and external datasets demonstrate both baseline model performance and the beneficial effects of fine-tuning on MMHU:

• Motion Prediction (MPJPE in cm, lower is better):
  • PhysMoP: 0.4 (frame 1), 1.7 (f3), 9.0 (f7), 14.4 (f9), 26.3 (f13), 36.2 (f17), 45.3 (f21), 54.3 (f24).
  • AuxFormer: 17.0, 32.7, 47.8, …, 86.1.
  • CIST-GCN: 18.5, 25.3, 37.2, …, 47.4.
• Motion Generation:
  • Real motions: FID ≈ 0.002.
  • MotionDiffuse: FID ≈ 39.28, Multi-modality ≈ 2.36.
  • MotionGPT: FID ≈ 27.06, Multi-modality ≈ 5.42.
• Behavior VQA (Micro-F1, 13 classes):
  • GPT4o-mini: 64.8%.
  • Mantis-8B-SigLIP: 58.4%.
  • MiniCPM-o-2_6: 52.2%.
  • Qwen2-VL-7B: 52.1%.
• Cross-task Benefits:
  • On 3DPW (motion prediction), finetuning reduces MPJPE from 47.67 cm to 38.18 cm; ACCL from 3.8 to 2.7.
  • For Motion Generation (MMHU split), MotionDiffuse: FID reduces from 39.27 to 1.86; MotionGPT: 27.06 to 8.44.
  • Behavior VQA (MMHU-T), Qwen2.5-VL accuracy increases from 35.31% to 67.77%, F1 from 44.72% to 68.54%.
  • Intention prediction (JAAD): accuracy increases from 84.49% to 91.89%, F1 from 84.45% to 91.89%, AuROC from 92.98% to 97.72%.

These results substantiate MMHU’s utility for fostering progress and transferability across multimodal human-centric understanding tasks (Li et al., 16 Jul 2025).

5. Biases, Limitations, and Annotation Challenges

Several critical challenges and limitations are observed in MMHU:

• Class imbalance: Certain behaviors (e.g., “crossing street”) are frequent, whereas others (e.g., “using wheelchair”) are rare, potentially leading to biased learning and reduced generalization for underrepresented behaviors.
• Monocular input constraints: Reliance on monocular video data hinders depth estimation and occlusion reasoning, thereby restricting the full understanding of complex spatial dynamics.
• Annotation fidelity: The pipeline’s automation is constrained by VLM reliability and the robustness of QA-format parsing. Persistent failures in annotation formatting are flagged for human review, indicating limitations in current VLM architectures for precise, large-scale behavior labeling.

A plausible implication is that for deployment in robust, domain-diverse driving systems, augmentation with additional sensor modalities (e.g., LiDAR, radar) and advances in VLM robustness are imperative.

6. Open Questions and Future Trajectories

Ongoing and future research avenues related to MMHU include:

• Taxonomy expansion: Extending the range of annotated behaviors to include fine-grained intentions (e.g., “waiting,” “glancing”) and complex multi-agent interactions.
• Multimodal integration: Fusing additional sensory modalities such as LiDAR, radar, and audio to enrich behavioral cues beyond monocular vision.
• Real-time deployment: Optimizing annotation and inference pipelines for efficient on-vehicle operation, specifically focusing on the computational efficiency of joint motion reconstruction and VQA.
• Scaling annotation automation: Investigating smaller, more robust VLMs to alleviate the need for manual labeling and to enhance reliability and scalability of in-domain annotation.

This suggests MMHU will continue to serve as a foundational resource for multimodal research in human behavior understanding, particularly within the autonomous vehicle research domain.

7. Significance and Broader Context

MMHU addresses the previously unmet need for a unified, large-scale, richly annotated benchmark dedicated to human behavior analysis in the context of autonomous driving. By combining 3D motion data, hierarchical semantic descriptions, and explicit safety-critical behavior labels, MMHU enables systematic evaluation and cross-task transfer for advanced perception and behavior understanding pipelines. The dataset’s scale and methodological rigor set a new precedent in the benchmarking of human-in-the-loop and machine-annotated datasets, while its recognized limitations highlight open challenges and opportunities for future research in multimodal machine perception and behavior recognition (Li et al., 16 Jul 2025).