Waymo-3DSkelMo: 3D Skeletal Motion Benchmark

• Waymo-3DSkelMo is a multi-agent 3D skeletal motion dataset offering high-fidelity pedestrian pose estimation and forecasting in autonomous driving.
• The dataset fuses LiDAR and camera data through SMPL mesh recovery, slerp interpolation, and Neural Motion Field optimization for dense and accurate 3D annotations.
• Multi-modal frameworks and detailed interaction semantics enable significant performance improvements in pose estimation and forecasting, with up to a 31% reduction in MPJPE.

Waymo-3DSkelMo is a large-scale, multi-agent 3D skeletal motion dataset and associated multi-modal estimation framework, designed for high-fidelity human motion perception and forecasting in the context of autonomous driving. Originating from raw Waymo Open Dataset LiDAR and camera data, it provides temporally coherent, interaction-rich 3D pose annotations and benchmarks that underpin a variety of state-of-the-art learning algorithms for pedestrian understanding, forecasting, and multi-modal sensor fusion in real-world driving scenarios (Zhu et al., 13 Aug 2025, Bauer et al., 2023, Zhu et al., 9 Feb 2026).

1. Dataset Construction, Scope, and Statistics

Waymo-3DSkelMo spans 14,419 seconds (approximately 4 hours) of urban driving, encompassing 837 distinct scenarios from the Waymo Open Dataset. Each scene contains, on average, 27.1 pedestrians (peaking at 250 agents), with 3D poses densely annotated at 10 FPS for a total of 2,438,145 skeleton instances (Zhu et al., 13 Aug 2025). This scale and density markedly surpass the earlier Waymo v2.0, which offered only ≈10,000 manually labeled keypoints.

The dataset constructs dense pedestrian pose sequences from five synchronized LiDAR sensors via a pipeline that involves:

• Point Cloud Extraction: Fusing raw LiDAR range images using built-in extrinsic calibration.
• Parametric Mesh Recovery: Each pedestrian's partial point cloud is fit to an SMPL (Skinned Multi-Person Linear Model) mesh via LiDAR-HMR, yielding robust pose estimates even under significant occlusion.
• Spatiotemporal Alignment: Spherical linear (slerp) and linear interpolation fill missing rotations/translations to upsample from 10 to 30 FPS, followed by Frenet-frame normalization to ensure global trajectory consistency.
• Motion-Field Prior Optimization: A Neural Motion Field (NeMF), pretrained on the AMASS MOCAP corpus, further smooths and regularizes per-agent motion by minimizing a sum of L1 losses over joint rotations, local positions, root orientation, and root translation, heavily anchoring to available Waymo GT key frames.

Final outputs are provided as both per-joint 3D keypoints (skeletons) and full SMPL poses, downsampled to 10 FPS to match Waymo’s native temporal resolution.

2. Interaction Semantics and Multi-Agent Annotation

Waymo-3DSkelMo annotates explicit interaction semantics between agents, critical for modeling realistic roadside behavior:

• Taxonomy of interactions: Approach, Yield, Cross, and Group Walking.
• Labeling pipeline:

1. Automatic Discovery: Candidate events are identified via spatiotemporal clustering, leveraging proximity (<1.5 m) and relative velocity-angle features.
2. Semantic Assignment: Heuristics classify event types based on relative velocities and trajectory curvature.
3. Manual Refinement: Human raters verify and correct events using synchronized LiDAR and RGB visualization.

Every pedestrian and agent pair is thus linked with event markers and semantic interaction classes at the frame level, supporting research in multi-agent scene understanding, behavior prediction, and downstream planning.

3. Multi-Modal 3D Pose Estimation Frameworks

Several architectures leverage Waymo-3DSkelMo for 3D pose estimation, notably exploiting both LiDAR and RGB modalities:

• Three-Branch Late Fusion (AlphaPose+PointNet):
  • 2D Branch: Monocular RGB crops are processed by a pretrained 2D pose estimator (e.g., AlphaPose) generating $\mathbf{X} \in \mathbb{R}^{N_J \times 2}$ keypoints and confidences.
  • LiDAR Branch: Cropped LiDAR points within 3D bounding boxes are encoded via a PointNet-style MLP to yield global features $f_p$ and individual pose predictions $\mathbf{Y}_p \in \mathbb{R}^{N_J \times 3}$.
  • Fusion Layer: Late fusion either concatenates or linearly weighs predictions (with learned fusion weights $w_{2D}$, $w_{p}$), outputting final skeletons $\mathbf{Y}$.
  • Pseudo-Label Generation: 2D joints are matched to the M closest LiDAR points in projected image space, then “soft-selected” using a spatial exponential weighting for robust 3D pseudo-label assignment
  • Loss Functions: Weakly supervised learning is performed via a reliability-weighted MPJPE on pseudo-labels. Supervised MPJPE is also supported where GT is available (Bauer et al., 2023, Zheng et al., 2021).
• Cascaded Heatmap-PointNet and Segmentation:
  • Heatmap Extraction: ResNet-50 backbone produces K keypoint heatmaps from cropped RGB.
  • LiDAR Augmentation: LiDAR points are augmented with sampled RGB heatmap vectors, then globally encoded and segmented with an auxiliary classification head.
  • Objective: Minimizes heatmap L2, smooth-L1 pseudo-3D regression, and cross-entropy segmentation losses.

Performance benchmarks demonstrate that late-fusion models yield substantial accuracy gains over single-modality approaches (up to 31% MPJPE reduction compared to 2D-only lifting), and that segmentation and semantic branches further boost per-joint accuracy, especially for distal joints (e.g., wrists, elbows, ankles) (Zheng et al., 2021, Bauer et al., 2023).

4. Benchmarks and Quantitative Results

Waymo-3DSkelMo enables both estimation and forecasting benchmarks across diverse multi-agent contexts:

• Pose Estimation:
  • On the Waymo Open Dataset, multi-modal fusion achieves MPJPE ≈ 8.58 cm (supervised; late-fusion), outperforming 2D lifting (12.52 cm) and LiDAR-only (11.22 cm).
  • Weak supervision with pseudo-labels achieves MPJPE as low as 9.01 cm, down from 12.14 cm with basic approaches.
• Multi-Agent Pose Forecasting:
  • Tasks: Observing 2 s (50 frames at 25 FPS), predict the next 1 s (25 frames), with agent densities from 2 to 5.
  • Metrics: Joint Position Error (JPE), Aligned Position Error (APE), and Final Displacement Error (FDE).
  • Results (TBIFormer baseline):
  • 3-agent clips achieve lower JPE/APE/FDE than both 2-agent and 5-agent settings (e.g., overall JPE drops from 326 mm at 2 agents to 172 mm at 3 agents), highlighting the benefit of context (Zhu et al., 13 Aug 2025).

A summary of example results:

Setting	MPJPE (cm)	APE (mm)	FDE (mm)
2 agents (JPE)	32.6	109	287
3 agents (JPE)	17.2	125	110

5. Extensions: Vehicular Context and Conditional Forecasting

The dataset has been extended with vehicle context to support modeling pedestrian–vehicle interactions (Zhu et al., 9 Feb 2026):

• 3D Vehicle Bounding Boxes: Direct alignment using the Waymo Open Dataset’s vehicle boxes; tracks with poor coverage are filtered.
• Scene Sampling: Temporal segments are sampled by pedestrian and vehicle count with spatial thresholds (e.g., up to four vehicles within 15 m), creating a range of interaction complexities.
• Forecasting Models: TBIFormer-based models are enhanced with a vehicle encoder and a Pedestrian–Vehicle Interaction Cross-Attention (PVI-CA) module, allowing pedestrian queries to attend to temporally encoded vehicle edge-group features.
• Performance: Vehicle-conditioned models achieve up to 21% reduction in MPJPE compared to pedestrian-only forecasting, depending on agent densities and vehicle arrangement.
• Ablation: Naive inclusion of vehicle centroids hurts joint prediction, while dedicated vehicle encoding and trajectory-aware cross-attention are consistently effective.

6. Usage, Access, and Research Best Practices

Waymo-3DSkelMo (data and code) is released under CC-BY-NC-SA 4.0 (matching Waymo’s non-commercial license). The dataset includes:

• Python scripts for point cloud processing and mesh recovery.
• Official implementations for SMPL fitting, NeMF motion refinement, benchmarking, and interaction annotation.
• Dense skeleton and mesh files at both 10 and 30 FPS.
• Event-based interaction annotations in JSON.
• Best practice guidelines: Align with the 10 FPS split for cross-sensor analysis, use mesh or skeleton representations as appropriate, follow provided agent-density splits for consistent benchmarking, and exploit provided interaction labels for multi-agent behavior modeling (Zhu et al., 13 Aug 2025).

A plausible implication is that Waymo-3DSkelMo represents a foundational benchmark for multimodal scene understanding and prediction in highly interactive, real-world traffic scenarios, bridging perception and prediction disciplines across AV research and related fields.