- The paper introduces a vehicle-conditioned 3D pose forecasting model that extends TBIFormer with a vehicle encoder and pedestrian–vehicle cross-attention using aligned Waymo LiDAR data.
- Vehicle context improves MPJPE by roughly 1.5%–21% and produces the largest gains in complex scenes, including about 15.9% lower MPJPE with three pedestrians and up to 18% improvement as vehicle count increases.
- The results show that structured cross-attention is effective while naively encoding vehicle centers as body parts worsens performance, and distant vehicles can add conditioning noise.
Overview
This paper addresses 3D pedestrian pose forecasting in autonomous driving scenarios by explicitly conditioning predictions on surrounding vehicle information. The authors make three contributions: (1) an extension of the Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes and a scene-level sampling scheme that categorizes interactions by pedestrian and vehicle count; (2) a vehicle-conditioned forecasting network built on TBIFormer that fuses pedestrian and vehicle features via a dedicated cross-attention module; and (3) extensive benchmarking demonstrating consistent, though sometimes modest, accuracy gains from incorporating vehicles. The work targets a genuine gap: prior interaction-aware predictors operate on 2D coordinates or ego-centric imagery, while existing 3D pose forecasting methods consider only pedestrian–pedestrian interactions.
Dataset construction and scene segmentation
The starting point is Waymo-3DSkelMo, which contains 2,438,145 reconstructed 3D skeletal poses derived from raw LiDAR range images using LiDAR-HMR with SMPL body models and NeMF motion priors for naturalness. Because this data is spatially and temporally aligned with Waymo's perception annotations, the authors can attach each vehicle's 3D bounding box to the pedestrian skeletons in a common coordinate frame, yielding true multi-agent pedestrian–vehicle scenes rather than synthetic compositions or RGB-estimated poses.
A KDTree-based segmentation pipeline groups pedestrians by proximity, retaining scenes of one to three pedestrians — the counts above this range decline sharply. Segments are kept only if the minimum-over-time maximum pairwise pedestrian root distance R is below 18 m, matching the spatial scale of TBIFormer's original CMU-Mocap (UMPM) training data. Vehicles are selected if their average distance to the nearest pedestrian over the time window falls below a threshold; a 0–15 m range with up to four vehicles was chosen as a practical trade-off between training set size and meaningful interaction, producing twelve experimental conditions (1–3 pedestrians × 1–4 vehicles). The authors concede that this segmentation limits the ability to study richer pedestrian–pedestrian interactions, and that the threshold choice is acknowledged as a heuristic to be justified more rigorously in future work.
Network architecture
The forecasting problem is formulated in displacement space: observed pedestrian poses and vehicle bounding boxes (represented by their eight corners) are converted to frame-to-frame displacements, compressed via DCT to discard high-frequency components, and the model predicts future pedestrian displacements that are recovered through IDCT. Vehicle motion serves purely as a conditioning signal; no future vehicle trajectory is predicted, and the training loss is an MPJPE-based reconstruction objective on pedestrian displacements alone.
The architecture extends TBIFormer, preserving its Trajectory-Aware Relative Position Encoding (TRPE) and Temporal Body Partition Module on the pedestrian branch. A new vehicle encoder processes the displacement sequences of bounding-box corners, partitioned into groups analogous to body parts — the full model uses the 12 edges of the box as groups. The Pedestrian–Vehicle Interaction Cross-Attention (PVI-CA) module treats pedestrian features as queries and vehicle features as keys/values, with a TRPE-derived contextual bias BTRPE​ injected into the attention logits so that each pedestrian attends selectively to trajectory-relevant vehicles. A standard Transformer decoder generates future motion from global body query tokens conditioned on the fused representation.
Experimental results
Since no prior method performs vehicle-conditioned 3D pose forecasting, TBIFormer trained on pedestrians only serves as the baseline under identical data splits. Evaluation uses MPJPE, APE (root-aligned local pose error), and FDE at 0.2 s, 0.6 s, and 1.0 s horizons after 2 s of input.
Three findings stand out:
Consistent gains from vehicle context. Including vehicles improves all metrics across nearly all conditions, with relative MPJPE improvements ranging from roughly 1.5% to 21%, APE improvements of 1.8%–12%, and FDE improvements of 0.7%–15%. The single reported exception is the one-pedestrian scenario with four vehicles, where 1.0 s performance slightly degrades — attributed not to model capacity but to distant, less relevant vehicles introducing noise into conditioning.
Gains grow with pedestrian count. In the one-pedestrian, one-vehicle case, average MPJPE drops only ~2.9% (211 → 205 mm), whereas the two-pedestrian case improves ~6.4% (213 → 200 mm) and the three-pedestrian case improves ~15.9% (236 → 199 mm). This suggests vehicle context becomes more informative as scene complexity increases, since multi-pedestrian scenes admit richer interaction structure.
Gains grow with vehicle count. In the two-pedestrian scenario, overall MPJPE improvement rises steadily from ~6% with one vehicle to ~18% with four vehicles. The authors note that absolute metrics across different vehicle counts are not directly comparable due to differing underlying training data, so these comparisons rest on relative improvements within each condition.
The ablation studies, conducted on the two-pedestrian/three-vehicle subset, yield a notable negative result: naively treating vehicle centers as pseudo body parts within vanilla TBIFormer worsens all metrics by roughly 2.5% relative to the pedestrian-only baseline, indicating that crude vehicle injection is counterproductive. A dedicated vehicle branch with plain cross-attention cuts MPJPE and FDE by about 10.7% and 13.4% respectively, and PVI-CA with TRPE adds further marginal gains (MPJPE 218.8 → 194.8 mm versus the TBIFormer baseline of 218.8 mm). Performance is remarkably insensitive to corner grouping granularity — configurations from 1 to 12 groups differ by well under 1 mm — so the choice of 12 edge groups is justified mainly on geometric expressiveness grounds rather than measurable accuracy differences.
Qualitative results corroborate the quantitative trend: at short horizons both models track ground truth comparably, while at 0.6 s and 1.0 s the baseline drifts ahead of or lags behind the true trajectories, whereas the vehicle-conditioned model follows them more closely.
Limitations and open questions
Several caveats bear directly on interpreting the results. The improvements in single-pedestrian scenes are small (often under 3%), and one configuration shows degradation, indicating vehicle context is not uniformly beneficial. The 15 m vehicle selection radius and 18 m pedestrian spread thresholds are empirical choices; the paper itself states the trade-off "will be further justified with evidence from prior studies in future work." The framework omits static scene structure and non-vehicle agents such as cyclists, so the reported gains bound only what vehicle geometry contributes. Finally, because the skeletal data is reconstructed from LiDAR via learned pose estimation and motion priors, forecast quality inherits any reconstruction artifacts, and the paper does not quantify how reconstruction error propagates into forecasting metrics. An open question is whether PVI-CA-style conditioning generalizes to heterogeneous agent sets beyond rigid bounding-box representations.
Conclusion
The paper establishes a concrete benchmark and architecture for vehicle-conditioned 3D pedestrian pose forecasting on real autonomous driving data, showing that explicit, structure-aware fusion of vehicle bounding boxes yields reliable — if scenario-dependent — accuracy gains over pedestrian-only baselines, with the largest benefits in multi-pedestrian, multi-vehicle scenes. The negative result for naive vehicle-as-pedestrian encoding and the insensitivity to corner grouping provide useful design guidance for follow-up interaction modeling work.