HABIT: Human Action Benchmark for Interactive Traffic in CARLA

Abstract: Current autonomous driving (AD) simulations are critically limited by their inadequate representation of realistic and diverse human behavior, which is essential for ensuring safety and reliability. Existing benchmarks often simplify pedestrian interactions, failing to capture complex, dynamic intentions and varied responses critical for robust system deployment. To overcome this, we introduce HABIT (Human Action Benchmark for Interactive Traffic), a high-fidelity simulation benchmark. HABIT integrates real-world human motion, sourced from mocap and videos, into CARLA (Car Learning to Act, a full autonomous driving simulator) via a modular, extensible, and physically consistent motion retargeting pipeline. From an initial pool of approximately 30,000 retargeted motions, we curate 4,730 traffic-compatible pedestrian motions, standardized in SMPL format for physically consistent trajectories. HABIT seamlessly integrates with CARLA's Leaderboard, enabling automated scenario generation and rigorous agent evaluation. Our safety metrics, including Abbreviated Injury Scale (AIS) and False Positive Braking Rate (FPBR), reveal critical failure modes in state-of-the-art AD agents missed by prior evaluations. Evaluating three state-of-the-art autonomous driving agents, InterFuser, TransFuser, and BEVDriver, demonstrates how HABIT exposes planner weaknesses that remain hidden in scripted simulations. Despite achieving close or equal to zero collisions per kilometer on the CARLA Leaderboard, the autonomous agents perform notably worse on HABIT, with up to 7.43 collisions/km and a 12.94% AIS 3+ injury risk, and they brake unnecessarily in up to 33% of cases. All components are publicly released to support reproducible, pedestrian-aware AI research.

• The paper introduces HABIT, a benchmark that integrates thousands of real-world pedestrian motions into CARLA for nuanced evaluation of autonomous driving systems.
• It employs a scalable motion processing pipeline with semantic filtering, visual verification, and retargeting to ensure biomechanical and physical realism.
• Evaluation reveals that state-of-the-art planners experience increased collision rates and injury risks, challenging traditional safety metrics based on scripted benchmarks.

HABIT: A High-Fidelity Benchmark for Realistic Pedestrian Behavior in Autonomous Driving Simulation

Motivation and Context

Autonomous driving (AD) systems face significant challenges in modeling and forecasting human behavior, particularly pedestrian dynamics that are inherently unpredictable and context-dependent. Most existing simulation frameworks emphasize photorealistic rendering and high-fidelity sensor modeling but employ simplistic, rule-based scripts to control pedestrian agents. This limitation fosters development of planners and predictors that generalize poorly to real-world conditions, where pedestrian intent, trajectory, and interactions are diverse and ambiguous. The narrow focus on binary collision metrics further underestimates system risks, as it fails to consider the severity of potential injuries or the overcautious actions of planners.

HABIT (Human Action Benchmark for Interactive Traffic) directly addresses these deficits by integrating thousands of real-world human motion sequences into the CARLA simulator via a modular pipeline that ensures biomechanical and physical realism. Through this approach, HABIT enables rigorous evaluation and diagnosis of AV perception, prediction, and planning modules in the presence of semantically rich, context-aware pedestrian behaviors.

Motion Processing Pipeline

HABIT's core innovation is a scalable pipeline that converts heterogeneous human motion data—sourced from AMASS motion capture repositories and open-source videos—into physically and temporally aligned SMPL-based representations. This ensures compatibility and seamless retargeting to the CARLA simulation environment.

Figure 1: Overview of the HABIT motion data processing pipeline, highlighting its extensibility via video-based motion extraction.

The pipeline uses:

• Text-based semantic filtering: NLP techniques process natural language descriptions to select candidate motions relevant for traffic contexts, drastically narrowing down a 30,000-motion pool to ~5,100 traffic-appropriate clips.
• Visual and kinematic verification: A structured, human-in-the-loop validation stage removes irrelevant or egregiously non-traffic-like motions.
• 6D trajectory and pose reconstruction: Conversion of pose sequences to continuous 6D root orientation vectors and translation velocities, which are cumulatively integrated to reconstruct temporally and physically consistent global pedestrian trajectories.

  Figure 2: Comparison of global root rotation over time, illustrating the physically grounded, temporally coherent trajectory reconstruction achieved by HABIT.

• Motion retargeting: Addressing discrepancies between SMPL's right-handed axis-angle joint representation and CARLA's left-handed Euler angles, including skeleton mapping, axis flipping, and pose offset corrections to preserve accurate articulation.

Figure 3: Canonical pose comparison between SMPL and CARLA skeletons, underscoring the complexity in retargeting for accurate simulation.

Benchmark Design and Scenario Generation

The curated set of 4,730 traffic-compatible motions is annotated with behavioral categories and integrated into a scenario generation pipeline within CARLA. The benchmark encompasses 110 routes, 12 weather conditions, 30 vehicles, and up to 30 pedestrian agents per scenario—supporting rigorous, systematic evaluation of planning modules in a broad array of urban traffic situations.

Figure 4: The scenario generation framework for HABIT, demonstrating retargeted motion placement and pedestrian control within CARLA scenes.

HABIT stratifies pedestrian actions into three core classes—Not Crossing, Attempting, and Crossing—based on forward displacement and semantic context. This supports controlled studies of rare events and nuanced social behaviors.

Figure 5: Distribution of motion tags across behavioral categories, reflecting the diversity and balance of the curated benchmark.

Figure 6: Trajectory statistics by behavior type, highlighting distinct mean and variance profiles for each category.

Evaluation Metrics and Analysis

Traditional collision-based metrics inadequately capture safety-critical nuances, such as the severity of injuries or overconservative braking. HABIT supplements CARLA’s standard metrics with:

• Abbreviated Injury Scale (AIS 3+): Employing a logistic regression model to estimate the risk of serious pedestrian injury based on collision speed, enabling clinically meaningful assessment of system harm potential.
• False Positive Braking Rate (FPBR): Quantifies the proportion of unnecessary braking events triggered by inaccurate or conservative pedestrian intent forecasts, exposing systemic conservatism in planning modules.

Experimental Results and Planner Diagnostics

Evaluation of leading end-to-end planners—InterFuser, TransFuser, and BEVDriver—on HABIT reveals substantially increased collision rates and injury risks compared to results on scripted benchmarks, despite similar or perfect CARLA Leaderboard scores.

Key findings include:

• InterFuser, which records near-zero collisions on CARLA Leaderboard, exhibits up to 5.24 collisions/km and 10.96% injury risk (AIS 3+) under HABIT.
• TransFuser, while less conservative (lower FPBR), incurs more frequent collisions and higher displacement errors on ambiguous or rare behaviors.
• BEVDriver lacks pedestrian intent forecasting capabilities, limiting its sensitivity to the full spectrum of scenarios.

Contradictory to prior benchmark claims of safety, all agents display substantial performance degradation and planner-specific weaknesses in the presence of realistic pedestrian behaviors.

Diagnosis of Failure Modes

Ablation studies expose characteristic pathologies in state-of-the-art methods:

• Overcautious Braking: In idle-only pedestrian settings, high FPBR values reflect systematic misinterpretation of static humans as threats, resulting in unproductive halts.

  Figure 7: Visualization of false positive braking over time, exemplifying unnecessarily prolonged halting for a stationary pedestrian.

• Directional Bias: When exposed to crossings from uncommon side initiations (e.g., left-initiated), planners trained on imbalanced data manifest a right-to-left prediction bias, underpinning late braking and missed threats.

  Figure 8: Qualitative illustration of directional prediction bias in planner forecasts, with BEV maps showing systematic errors.

Qualitative and Comparative Benchmarking

HABIT’s realism, diversity, and extensibility are visually and functionally validated through:

Figure 9: A montage from the HABIT Benchmark, demonstrating diverse pedestrian behaviors, lighting, and route configurations.

Figure 10: Overview of prevalent simulators, highlighting deficiencies in pedestrian gesture, skeleton control, and behavioral diversity.

Figure 11: Representative rare behaviors captured in HABIT—gestures, non-linear paths, contextual interactions, and reactive adaptation—showcasing capabilities absent from prior simulators.

Limitations and Future Directions

HABIT’s motion retargeting pipeline is fundamentally bounded by kinematic representation constraints (e.g., Euler angle gimbal lock) and the quality/diversity of upstream motion capture or pose estimation. Further improvements in spatial accuracy for video-sourced motions, generative modeling for motion diversity, and data-driven pedestrian spawning are noted as immediate avenues. The domain’s continued progress depends on tightening the sim2real loop, especially through integration of more sophisticated generative and physically grounded pedestrian models.

Conclusion

HABIT constitutes a significant advancement in pedestrian-aware benchmarking for AD research by embedding authentic, heterogeneous human motion and context-rich behavioral transitions in simulation. Quantitative and qualitative results show that state-of-the-art AD planners, while excelling on prior scripted benchmarks, exhibit critical failure modes when evaluated against HABIT’s real-world-inspired complexity, including increased collision rates, missed injury risks, and conservative reaction patterns. The benchmark’s extensible, open design fosters reproducibility and provides a foundation for the next generation of human-centric AV evaluation and embodied AI research.

(2511.19109)