SafeBench: Autonomous Driving Safety Benchmark

• Autonomous Driving SafeBench is a unified benchmark framework that integrates real and synthetic driving scenarios to evaluate ADS safety with data-driven rigor.
• It employs a modular pipeline—comprising data ingestion, scenario extraction, digital twin generation, and coverage optimization—to simulate challenging and diverse driving situations.
• The framework supports pre-certification and regulatory reporting by quantifying safety using metrics such as time-to-collision, failure rate, and comprehensive risk assessments.

Autonomous Driving SafeBench represents a unified framework and set of methodologies for the large-scale, rigorous, and reproducible safety evaluation of autonomous driving systems (ADS). Emerging from the need to move beyond ad hoc mileage-based metrics and fragmented toolchains, SafeBench conceptually integrates real and synthesized critical driving scenarios, digital-twin generation, systematic coverage measurement, and actionable safety criteria into a modular, data-driven pipeline suitable for pre-certification assessment, comparative research, and regulatory reporting (Pathrudkar et al., 2023). It is not a single platform but a reference architecture and methodology instantiated in various forms, notably within the SAFR-AV platform.

1. SafeBench Architecture: End-to-End Workflow

SafeBench, as embodied in SAFR-AV, is organized into four sequential, horizontally scalable modules—each handling a distinct aspect of the scenario generation and evaluation pipeline (Pathrudkar et al., 2023):

1. Data Ingestion Pipeline: Ingests batch or stream multi-sensor data (video, LiDAR, radar, GPS/IMU). Data is shard-indexed by spatiotemporal keys and enriched with basic scene understanding (object detection, tracking, event annotation). Outputs are stored in a hybrid back end: time-series DB for trajectories, document store for events, and relational DB for metadata.
2. Scenario Identifier & Search: Codifies behavioral competencies (e.g., “lane change”, “intersection turn”) as query templates over the metadata store. Efficient sub-second lookup yields candidate time windows satisfying these behavioral patterns.
3. Real2Sim Digital Twin Generation: Each identified scenario is reconstructed as a digital twin, aligning multi-sensor situational awareness data to an OpenDRIVE canonical map. Static and dynamic entities are exported via OpenSCENARIO (v1.1/2.0), enabling high-fidelity replay within standard simulation environments (CARLA, Prescan).
4. SceVar Coverage Optimization: Learns empirical joint probability distributions over scenario parameters (entry speed, turning radius, curvature, etc.) from real data. This defines “logical scenarios” whose random samplers can now densely cover the operational design domain (ODD) by proposing new synthetic variants. Coverage is tracked and additional samples are generated to saturate under-sampled regions.

This pipeline supports the construction of a benchmark suite that is both maximally challenging (via edge-case injection) and representative (through empirical grounding in real-world data).

2. Scenario Extraction and Clustering

At the core of SafeBench’s scenario mining, a two-stage filter-and-cluster process operates over temporal windows of egocentric and neighboring trajectories. Features extracted per window include relative displacements ($\Delta x, \Delta y$), speed differences ($\Delta v$), heading rates ($\theta_\mathrm{yaw}$), turn flags, and intersection distances (Pathrudkar et al., 2023).

Given a target behavioral competency, windows are boolean-filtered and clustered via Mahalanobis or Euclidean distance in feature space:

$$d(\phi_{i},\phi_{j}) = \sqrt{(\phi_{i} - \phi_{j})^\top \Sigma^{-1}(\phi_{i} - \phi_{j})}$$

A tunable threshold $\tau$ collapses near-duplicates, ensuring that the retained scenario set remains both non-redundant and diverse.

3. Digital Twin Generation for Software-in-the-Loop (SIL) Testing

The Real2Sim module reconstructs scenario digital twins to enable faithful SIL replay with industry-standard simulators.

• Static and Dynamic Fusion: Sensor fusion combines LiDAR-based object detection with camera segmentation and radar clustering, producing temporally aligned 6-DOF tracks and semantic labels.
• Map Alignment: Lacking an HD map, OpenDRIVE lane geometries are inferred from infrastructure trajectories. If present, HD maps are registered using ICP on point clouds.
• OpenSCENARIO Export: All relevant actors/entities are structured into maneuverGroups with precise trigger conditions, e.g., "RelativeDistance$(Ego,NPC) <$ 50 m $\implies$ NPC begins a left turn."
• Scenario Fidelity Measurement: Static map correctness is assessed by Intersection over Union (IoU) with ground truth. Actor responsiveness is benchmarked with Multiple Object Tracking Accuracy (MOTA) or event F1-scores.

This process supports consistent, fine-grained evaluation of any autonomy stack within a scenario’s ODD.

4. Coverage, Statistical Formalism, and Safety Metrics

SafeBench’s scenario space $\Theta$ is modeled by parameter vectors $\theta = (v, \rho, \alpha, \ldots)$ where $v$ is velocity, $\rho$ radius, $\alpha$ yaw rate, etc. SceVar fits joint PDFs $\hat f(\theta)$ using Gaussian mixture models or kernel density estimators:

$$\hat f(\theta) = \frac{1}{N} \sum_{i=1}^N K_h(\theta - \theta_i)$$

Logical scenario coverage is defined as:

$$C = \frac{\int_{\Theta} w(\theta)\, \mathbb{I}\{\theta\in T\}\, d\theta}{\int_\Theta w(\theta) d\theta}$$

A threshold (e.g., $C>90\%$) defines adequate exploration of the parameter space. In practice, $\Theta$ is discretized into bins and occupancy is tallied.

Safety during SIL runs is quantified via:

• Time-to-Collision (TTC):

$$\mathrm{TTC}(t) = \frac{d_{12}(t)}{v_{1\rightarrow2}(t)}$$

• Post-Encroachment Time, Minimum Safe Distances, Lane Departure Flags
• Failure Rate (FR):

$$\mathrm{FR} = \frac{\text{number of failed runs}}{\text{total runs}}$$

Binary failure outcomes and quantiles (e.g., 5th percentile TTC) provide risk assessment across scenario distributions.

5. Benchmark Protocol and Pre-certification Utility

SafeBench enables systematic, transparent, and reproducible evaluation of ADS stacks:

1. Run $N_\mathrm{real}$ concrete scenarios in SIL; record pass/fail, TTC distributions.
2. Sample $N_\mathrm{syn}$ synthetic scenarios until coverage $C \geq C^*$.
3. Aggregate a failure map in $(v, \rho, \alpha)$ space to localize system weaknesses.
4. Report: Coverage Score $C$, Failure Rate $\mathrm{FR}$, worst-case TTC quantiles, and safety frontier plots.

Compared to traditional benchmarks (massive mileage, expert-crafted scenario banks), SafeBench delivers statistically complete scenario suites (coverage-driven sampling), data-driven realism, and adaptive discovery of low-density, high-failure regions in the scenario space (Pathrudkar et al., 2023).

6. Integration with Broader Evaluation Ecosystems

SafeBench’s modular approach has heavily influenced subsequent benchmarking systems, including those emphasizing regulatory alignment, behavioral safety, perception robustness, and closed-loop system integrity:

• Safety2Drive leverages a SafeBench-style structure with 70 regulation-compliant test items, threat injection for natural/adversarial corruptions, and multi-dimensional evaluation (perception + system-level). Comparative analyses highlight SafeBench’s narrower original coverage and lack of closed-loop corruptions or extensive regulatory mapping (Li et al., 20 May 2025).
• Bench2Drive expands scenario diversity, granularity, and fairness via 44 closed-loop corner-case scenarios across 220 routes, but does not integrate coverage-driven augmentation characteristic of SafeBench (Jia et al., 2024).
• Behavioral Safety Assessment applies a two-component SafeBench (Driver Licensing Test and Driving Intelligence Test) for both scenario library-based competence and large-scale, statistically significant crash-rate quantification, establishing a blueprint for third-party AV qualification (Liu et al., 22 May 2025).

7. Limitations, Impact, and Directions

SafeBench, as instantiated in SAFR-AV and related platforms, addresses core deficiencies in previous mileage-based testing and non-standardized scenario generation by offering a formal, coverage-driven, and data-grounded approach. However, simulation gap, dynamic agent modeling deficiencies, and scenario fidelity limit direct deployment as a certification-only tool—further, integration with evolving perception modalities and behavioral/interaction risk models remains ongoing.

Nevertheless, SafeBench methodology closes the loop from real and synthetic data, through principled scenario extraction and logical coverage, to metric-driven benchmarking—constituting a principled reference for pre-certification, comparative research, and regulatory safety assessment in autonomous driving (Pathrudkar et al., 2023).