---
title: Bench2Drive Autonomous Driving Benchmark
url: https://www.emergentmind.com/topics/bench2drive-benchmark
type: topic
---

# Bench2Drive Autonomous Driving Benchmark

Bench2Drive is a standardized, closed-loop benchmarking suite for autonomous driving systems—particularly end-to-end autonomous driving (E2E-AD)—aimed at providing granular, multi-ability evaluation under diverse scenarios, weather conditions, and hardware configurations. It establishes a common data foundation, execution protocol, and metric system for comparing models, hardware accelerators, and integrated pipelines across both simulated and physical platforms.

## 1. Conceptual Framework and Objectives

Bench2Drive centers on fair, realistic, and multi-level testing of E2E-AD systems. Its primary goals are: (i) enabling reproducible, use-case-controlled comparisons for vision-driven autonomous driving tasks; (ii) disentangling model skill across 44 interactive scenarios and 23 weathers on 12 distinct urban layouts; (iii) exposing model weaknesses not captured by open-loop or long-route benchmarks; and (iv) facilitating hardware–software co-design via standardized performance indicators. The framework targets both system-on-chip (SoC) and configurable IP-core accelerators, as well as vision–language models and multi-modal large models when applicable [2406.03877].

## 2. Dataset, Scenario Coverage, and Protocol

### Dataset Composition

Bench2Drive provides a CARLA-based dataset with 2,000,000 annotated frames from 10,000 short clips, sampled uniformly over:

- **44 interactive scenarios:** including cut-in, overtaking, detour, merging, yield, emergency braking, traffic sign compliance, parked obstacles, pedestrian crossings, highway merges, accident blocking, and sequential lane changes.
- **23 weather variants:** ranging from sunny, foggy, rainy, and overcast to night and rare conditions.
- **12 towns:** urban, village, campus, and highway layouts.

Each frame records multi-modal sensor signals: 1× LiDAR 64-beam, 6× forward RGB cameras (900×1600, JPEG quality=20), 5× radar, IMU/GNSS vehicle states, ground-truth 3D bounding boxes, semantic/instance segmentation, HD-maps, traffic light/sign states, and expert RL features.

### Evaluation Routes

The core protocol comprises 220 closed-loop routes: 5 per scenario × scenario × weather × location, each ≈150 m, designed to guarantee skill-specific attribution and to reduce metric variance versus standard CARLA benchmarks.

### Execution Steps

For each route, agents interface directly with raw sensor input, generate trajectory or low-level controls, and interact with dynamic traffic agents. Termination is triggered by infractions or successful route completion, within a fixed time.

## 3. Twofold Benchmark Granularity and Hardware Evaluation

Bench2Drive formally evaluates both macro-level (entire model/task pipeline) and meso-level (feature extractor submodule) performance:

- **Macro-level tasks:** Semantic segmentation (FCN-8s + VGG-16₀.₂₅ backbone, 1920×1080); object detection (SSD + VGG-16₀.₂₅, 1920×1080); action recognition (CNN-LSTM, 120×80 pedestrian patches).
- **Meso-level (submodule):** Four architectures—VGG-16₀.₂₅ (vanilla conv), SqueezeNet (fire modules), MobileNet_v2 (depthwise/inverted residual), SparseNet-40 (dense blocks).

The *meso-level* protocol isolates submodule bottlenecks by benchmarking cutoff points (e.g., conv5_3 layer in VGG-16₀.₂₅), allowing fine-grained cross-validation across heterogeneous hardware backends [2008.10293].

## 4. Performance Metrics and Reporting

Bench2Drive defines a robust set of indicators:

- **Throughput (images/sec):**
  \( \mathrm{Thr} = \frac{N_{\mathrm{inf}}}{T_{\mathrm{total}}} \)
- **Latency (ms/inference):**
  \( T_{\mathrm{latency}} = t_{\mathrm{end}} - t_{\mathrm{start}} \)
- **Energy per inference (mJ):**
  \( E = P_{\mathrm{avg}} \cdot T_{\mathrm{latency}} \)
- **External memory footprint (MB):** peak DRAM, weights+activations.
- **On-chip buffer (MB):** largest SRAM allocation.
- **Hardware–Model Mismatch (%):**
  \( \mathrm{Mismatch} = 100\% - \mathrm{Utilization} \)
  with
  \( \mathrm{Utilization} = \frac{\mathrm{Measured~TOPS}}{\mathrm{Peak~TOPS}} \times 100\% \)
- **CARLA Leaderboard metrics (closed-loop):** Success Rate (SR), Driving Score (DS), route completion ratio (RC), efficiency, comfort, event-specific Skill Score.

System outputs and metrics undergo internal consistency and accuracy cross-validation for each run [2406.03877, 2503.07656, 2008.10293].

#### Sample Results (HWA Benchmarking)
| Model         | Mode      | Thr (img/s) | Lat (ms) | E (mJ) | Mem (MB) | Mismatch (%) |
|---------------|-----------|-------------|----------|--------|----------|--------------|
| VGG-16₀.₂₅   | Unopt     | 38          | 26.3     | 12.5   | 45       | 42           |
| VGG-16₀.₂₅   | Optimized | 72          | 13.9     | 8.1    | 30       | 22           |
| SqueezeNet    | Unopt     | 52          | 19.2     | 9.8    | 25       | 35           |
| SqueezeNet    | Optimized | 95          | 10.5     | 6.3    | 18       | 15           |
| MobileNet_v2  | Unopt     | 45          | 22.2     | 11.7   | 28       | 48           |
| MobileNet_v2  | Optimized | 80          | 12.5     | 7.5    | 20       | 28           |
[2008.10293]

#### Closed-Loop Model Performance (multi-ability breakdown)
| Model               | Success Rate | Driving Score | Merging | Overtaking | Emergency Brake | Mean Ability |
|---------------------|--------------|--------------|---------|------------|-----------------|-------------|
| AD-MLP              | 0.00%        | 18.05%       | 0.00    | 0.00       | 0.00            | 0.00        |
| UniAD-Tiny          | 13.18%       | 40.73%       | 4.11    | 12.50      | 14.54           | 11.94       |
| DriveAdapter*       | 30.71%       | 42.91%       | 29.23   | 20.00      | 34.71           | 38.23       |
| DriveTransformer-LG | 35.01%       | 63.46%       | 17.57   | 35.00      | 48.36           | 38.60       |
| WoTE                | 31.36%       | 61.71%       | —       | —          | —               | —           |
[2406.03877, 2503.07656, 2504.01941]

## 5. Architectural and Evaluation Insights

Bench2Drive provides several architectural and experimental findings:

- **Expert feature distillation** markedly improves closed-loop skill performance.
- **Unified transformer architectures** (task parallelism, sensor and temporal cross-attention) enable superior training stability and multi-scenario robustness [2503.07656].
- **Model-based trajectory evaluation** via BEV world models delivers better real-time candidate scoring, enhancing SR and DS modestly over rule-based selection [2504.01941].
- **Skill attribution**: traffic sign compliance and rule-based give-way tasks are mastered more reliably than interactive maneuvers, which remain brittle even for strong baselines.
- **Open-loop L2 errors do not predict closed-loop driving performance**, highlighting distribution shift and feedback sensitivity.

## 6. Hardware–Model Co-design Recommendations

Bench2Drive supports detailed accelerator selection and optimization guidelines:

- **Large conv nets** (uniform compute) are best on vector accelerators with high SRAM.
- **SqueezeNet/fire modules** map to engines with dedicated 1×1/3×3 conv.
- **Mobile/lightweight nets** (depthwise separable) require channel-parallel architectures and flexible tiling.
- **Dense connectivity** and multi-scale heads demand on-chip scratchpads with dynamic allocation for fast memory reuse; absence leads to bandwidth stalls.
- **LSTMs/recurrent nets** need stateful operator support and irregular control flow compatibility [2008.10293].
- **Benchmark design** mandates hybrid granularity, realistic resolutions, and comprehensive PI collection for cross-validation and mismatch analysis.

## 7. Extensions and Related Benchmarks

Bench2Drive forms the reference baseline for evaluating both hardware accelerator efficacy (original Bosch Deep Learning Hardware Benchmark), general edge-computing platforms (CAVBench) [1810.06659], and emerging frameworks integrating hierarchical capability assessment, hardware–software adaptation, and closed-loop MLLM/VLM evaluation [2508.02028, 2506.09557, 2412.09647]. Its principles—multi-granularity, closed-loop skill isolation, and fair data standardization—have been widely adopted in subsequent benchmarks for autonomous driving.

Bench2Drive thus establishes rigorous, multi-skill, hardware-sensitive benchmarking for autonomous driving, making it central to evaluating real-world readiness and catalyzing advances across model architectures, optimization toolchains, and embedded accelerator design.

Source: https://www.emergentmind.com/topics/bench2drive-benchmark