DriveE2E: Real-to-Sim Autonomous Benchmark

Updated 4 July 2026

DriveE2E is a closed-loop benchmark that integrates 800 dynamic traffic scenarios and 15 real-world digital twins to simulate urban intersections for end-to-end autonomous driving evaluation.
It leverages a real-to-simulation pipeline using multi-view video data and calibrated roadside sensors to extract authentic vehicle, pedestrian, and cyclist trajectories.
The framework evaluates models with metrics like Success Rate and Driving Score, highlighting discrepancies between open-loop and closed-loop performance.

DriveE2E is a closed-loop benchmark for end-to-end autonomous driving that integrates real-world urban-intersection traffic scenarios into the CARLA simulator through a real-to-simulation pipeline with infrastructure cooperation. It combines 800 dynamic traffic scenarios extracted from a 100-hour multi-view video corpus with 15 CARLA-compatible digital twins of real intersections, and evaluates end-to-end models under recorded traffic, weather, lighting, and geometry conditions rather than manually scripted simulator traffic (Yu et al., 28 Sep 2025).

1. Scope and benchmark rationale

DriveE2E was introduced against a specific limitation of existing CARLA-based closed-loop benchmarks: they rely on manually configured traffic scenarios that can diverge from real-world conditions, which limits their ability to reflect actual driving performance. The framework addresses this by extracting dynamic scenarios from real traffic observed by elevated infrastructure sensors and replaying them inside intersection-level digital twins whose appearance and environmental state are matched to the source recordings (Yu et al., 28 Sep 2025).

Within the broader landscape of end-to-end driving evaluation, DriveE2E occupies a distinct position. Open-loop datasets such as WOD-E2E emphasize rare long-tail events and human-aligned trajectory scoring, but remain open-loop only and explicitly note future integration with high-fidelity simulators for closed-loop validation (Xu et al., 30 Oct 2025). DriveE2E instead makes closed-loop execution the central object of study. This suggests that its primary contribution is not a new policy architecture or a new open-loop metric, but a reproducible environment in which end-to-end agents can be evaluated against real logged traffic patterns at complex urban intersections.

A central design choice is that realism is pursued through scenario provenance and environmental fidelity. The benchmark is described as challenging because of diversity in driving behaviors, locations, weather conditions, and times of day at complex urban intersections (Yu et al., 28 Sep 2025). At the same time, its realism is bounded by the simulator design: other traffic participants are replayed from logs rather than reacting online. A plausible implication is that DriveE2E is optimized for evaluating perception-planning-control under realistic exogenous traffic and scene appearance, rather than for studying fully reactive multi-agent negotiation.

2. Real-world scenario extraction

The source data were collected at fifteen complex urban intersections in the Beijing demonstration zone. Each intersection was instrumented with 4 pairs of elevated roadside RGB cameras plus blind-spot views, and the sensors were time-synced and intrinsically/extrinsically calibrated. Over 100 hours of multi-view video at 10 Hz were recorded, together with traffic-light states and timestamped weather/illumination logs (Yu et al., 28 Sep 2025).

The extraction pipeline uses a monocular/multi-view 3D detector and a tracker. Each frame was processed by ImVoxelNet and AB3DMOT, then fused across views to produce per-object trajectories with class labels such as car, pedestrian, and cyclist, along with unique IDs. Trajectories were scored for completeness and rule compliance, and low-quality or heavily occluded sequences were discarded (Yu et al., 28 Sep 2025).

From the resulting database, 800 representative clips were selected to cover eight high-level behaviors, six weather conditions, and the full diurnal range from dawn through night. The behavior categories include examples such as “Making a Left Turn,” “Competing with Other Vehicles,” and “Interaction with Pedestrians/Cyclists.” For each selected clip, the ego vehicle was designated as the one visible throughout the video and exhibiting a variety of maneuvers. Sensor streams comprising multi-camera images, LiDAR scans, and radar points were temporally aligned to 10 Hz, and no synthetic augmentation such as flipping or cropping was applied beyond standard time-interpolation when needed (Yu et al., 28 Sep 2025).

This extraction procedure is consequential for benchmark character. Because scenario selection is conditioned simultaneously on maneuver class, weather, and time of day, the benchmark is not merely a collection of intersection traversals but a stratified set of urban interaction episodes. The additional infrastructure-view coverage also increases observability beyond an ego-only perspective, which later appears in the benchmark’s occlusion ablation.

3. Digital twins and real-to-simulation instantiation

The static environment is built from high-definition maps and on-site appearance references. Vectorized lane centerlines, crosswalks, stop lines, and curb geometries were exported from high-definition maps in an Argoverse-style format and imported into RoadRunner. Satellite imagery and street-view photos were then used for manual refinement of curb alignment, lane widths, and crosswalk painting (Yu et al., 28 Sep 2025).

Buildings and roadside structures were fetched from OpenStreetMap and modeled in Blender so that textures, including façade colors and window shapes, matched actual on-site appearance. Traffic lights and signal poles were added in their real-world positions. The final scenes were exported in CARLA’s town format, yielding 15 static assets, one per intersection, with both geometry and PBR textures. Environmental parameters such as sky color, sun angle, and volumetric fog were configured to reproduce each clip’s logged lighting and weather state (Yu et al., 28 Sep 2025).

Scenario instantiation is then performed on top of these static assets. For a given clip, DriveE2E loads the corresponding CARLA town and spawns each non-ego agent at its recorded initial pose. Each agent is mapped to the closest CARLA blueprint, including vehicle types and pedestrian meshes, and its future path is controlled by log replay so that its $(x,y,z)$ position at simulation time $t$ exactly matches the recorded trajectory. The designated ego vehicle uses a nuScenes-style rig with 1 $\times$ 64-ch LiDAR, 6 $\times$ 900 $\times$ 1600 RGB cameras for 360° coverage, 5 $\times$ 100 m radar, and IMU+GPS. Sensor intrinsics and extrinsics are taken from the real installation and reused in CARLA (Yu et al., 28 Sep 2025).

Synchronization is equally rigid. The real-world time base at 10 Hz is the simulation time base, and all log-replay trajectories and sensor ticks are executed on this clock. No additional domain randomization is applied to dynamics; visual realism derives from the high-fidelity static asset and the real logged weather (Yu et al., 28 Sep 2025). This suggests an intentional separation between environmental fidelity and agent reactivity: the framework reconstructs what happened, rather than sampling from a broader stochastic dynamics model.

4. Task definition, protocol, and metrics

Each scenario defines a route-following task grounded in the original recording. The ego vehicle is tasked with traveling from its recorded start $(x_{src}, y_{src})$ to its recorded goal $(x_{dst}, y_{dst})$ along the original path. The benchmark covers eight top-level behaviors and 14 subtypes, with examples including STR, LFT, RT, COV, IPC, YLW, UT, and STP (Yu et al., 28 Sep 2025).

The evaluation protocol is closed-loop only for the ego vehicle. Ego control is generated by an end-to-end model, while all other agents remain on log replay and are therefore nonreactive. A run is counted as a success if the ego reaches $(x_{dst}, y_{dst})$ within a generous time budget and incurs neither collision nor off-road departure; otherwise the run is a failure (Yu et al., 28 Sep 2025).

DriveE2E reports Success Rate, Collision Rate, Driving Score, and optionally Time-to-Goal. The Driving Score follows CARLA LB V2 and is computed from route completion and infraction penalties: for route $i$ , the completion indicator is $t$ 0 and each infraction contributes a penalty coefficient $t$ 1 (Yu et al., 28 Sep 2025). Weather and time of day are not free simulator settings but are set to match the original logs through CARLA Weather Parameters and sun altitude/azimuth (Yu et al., 28 Sep 2025).

Two features of this protocol are methodologically important. First, it provides a true closed-loop control test rather than an open-loop trajectory-matching score. Second, because the background traffic is replayed rather than behaviorally coupled to the ego, failure modes are dominated by whether the ego can cope with realistic scene complexity under fixed surrounding-agent futures. This differs from open-loop human-preference metrics such as WOD-E2E’s Rater Feedback Score and from world-model-based online evaluators such as WoTE, which score candidate trajectories by predicted future BEV outcomes (Xu et al., 30 Oct 2025, Li et al., 2 Apr 2025).

5. Baseline models and empirical results

Five imitation-learning systems were re-implemented as baselines: AD-MLP, TCP, VAD, UniAD, and MomAD. The 800 scenes were split into 400 train, 200 validation, and 200 test scenes (Yu et al., 28 Sep 2025).

Model	Open-loop Avg L2	Closed-loop SR / DS / Time
AD-MLP	8.36 m	1.0% / 29.01% / 35.1
TCP	2.56 m	10.0% / 48.47% / 32.9
VAD	0.89 m	35.0% / 62.29% / 79.9
UniAD	1.08 m	47.0% / 77.62% / 103.1
MomAD	0.98 m	29.6% / 60.98% / 104.0

The open-loop ranking and closed-loop ranking are related but not identical. VAD attains the best open-loop average $t$ 2 error at 0.89 m, whereas UniAD achieves the highest closed-loop Success Rate and Driving Score at 47.0% and 77.62%, respectively (Yu et al., 28 Sep 2025). The paper explicitly notes that high open-loop $t$ 3 accuracy often correlates with better closed-loop performance, but the ranking can differ, which underscores the necessity of closed-loop benchmarking.

Behavior-wise analysis sharpens the difficulty profile. UniAD’s per-behavior Success Rate ranges from 40–100% on simpler tasks such as STR and STP, but drops to 40–60% on interactive tasks such as COV and IPC. The most substantial degradation occurs in IPC and COV, confirming that realistic multi-agent interaction is a key challenge (Yu et al., 28 Sep 2025).

An ablation on occlusion-filtered scenarios further isolates the contribution of infrastructure-derived traffic. When all traffic participants occluded from the ego cameras are removed, VAD’s Driving Score rises from 62.29% to 65.89% (Yu et al., 28 Sep 2025). This result directly supports the benchmark’s central claim that infrastructure-view reconstruction materially increases task difficulty by preserving traffic participants that an ego-only reconstruction might omit.

6. Interpretation, limitations, and relation to surrounding research

DriveE2E is best understood as an evaluation framework rather than a driving architecture. Contemporary end-to-end driving models such as DriveTransformer, DriveMoE, SafeDrive, DriveSafer, and Fast-dDrive are policy or planning systems; DriveE2E provides a closed-loop environment in which such systems can be stress-tested under real-to-simulation urban-intersection scenarios (Jia et al., 7 Mar 2025, Yang et al., 22 May 2025, Kim et al., 21 Feb 2026, Sural et al., 16 May 2026, Zhang et al., 22 May 2026). This suggests a division of labor in the field: model papers increasingly focus on sparse world models, safety reasoning, MoE specialization, or efficient VLA decoding, whereas benchmark papers increasingly focus on whether those design choices survive closed-loop execution.

The benchmark also clarifies a persistent misconception in end-to-end evaluation: low open-loop error is not equivalent to robust closed-loop driving. That conclusion aligns with a broader trend in the literature. WOD-E2E argues that conventional open-loop metrics can miss the multi-modal and safety-critical character of long-tail scenarios, while DriveE2E shows that even among strong imitation-learning baselines, closed-loop ordering can differ from open-loop $t$ 4 ordering (Xu et al., 30 Oct 2025, Yu et al., 28 Sep 2025). A plausible implication is that future evaluation stacks will combine human-aligned open-loop scoring, world-model-based counterfactual assessment, and real-to-simulation closed-loop replay rather than relying on a single metric family.

A second misconception concerns what “realistic” means in simulator evaluation. In DriveE2E, realism comes from real logged traffic, matched geometry, and matched environmental conditions, but other agents remain nonreactive log replays (Yu et al., 28 Sep 2025). That design improves reproducibility and keeps comparisons controlled, yet it does not amount to a full interactive traffic simulator. The distinction matters when interpreting failure modes on COV and IPC: the benchmark captures complex interactions as they occurred, but does not model how surrounding agents would have adapted to counterfactual ego behavior.

Finally, the term “DriveE2E” is not unique across the literature. In “End-to-End Latency Measurement Methodology for Connected and Autonomous Vehicle Teleoperation,” DriveE2E denotes a hardware-and-software framework for measuring Motion-to-Motion, Glass-to-Glass, and overall End-to-End latency in teleoperated driving, rather than a CARLA benchmark for autonomous driving evaluation (Provost et al., 19 Feb 2026). In the autonomous-driving benchmarking context, however, DriveE2E most specifically refers to the real-to-simulation closed-loop benchmark introduced in 2025 (Yu et al., 28 Sep 2025).