---
title: CARLA Leaderboard v2.0 Benchmark
url: https://www.emergentmind.com/topics/carla-leaderboard-v2-0
type: topic
---

# CARLA Leaderboard v2.0 Benchmark

CARLA Leaderboard v2.0 is an open simulation-based testing and evaluation environment for autonomous vehicles, built atop the CARLA simulator, and used as a reproducible, extensible benchmark for comparing autonomous driving systems in closed-loop operation. In this setting, an agent must navigate predefined routes in urban and suburban environments, or long routes up to \(10 \times 10\ \mathrm{km}^2\) in the Challenge 2.0 setting, under diverse conditions such as urban, highway, night, fog, and rain, while handling varied scenarios including pedestrians, merges, and unprotected turns [2509.22754][2406.10165]. Across the literature, it functions both as a public competition environment and as a common evaluation platform for motion planning, end-to-end driving, modular stacks, robustness testing, and dataset construction [2511.14876][2509.22754].

## 1. Task structure and benchmark tracks

CARLA Leaderboard v2.0 is organized around route-following under closed-loop control. An agent starts at a fixed point with a sequence of waypoints, given as GPS coordinates, that define the route, and each route can contain multiple scenario segments such as parking exit or obstacle avoidance [2509.22754]. The benchmark is commonly described through two tracks: a **SENSORS Track**, in which agents use simulated sensors such as cameras, lidar, and radar, and a **MAP Track**, in which agents have sensor data plus HD map access [2509.22754].

This route-based structure makes the benchmark simultaneously a planning problem, a perception problem, and a systems-integration problem. Kyber-E2E’s analysis of the 2023 map-track winner emphasizes that Leaderboard 2.0 features more diverse, challenging scenarios and provides no autopilot “expert” for data collection, which changes the development regime relative to earlier CARLA-based work [2405.01394]. In practice, this has encouraged heterogeneous solution classes, from modular stacks with explicit sensing, localization, tracking, and planning, to camera-only end-to-end policies, object-centric planners, and reinforcement-learning-based world-model systems.

The platform is also used beyond its original competition setting. A comparative study of methods from CARLA, nuPlan, and the Waymo Open Dataset selected CARLA Leaderboard v2.0 as a common evaluation platform precisely because it is open source and reproducible, and because its closed-loop protocol exposes failure modes that are not visible in open-loop evaluation [2509.22754]. This usage has made the benchmark a point of contact between otherwise separate planning ecosystems.

## 2. Scenario design, route decomposition, and event coverage

A central feature of Leaderboard v2.0 is its emphasis on scenario diversity. TaCarla reports that the dataset was collected based on the training and validation routes defined by the CARLA Leaderboard 2.0 challenge in CARLA 0.9.15, and describes the benchmark as comprising **36 distinct scenarios** simulating real-life traffic complexities such as lane changing, interacting with pedestrians, traffic lights, emergency vehicles, and unexpected stops [2602.23499]. Because the XML route files contain multiple scenarios, the TaCarla authors state that they “meticulously separated them into distinct routes based on the trigger points where each scenario is executed,” with separation performed between two consecutive scenarios on the original routes [2602.23499].

The scenarios highlighted in work built on the benchmark include Accident, Lane Changing, ConstructionObstacle, HazardAtSideLane, ParkedObstacle, ParkingCrossingPedestrian, ParkingExit, and YieldToEmergencyVehicle [2602.23499]. TaCarla specifically notes that these scenarios frequently require lane changing and monitoring adjacent vehicles, making them more complex than basic navigation [2602.23499]. In a different characterization, Think2Drive describes CARLA v2 as adding **39 common events** in the driving scene and providing a more quasi-realistic testbed compared to CARLA v1 [2402.16720]. Read together, these descriptions indicate that the benchmark is defined not merely by route following, but by dense insertion of safety-critical or long-tail events.

Scenario density is itself a methodological issue. Think2Drive notes that official CARLA v2 routes are long and scenario-rich, with **60+ scenarios per \(>7\) km route**, and argues that this makes training and fine-grained evaluation slow and ambiguous [2402.16720]. Its CornerCase-Repository therefore restructures evaluation around short routes with only one scenario each, enabling per-scenario attribution rather than route-level aggregation [2402.16720]. This suggests that route decomposition is not just a data-engineering convenience; it is a way of turning the leaderboard’s composite tasks into analyzable experimental units.

## 3. Metrics, ranking, and score variants

The benchmark’s standard reporting centers on multiplicative aggregation of route progress and infractions. One formulation used in evaluations of Leaderboard agents is
\[
Sc = R \cdot P,
\]
where \(R\) is route completion rate and
\[
P = \prod_{i \in I} p_i^{n_i},
\]
with \(p_i\) the penalty rate for infraction type \(i\) and \(n_i\) the number of infractions of type \(i\) [2501.12090]. Closely related formulations appear in other papers as
\[
\text{DS}_i = R_i \cdot P_i
\]
for Driving Score, Route Completion, and Infraction Penalty [2509.22754], or as **Driving Score**, **Route Score**, and **Penalty** in TaCarla’s description of official CARLA Leaderboard v2.0 closed-loop metrics [2602.23499]. Across papers, the nomenclature varies, but the underlying structure is a coupling of progress with penalties for collisions, traffic violations, route deviations, and related failures.

This multiplicative design has been criticized. Think2Drive argues that the CARLA Leaderboard’s original scoring system leads to counterintuitive results on long, scenario-rich routes because even a minor, rare infraction can drastically reduce the total score [2402.16720]. It proposes a **Weighted Driving Score (WDS)**,
\[
\text{WDS} = \text{RC} \times \prod_{i=1}^{m} (\text{penalty}_i)^{n_i},
\]
where \(n_i\) is normalized by the number of scenarios on the route, with the stated goal of fairer comparison across routes of different lengths and densities [2402.16720]. The proposal does not replace the leaderboard metric, but it makes explicit a recurring concern: route-level aggregation can confound safety, difficulty, and scenario count.

The leaderboard has also become the basis for broader task suites. TaCarla uses the Leaderboard 2.0 platform for both open-loop and closed-loop evaluation, with open-loop planning metrics such as ADE, FDE, AHE, and FHE at \(1\) s, \(2\) s, and \(4\) s horizons; object detection metrics following nuScenes, including mAP, ATE, ASE, AOE, and AVE; lane and centerline detection with \(\text{AP}_c\), \(\text{AP}_f\), and \(F1\); and traffic-light detection with COCO-style AP and \(AP_{50}\) [2602.23499]. In that sense, the leaderboard is no longer only a challenge interface; it is also a scaffold for multi-task evaluation protocols.

## 4. Representative methods and architectural responses

The diversity of methods evaluated on CARLA Leaderboard v2.0 is unusually broad. On the modular side, Kyber-E2E, the top submission to the 2023 map track, uses five main components—sensing, localization, perception, tracking/prediction, and planning/control—and attributes a significant part of its performance to language-assisted perception models and Inverse Reinforcement Learning for motion-planner cost tuning [2405.01394]. Its reported leaderboard results are a **Driving Score** of **3.109**, **Route Completion** of **5.285**, and **Infraction Penalty** of **0.669** [2405.01394]. PaaS represents a different modular lineage: a reactive planner in Frenet frame under complex urban street constraints, ranked **3rd out of 9 submissions** on the CADL map track with **Driving Score 48.24**, **Route Completion 60.68**, and **Infraction Penalty 0.84** [2304.08252].

On the end-to-end side, CarLLaVA shows that Leaderboard 2.0 can also be a proving ground for camera-only vision-language systems. It uses only camera images, ego-vehicle speed, and the next two target points; requires no LiDAR, no semantic maps, no BEV supervision, and no complex auxiliary labels; and reports **1st place in the sensor track** with **Driving Score 6.87**, **Route Completion 18.08**, and **Infraction Score 0.42** [2406.10165]. A key design element is its semi-disentangled output representation, in which time-conditioned waypoints are used for longitudinal control and space-conditioned path waypoints for lateral control [2406.10165].

Leaderboard 2.0 has also motivated object-centric and reinforcement-learning approaches. PlanT 2.0 introduces additional object classes—pedestrians, static objects, emergency vehicles, stop signs, and traffic lights—plus a \(64\) m \(\times 64\) m BEV SD-map, enlarged detection range, and disentangled lateral and longitudinal outputs in order to handle the scenarios newly introduced by the challenging CARLA Leaderboard 2.0 [2511.07292]. Raw2Drive, by contrast, frames the benchmark as an end-to-end model-based RL problem from raw sensor input, and is described as the only RL based end-to-end method on CARLA Leaderboard 2.0 and Bench2Drive; its reported validation/test results on CARLA v2 are **Driving Score 4.12/3.56**, **Route Completion 9.32/6.04**, and **Infraction Score 0.43/0.42** [2505.16394]. Collectively, these systems show that Leaderboard v2.0 is not aligned with a single architectural doctrine.

## 5. Datasets, infrastructure, and benchmark-centered extensions

A substantial ecosystem has formed around the leaderboard. TaCarla is explicitly built from Leaderboard 2.0 routes and scenarios, with **over 2,850,000 frames** recorded at **10 Hz** and sensor data captured with the **NuScenes sensor configuration**: **6 RGB cameras, 5 radars, and 1 LiDAR**, together with bird’s-eye view RGB, depth images, instance segmentation, and semantic segmentation [2602.23499]. The dataset supports planning, dynamic object detection, lane divider detection, centerline detection, traffic light recognition, prediction tasks, depth prediction, and visual-language action models, and introduces an IDF-based rarity score,
\[
\mathrm{Rarity}(W_t) = \frac{1}{|W_t|} \sum_{w \in W_t} \log\!\left( \frac{1+l_N}{1 + \sum_{n \in N} \mathbf{1}_{\{\, w \in n \,\}} } \right),
\]
followed by min-max normalization to \([0,1]\), to quantify how rare or long-tail a scenario is in the corpus [2602.23499].

Tooling for reusing leaderboard agents has also expanded. PCLA is an open-source Python testing framework that includes **nine high-performing pre-trained autonomous agents** from the Leaderboard challenges and is designed to deploy Leaderboard agents onto a vehicle without relying on the Leaderboard codebase [2503.09385]. It exposes a uniform interface, allows researchers to switch between agents without requiring modifications to CARLA versions or programming environments, and is compatible with the latest CARLA version, tested up to **0.9.15**, despite the original Leaderboard codebase being written for **CARLA 0.9.10** [2503.09385]. This decoupling is important because it turns leaderboard submissions into reusable experimental objects rather than one-off competition artifacts.

Extensions have also targeted realism and security. HABIT integrates with CARLA’s Leaderboard and replaces scripted pedestrian behavior with curated SMPL-format human motions, yielding **4,730 traffic-compatible pedestrian motions** and extending evaluation with metrics such as **Abbreviated Injury Scale (AIS 3+)** and **False Positive Braking Rate (FPBR)** [2511.19109]. A separate adversarial-machine-learning study uses Leaderboard agents as black-box systems and streams adversarial patches into CARLA at runtime against **TCP**, **NEAT**, and **Rails**, showing that some attacks can mislead model predictions while agent-specific modules such as PID control or GPS-based rules can overrule attacker-manipulated outputs [2511.14876]. These developments move the benchmark from pure performance ranking toward richer stress testing.

## 6. Limitations, critiques, and research implications

A recurring criticism is that leaderboard scores can hide the causes of failure. The CCTest comparison argues that the Leaderboard provides broad but shallow coverage: it can miss accidents in critical but potentially safe situations because scenario generation does not guarantee that key parameters such as positions and speeds create the most challenging safe cases [2501.12090]. The same study states that the Leaderboard does **not distinguish between avoidable and unavoidable accidents**, and reports that **about 24% of accidents in “normal” scenarios** were caused by background vehicles but still counted against the ego vehicle’s score [2501.12090]. This directly challenges the interpretation of aggregate leaderboard metrics as direct indicators of agent responsibility.

A second critique concerns behavioral realism. HABIT states that the default CARLA Leaderboard relies on deterministic, scripted pedestrian and vehicle behaviors, and shows that agents with close or equal to zero collisions per kilometer on the standard leaderboard perform notably worse when evaluated with richer pedestrian motions, reaching up to **7.43 collisions/km**, **12.94% AIS 3+ injury risk**, and unnecessary braking in up to **33%** of cases [2511.19109]. This result does not invalidate the leaderboard, but it does show that script-based evaluation can miss planner weaknesses that remain hidden in simpler simulations.

A third critique is data-centric. PlanT 2.0 argues that lack of obstacle diversity, rigid expert behaviors, and overfitting to a fixed set of expert trajectories produce shortcut learning and structural flaws in closed-loop driving [2511.07292]. Its controlled perturbation analysis exposes failures such as position-based shortcuts, premature scenario anticipation, proximity-margin failures, and undefined crash handling [2511.07292]. A related concern appears in cross-benchmark comparison work, which finds that models optimized for one environment may not generalize without considerable adaptation of map format, traffic scenario diversity, vehicle dynamics, and planning horizons [2509.22754].

Taken together, these critiques indicate that CARLA Leaderboard v2.0 is best understood as a strong but incomplete benchmark. It is highly valuable for reproducible, route-based, closed-loop comparison, and it has become a substrate for datasets, modular and end-to-end methods, world-model RL, human-behavior simulation, and adversarial evaluation. A plausible implication is that leaderboard performance alone is insufficient for safety validation, and that scenario-level analysis, richer human behavior, calibrated safety metrics, and more diverse training data are necessary complements to aggregate Driving Score.

Source: https://www.emergentmind.com/topics/carla-leaderboard-v2-0