---
title: 'CARLA Benchmarks: Autonomous Driving Evaluation'
url: https://www.emergentmind.com/topics/carla-benchmarks
type: topic
---

# CARLA Benchmarks: Autonomous Driving Evaluation

CARLA benchmarks are benchmarking protocols, datasets, and evaluation suites built on the open-source CARLA simulator for autonomous driving research, where development, training, and validation are conducted under flexible sensor suites and environmental conditions, and performance is examined through quantitative metrics such as success rate, infraction analysis, route completion, and driving score [1711.03938]. The benchmark ecosystem has expanded from urban navigation in two towns to real-traffic scenario replay, digital twins, collaborative occupancy prediction, 3D mapping, SLAM, adversarial robustness, autonomous racing, and air-ground embodied intelligence. In a separate literature, the acronym CARLA also denotes a Python library for benchmarking counterfactual explanation and algorithmic recourse methods, and a recursive location-assignment algorithm in activity-based transport models [2108.00783].

## 1. Foundational benchmark structure in the CARLA simulator

The original CARLA benchmark formalized autonomous urban driving evaluation around two distinctive urban maps, “Town 1” and “Town 2,” with Town 1 used for training and Town 2 withheld for testing, 18 combinations of weather and illumination, and four navigation tasks of increasing difficulty: Straight, One Turn, Navigation, and Navigation with Dynamic Obstacles [1711.03938]. Each $(\text{task}, \text{town}, \text{weather})$ combination is run for 25 episodes, the agent must reach the goal within a time budget computed as the optimal time at 10 km/h, and episodes do not terminate on infractions, although opposite-lane driving, sidewalk incursions, and collisions with static objects, cars, and pedestrians are logged. The central success metric is

$$
\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,
$$

and infractions are reported as average distance between infractions of a given type, in kilometers. The same benchmark also defined a reinforcement-learning reward,
$$
r_t = 1000 (d_{t-1} - d_t) + 0.05 (v_t - v_{t-1}) - 0.00002 (c_t - c_{t-1}) - 2 (s_t - s_{t-1}) - 2 (o_t - o_{t-1}),
$$
linking route progress, speed, collision damage, sidewalk overlap, and opposite-lane overlap [1711.03938].

The benchmark’s initial comparative role was to contrast a modular pipeline, an end-to-end model trained via imitation learning, and an end-to-end model trained via reinforcement learning. The reported results established two recurring themes in later CARLA work: difficulty grows sharply when dynamic actors are introduced, and generalization to a new town is substantially harder than generalization to new weather [1711.03938].

A later survey of around 100 peer-reviewed papers systematized how this template evolved in reinforcement-learning work. It reported that more than 80% of existing studies still rely on model-free methods such as DQN, PPO and SAC, and consolidated the most common metrics as success rate, collision rate, lane deviation, and driving score [2509.08221]. The same review identified persistent limitations: sparse rewards, sim-to-real transfer, safety guarantees, limited behaviour diversity, non-standardized metrics and scenarios, and over-reliance on Town01 and Town02. This suggests that CARLA benchmarking matured early as a controlled evaluation protocol, but standardization across papers remained incomplete [2509.08221].

## 2. Real-traffic, digital-twin, and closed-loop CARLA benchmarks

Several benchmark families were introduced specifically to reduce the gap between handcrafted simulation scenarios and real-world traffic. CARLA Real Traffic Scenarios (CRTS) constructed over 60,000 interactive tactical scenarios from real-world trajectory datasets: over 2,000 lane change scenarios from NGSIM and 64,000 roundabout scenarios from openDD, together with 9 new high-fidelity CARLA maps reflecting the source layouts [2012.11329]. In CRTS, the ego vehicle is controlled in closed loop while all other vehicles follow ground-truth previously-recorded trajectories. The benchmark supports bird’s-eye view, four RGB cameras or a front-facing camera, and LiDAR plus camera observations, and defines sparse, dense, and no-failure reward schemes. Validation success rates showed that bird’s-eye view reached \(0.787 \pm 0.029\) on lane changes and \(0.862 \pm 0.037\) on roundabouts, while dense reward improved NGSIM lane-change performance to 0.828 [2012.11329].

DriveE2E extended the real-to-simulation idea to closed-loop end-to-end driving at urban intersections. It extracted 800 dynamic traffic scenarios from a 100-hour video dataset captured by high-mounted infrastructure sensors and created static digital twin assets for 15 real-world intersections with consistent visual appearance [2509.23922]. In its log-replay mode, non-ego traffic agents do not react to the ego vehicle and follow pre-recorded trajectories, while the ego vehicle receives simulated sensor data and must complete a source-destination route. The benchmark adopts success rate,
$$
\text{SR}=\frac{\#~\text{successfully completed routes (no collisions/violations)}}{\#~\text{total routes}},
$$
and a driving score defined from route completion and infraction penalties. Reported baselines showed that UniAD achieved 47.00% SR and 77.62% DS, VAD achieved 35.00% SR and 62.29% DS, and the ranking under closed-loop evaluation diverged from open-loop L2 error ranking [2509.23922].

R-CARLA targeted autonomous racing, where standard CARLA’s built-in physics are inadequate for scaled vehicles and high-slip dynamics. It disables CARLA’s physics engine and exposes a ROS interface for any user-supplied dynamics model, while retaining CARLA’s sensor suite and adding a digital-twin pipeline that converts real LiDAR/SLAM/IMU data into a CARLA map through point-cloud filtering, Poisson-disk sparsification, and Ball-Pivot meshing [2506.09629]. The benchmark is explicitly holistic and closed loop, spanning perception, estimation, planning, control, dynamics, and sensor simulation. Its reported results showed a 42% reduction in the vehicle-dynamics Sim-to-Real gap on average, an 82% reduction for sensor simulation across testing scenarios, and for the Gokart case a lap-time deviation decrease from 5.48 s in CARLA to 1.03 s in R-CARLA [2506.09629].

HABIT introduced a different realism axis: realistic and diverse human behavior. It integrates real-world human motion into CARLA through a modular, extensible, and physically consistent motion retargeting pipeline, reducing an initial pool of approximately 30,000 retargeted motions to 4,730 traffic-compatible pedestrian motions in SMPL format, and generates scenarios with 30 vehicles, 20 diverse action pedestrians, and 10 ambient pedestrians on 110 routes under 12 weathers [2511.19109]. HABIT augments standard CARLA metrics with Abbreviated Injury Scale and False Positive Braking Rate. The serious-injury risk is modeled as
$$
P_{\text{MAIS 3+}}(v) = \frac{1}{1 + \exp(3.164 - 0.288 v)}.
$$
Although InterFuser, TransFuser, and BEVDriver achieve close or equal to zero collisions per kilometer on the CARLA Leaderboard, HABIT reports up to 7.43 collisions/km, a 12.94% AIS 3+ injury risk, and unnecessary braking in up to 33% of cases [2511.19109]. The benchmark therefore reframes pedestrian interaction as a planner-diagnostic problem rather than a purely scripted obstacle-avoidance problem.

## 3. Perception, occupancy, and 3D mapping benchmarks

CARLA has also become a substrate for dense perception benchmarks that require annotations difficult to obtain in real traffic. A synthetic benchmark for collaborative 3D semantic occupancy prediction replayed OPV2V scenarios in CARLA and added a custom high-resolution semantic voxel sensor based on Unreal Engine collision and overlap functions [2506.17004]. The per-agent voxel volume is \(100 \times 100 \times 7\text{m}^3\) with \(0.1\text{m}\) voxel size, yielding a resolution of \(1000\times1000\times70\), and the annotations contain 24 semantic categories including “empty.” Three benchmark settings—Small, Medium, and Large—were defined with increasing prediction range. Evaluation uses class IoU and mean IoU,
$$
\text{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c}, \qquad
\text{mIoU}=\frac{1}{C}\sum_{c=1}^{C}\text{IoU}_c.
$$
The collaborative baseline Co3SOP-Base achieved 30.04, 27.50, and 27.00 mIoU at 25.6 m, 51.2 m, and 76.8 m respectively, consistently outperforming its single-agent counterpart, with gains increasing from +0.68 to +2.19 as the range expands [2506.17004].

Paris-CARLA-3D pursued matched real-synthetic 3D mapping. It combines synthetic CARLA point clouds totaling 700 million points with 60 million real points from Paris, both acquired with matched LiDAR and camera platforms, and uses the CARLA semantic tag set with 23 classes for label alignment across domains [2111.11348]. The benchmark supports semantic segmentation, instance segmentation, and scene completion. For semantic segmentation, the reported overall mIoU was 13.9% for PointNet++ and 37.5% for KPConv; for scene completion, CARLA pre-training followed by fine-tuning on Paris improved \(CD_{pred \leftrightarrow ori}\) from 10.7 cm to 7.5 cm [2111.11348]. The matched sensor configuration was introduced specifically to facilitate transfer learning and domain adaptation.

CDrone added viewpoint diversity absent from most CARLA benchmarks. Generated in epic rendering mode across 42 unique locations in 7 simulated worlds, it provides about 22,000 images each for train, validation, and test, with drone altitudes from 6.9 m to 60.6 m, camera viewing angles from \(7.57^\circ\) to \(88.89^\circ\), and rich annotations including full 3D bounding boxes, 2D projections, instance IDs, depth maps, and segmentation masks [2408.11958]. Its primary metric is \(AP_{3D}^{0.5}\), and unlike car-view datasets it evaluates full \(SO(3)\) rotation. On CDrone, GroundMix increased MonoCon-based performance to 10.70 AP for cars and 17.28 AP for trucks, while on Rope3D it improved AP from 38.07 to 47.72 [2408.11958].

TaCarla broadened the perception-and-planning regime to a large-scale Leaderboard 2.0-derived dataset with over 2.85 million frames at 10 Hz, a nuScenes-equivalent sensor suite of 6 RGB cameras, 5 radars, and 1 LiDAR, and labels for planning, dynamic object detection, lane divider detection, centerline detection, traffic light recognition, prediction tasks, and visual-language action models [2602.23499]. It also introduced a normalized rarity score inspired by inverse document frequency to measure how rarely the current state occurs in the dataset, allowing explicit analysis of long-tail events [2602.23499].

At the annotation level, work on improving bounding boxes in CARLA targeted a specific failure mode in synthetic data generation: ghost boxes on occluded objects. The proposed method synchronizes a semantic-segmentation camera with the RGB camera, checks class-consistent pixels inside each candidate box, retains a box if at least 10% of its area matches the object class, and raises the threshold to 50% when a box covers more than 70% of the image area [2509.16773]. With image size \(1280\times1280\), the reported mAP50 reached 0.966 for all objects up to 50 m, with precision 0.955 and recall 0.908 [2509.16773]. This benchmark contribution is narrower than full-scene suites, but it addresses label fidelity directly.

## 4. SLAM, calibration, adversarial robustness, and failure-oriented evaluation

IBISCape extended CARLA into a multi-modal SLAM benchmark by providing open-source APIs for synchronized telemetry from stereo RGB, stereo DVS, depth, IMU, GPS, ground-truth segmentation, ego-motion, and vehicle control signals [2206.13455]. It includes 34 datasets for calibration and SLAM under clear, moderate, and dynamic weather, introduces Checkerboard (7×7) and AprilGrid (6×6) targets into CARLA maps, and validates simulated IMU noise through Allan variance analysis over 168 hours of data. ORB-SLAM3 systems and BASALT VIO were evaluated using Absolute Trajectory Error and Relative Pose Error, with ORB-SLAM3 generally outperforming BASALT on ATE and RPE in a variety of settings, while both suffered degradation under highly dynamic weather [2206.13455].

CARLA-GeAR addressed adversarial robustness rather than nominal accuracy. Built on the CARLA Python API, it automatically generates photo-realistic synthetic datasets for semantic segmentation, 2D object detection, 3D stereo object detection, and monocular depth estimation, with adversarial patches attached to billboards or the back of a truck [2206.04365]. The generator supports clean and adversarial twin datasets, standard formats such as Cityscapes, COCO, and KITTI, and evaluation through task metrics including mIoU, COCO mAP, RMSE, and KITTI AP, as well as AUROC for attack detection. Its emphasis is systematic, repeatable comparison of defense and detection methods under physical patch attacks rather than scenario diversity alone [2206.04365].

ANTI-CARLA moved from robustness measurement to adversarial scenario search. It extends CARLA with a scenario description language for challenging weather, sensor faults, traffic, infrastructure, and route geometry, and uses passive samplers such as random, grid, and Halton sequences together with active samplers including Random Neighborhood Search and Guided Bayesian Optimization [2208.06309]. Test severity is aggregated through a weighted infraction score,
$$
TS = \sum_{k=1}^{n} w_k \cdot I_k.
$$
The central empirical result is diagnostic: although Learning By Cheating reached 100% in the CARLA benchmark, ANTI-CARLA found failures in up to 27% of adversarially generated test cases in Town3 when using Random Neighborhood Search [2208.06309]. This directly challenges the sufficiency of static benchmark suites for safety validation.

## 5. Multi-agent and embodied-intelligence extensions

CARLA-Air generalizes the benchmark space from ground vehicles to a unified air-ground environment. It integrates CARLA and AirSim inside a single Unreal Engine process, preserves both native Python APIs and ROS 2 interfaces, and synchronously captures up to 18 sensor modalities across aerial and ground platforms at each tick [2603.28032]. The platform supports air-ground cooperation, embodied navigation and vision-language action, multi-modal perception and dataset construction, and reinforcement-learning-based policy training. Its benchmark workflows include precision landing on a moving car, dual-view VLN/VLA data generation, synchronized multi-modal dataset collection, cross-view perception, and RL environments tested through 357 reset cycles with zero crashes or memory leaks during 3-hour continuous runs [2603.28032]. The emphasis on single-process execution and strict sensor synchronization reframes CARLA benchmarking as a spatial-temporal consistency problem, not only a route-completion problem.

A deployment-oriented variant of CARLA benchmarking appears in autonomous racing stacks implemented through ROS 2. One such system uses a 360° LiDAR, stereo camera, GNSS, and IMU in CARLA, reports reliable cone detection up to 35 m with LiDAR and depth estimation up to 20 m with the stereo camera, and quantifies system timing through 12–14 ms camera inference latency, 37.3 ms LiDAR processing latency, pose estimation at up to 100 Hz from an EKF, and mapping updates at 9.5 Hz [2511.11310]. Planning is evaluated with midpoint generation and Pure Pursuit,
$$
L_d = K_{ld} \times V + L_{fc}, \qquad
\delta = \arctan\!\left(\frac{2L\sin(\alpha)}{L_d}\right),
$$
while validation is carried out in CARLA before transfer to hardware [2511.11310]. This line of work is less about public benchmark suites than about cycle-for-cycle full-stack validation, but it uses the same CARLA benchmarking logic: realistic sensing, timing, and closed-loop control under controlled conditions.

## 6. Distinct benchmark uses of the CARLA acronym outside the simulator literature

Outside autonomous driving simulation, CARLA is also the name of a Python library for benchmarking counterfactual explanation and algorithmic recourse methods [2108.00783]. That CARLA supports 11 popular methods—AR, AR-LIME, CEM, DICE, GS, Wachter et al., CEM-VAE, CLUE, FACE-KNN, FACE-EPS, and REVISE—across Adult, Give Me Some Credit, and COMPAS, with logistic regression and multi-layer perceptron backends and wrappers for PyTorch and TensorFlow. Its six integrated evaluation measures are costs \((\ell_0,\ell_1)\), constraint violation, yNN, redundancy, success rate, and computation time. The benchmark’s main finding is that there is no universal winner: independence-based methods often produce sparser counterfactuals, while dependence-based methods often achieve higher neighborhood support [2108.00783].

A further, unrelated use appears in activity-based transport modeling, where CARLA denotes spatially Constrained Anchor-based Recursive Location Assignment [2509.18191]. Its empirical benchmarks use a sample of 1,000 individuals from a German Human Travel Survey and target locations in Hanover derived from OpenStreetMap. Compared with the Hörl relaxation-discretization approach, CARLA achieved lower mean total distance deviation at every runtime budget, and in one detailed comparison on 693 feasible individuals, runtime was 45 seconds for CARLA versus 89 seconds for Hörl, with CARLA winning in 630 cases, Hörl in 1 case, and 62 ties [2509.18191]. For the term “CARLA benchmarks,” this means acronym disambiguation is essential: in current arXiv usage, the phrase can refer either to benchmark suites built on the CARLA simulator or to benchmark frameworks in entirely different research areas.

Across these literatures, the unifying feature is not a single metric or task but a benchmarking philosophy: configurable environments, explicit protocols, and quantitative comparison under controlled variation. In the CARLA simulator literature specifically, the trajectory has moved from route-based urban navigation toward digital twins, dense scene annotation, adversarial search, human-interaction realism, and cross-domain embodied intelligence. This suggests that “CARLA benchmarks” now denotes an ecosystem of evaluation methodologies rather than a single benchmark suite.

Source: https://www.emergentmind.com/topics/carla-benchmarks