CARLA Benchmarks: Autonomous Driving Evaluation

• CARLA Benchmarks are a set of evaluation protocols, datasets, and simulation suites built on the CARLA simulator that standardize autonomous driving research.
• They employ quantitative metrics like success rate, collision analysis, and driving score while testing scenarios under varied weather, town layouts, and dynamic agents.
• Extensions of CARLA Benchmarks include digital twins, adversarial robustness tests, SLAM, and even evaluations in counterfactual explanations and activity-based transport models.

CARLA benchmarks are benchmarking protocols, datasets, and evaluation suites built on the open-source CARLA simulator for autonomous driving research, where development, training, and validation are conducted under flexible sensor suites and environmental conditions, and performance is examined through quantitative metrics such as success rate, infraction analysis, route completion, and driving score (Dosovitskiy et al., 2017). The benchmark ecosystem has expanded from urban navigation in two towns to real-traffic scenario replay, digital twins, collaborative occupancy prediction, 3D mapping, SLAM, adversarial robustness, autonomous racing, and air-ground embodied intelligence. In a separate literature, the acronym CARLA also denotes a Python library for benchmarking counterfactual explanation and algorithmic recourse methods, and a recursive location-assignment algorithm in activity-based transport models (Pawelczyk et al., 2021).

1. Foundational benchmark structure in the CARLA simulator

The original CARLA benchmark formalized autonomous urban driving evaluation around two distinctive urban maps, “Town 1” and “Town 2,” with Town 1 used for training and Town 2 withheld for testing, 18 combinations of weather and illumination, and four navigation tasks of increasing difficulty: Straight, One Turn, Navigation, and Navigation with Dynamic Obstacles (Dosovitskiy et al., 2017). Each $(\text{task}, \text{town}, \text{weather})$ combination is run for 25 episodes, the agent must reach the goal within a time budget computed as the optimal time at 10 km/h, and episodes do not terminate on infractions, although opposite-lane driving, sidewalk incursions, and collisions with static objects, cars, and pedestrians are logged. The central success metric is

$$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$

and infractions are reported as average distance between infractions of a given type, in kilometers. The same benchmark also defined a reinforcement-learning reward,

$$r_t = 1000 (d_{t-1} - d_t) + 0.05 (v_t - v_{t-1}) - 0.00002 (c_t - c_{t-1}) - 2 (s_t - s_{t-1}) - 2 (o_t - o_{t-1}),$$

linking route progress, speed, collision damage, sidewalk overlap, and opposite-lane overlap (Dosovitskiy et al., 2017).

The benchmark’s initial comparative role was to contrast a modular pipeline, an end-to-end model trained via imitation learning, and an end-to-end model trained via reinforcement learning. The reported results established two recurring themes in later CARLA work: difficulty grows sharply when dynamic actors are introduced, and generalization to a new town is substantially harder than generalization to new weather (Dosovitskiy et al., 2017).

A later survey of around 100 peer-reviewed papers systematized how this template evolved in reinforcement-learning work. It reported that more than 80% of existing studies still rely on model-free methods such as DQN, PPO and SAC, and consolidated the most common metrics as success rate, collision rate, lane deviation, and driving score (Delavari et al., 10 Sep 2025). The same review identified persistent limitations: sparse rewards, sim-to-real transfer, safety guarantees, limited behaviour diversity, non-standardized metrics and scenarios, and over-reliance on Town01 and Town02. This suggests that CARLA benchmarking matured early as a controlled evaluation protocol, but standardization across papers remained incomplete (Delavari et al., 10 Sep 2025).

2. Real-traffic, digital-twin, and closed-loop CARLA benchmarks

Several benchmark families were introduced specifically to reduce the gap between handcrafted simulation scenarios and real-world traffic. CARLA Real Traffic Scenarios (CRTS) constructed over 60,000 interactive tactical scenarios from real-world trajectory datasets: over 2,000 lane change scenarios from NGSIM and 64,000 roundabout scenarios from openDD, together with 9 new high-fidelity CARLA maps reflecting the source layouts (Osiński et al., 2020). In CRTS, the ego vehicle is controlled in closed loop while all other vehicles follow ground-truth previously-recorded trajectories. The benchmark supports bird’s-eye view, four RGB cameras or a front-facing camera, and LiDAR plus camera observations, and defines sparse, dense, and no-failure reward schemes. Validation success rates showed that bird’s-eye view reached $0.787 \pm 0.029$ on lane changes and $0.862 \pm 0.037$ on roundabouts, while dense reward improved NGSIM lane-change performance to 0.828 (Osiński et al., 2020).

DriveE2E extended the real-to-simulation idea to closed-loop end-to-end driving at urban intersections. It extracted 800 dynamic traffic scenarios from a 100-hour video dataset captured by high-mounted infrastructure sensors and created static digital twin assets for 15 real-world intersections with consistent visual appearance (Yu et al., 28 Sep 2025). In its log-replay mode, non-ego traffic agents do not react to the ego vehicle and follow pre-recorded trajectories, while the ego vehicle receives simulated sensor data and must complete a source-destination route. The benchmark adopts success rate,

$$\text{SR}=\frac{\#~\text{successfully completed routes (no collisions/violations)}}{\#~\text{total routes}},$$

and a driving score defined from route completion and infraction penalties. Reported baselines showed that UniAD achieved 47.00% SR and 77.62% DS, VAD achieved 35.00% SR and 62.29% DS, and the ranking under closed-loop evaluation diverged from open-loop L2 error ranking (Yu et al., 28 Sep 2025).

R-CARLA targeted autonomous racing, where standard CARLA’s built-in physics are inadequate for scaled vehicles and high-slip dynamics. It disables CARLA’s physics engine and exposes a ROS interface for any user-supplied dynamics model, while retaining CARLA’s sensor suite and adding a digital-twin pipeline that converts real LiDAR/SLAM/IMU data into a CARLA map through point-cloud filtering, Poisson-disk sparsification, and Ball-Pivot meshing (Brunner et al., 11 Jun 2025). The benchmark is explicitly holistic and closed loop, spanning perception, estimation, planning, control, dynamics, and sensor simulation. Its reported results showed a 42% reduction in the vehicle-dynamics Sim-to-Real gap on average, an 82% reduction for sensor simulation across testing scenarios, and for the Gokart case a lap-time deviation decrease from 5.48 s in CARLA to 1.03 s in R-CARLA (Brunner et al., 11 Jun 2025).

HABIT introduced a different realism axis: realistic and diverse human behavior. It integrates real-world human motion into CARLA through a modular, extensible, and physically consistent motion retargeting pipeline, reducing an initial pool of approximately 30,000 retargeted motions to 4,730 traffic-compatible pedestrian motions in SMPL format, and generates scenarios with 30 vehicles, 20 diverse action pedestrians, and 10 ambient pedestrians on 110 routes under 12 weathers (Ramesh et al., 24 Nov 2025). HABIT augments standard CARLA metrics with Abbreviated Injury Scale and False Positive Braking Rate. The serious-injury risk is modeled as

$$P_{\text{MAIS 3+}}(v) = \frac{1}{1 + \exp(3.164 - 0.288 v)}.$$

Although InterFuser, TransFuser, and BEVDriver achieve close or equal to zero collisions per kilometer on the CARLA Leaderboard, HABIT reports up to 7.43 collisions/km, a 12.94% AIS 3+ injury risk, and unnecessary braking in up to 33% of cases (Ramesh et al., 24 Nov 2025). The benchmark therefore reframes pedestrian interaction as a planner-diagnostic problem rather than a purely scripted obstacle-avoidance problem.

3. Perception, occupancy, and 3D mapping benchmarks

CARLA has also become a substrate for dense perception benchmarks that require annotations difficult to obtain in real traffic. A synthetic benchmark for collaborative 3D semantic occupancy prediction replayed OPV2V scenarios in CARLA and added a custom high-resolution semantic voxel sensor based on Unreal Engine collision and overlap functions (Wu et al., 20 Jun 2025). The per-agent voxel volume is $100 \times 100 \times 7\text{m}^3$ with $0.1\text{m}$ voxel size, yielding a resolution of $1000\times1000\times70$, and the annotations contain 24 semantic categories including “empty.” Three benchmark settings—Small, Medium, and Large—were defined with increasing prediction range. Evaluation uses class IoU and mean IoU,

$$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$0

The collaborative baseline Co3SOP-Base achieved 30.04, 27.50, and 27.00 mIoU at 25.6 m, 51.2 m, and 76.8 m respectively, consistently outperforming its single-agent counterpart, with gains increasing from +0.68 to +2.19 as the range expands (Wu et al., 20 Jun 2025).

Paris-CARLA-3D pursued matched real-synthetic 3D mapping. It combines synthetic CARLA point clouds totaling 700 million points with 60 million real points from Paris, both acquired with matched LiDAR and camera platforms, and uses the CARLA semantic tag set with 23 classes for label alignment across domains (Deschaud et al., 2021). The benchmark supports semantic segmentation, instance segmentation, and scene completion. For semantic segmentation, the reported overall mIoU was 13.9% for PointNet++ and 37.5% for KPConv; for scene completion, CARLA pre-training followed by fine-tuning on Paris improved $$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$1 from 10.7 cm to 7.5 cm (Deschaud et al., 2021). The matched sensor configuration was introduced specifically to facilitate transfer learning and domain adaptation.

CDrone added viewpoint diversity absent from most CARLA benchmarks. Generated in epic rendering mode across 42 unique locations in 7 simulated worlds, it provides about 22,000 images each for train, validation, and test, with drone altitudes from 6.9 m to 60.6 m, camera viewing angles from $$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$2 to $$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$3, and rich annotations including full 3D bounding boxes, 2D projections, instance IDs, depth maps, and segmentation masks (Meier et al., 2024). Its primary metric is $$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$4, and unlike car-view datasets it evaluates full $$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$5 rotation. On CDrone, GroundMix increased MonoCon-based performance to 10.70 AP for cars and 17.28 AP for trucks, while on Rope3D it improved AP from 38.07 to 47.72 (Meier et al., 2024).

TaCarla broadened the perception-and-planning regime to a large-scale Leaderboard 2.0-derived dataset with over 2.85 million frames at 10 Hz, a nuScenes-equivalent sensor suite of 6 RGB cameras, 5 radars, and 1 LiDAR, and labels for planning, dynamic object detection, lane divider detection, centerline detection, traffic light recognition, prediction tasks, and visual-language action models (Gorgulu et al., 26 Feb 2026). It also introduced a normalized rarity score inspired by inverse document frequency to measure how rarely the current state occurs in the dataset, allowing explicit analysis of long-tail events (Gorgulu et al., 26 Feb 2026).

At the annotation level, work on improving bounding boxes in CARLA targeted a specific failure mode in synthetic data generation: ghost boxes on occluded objects. The proposed method synchronizes a semantic-segmentation camera with the RGB camera, checks class-consistent pixels inside each candidate box, retains a box if at least 10% of its area matches the object class, and raises the threshold to 50% when a box covers more than 70% of the image area (Chaar et al., 20 Sep 2025). With image size $$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$6, the reported mAP50 reached 0.966 for all objects up to 50 m, with precision 0.955 and recall 0.908 (Chaar et al., 20 Sep 2025). This benchmark contribution is narrower than full-scene suites, but it addresses label fidelity directly.

4. SLAM, calibration, adversarial robustness, and failure-oriented evaluation

IBISCape extended CARLA into a multi-modal SLAM benchmark by providing open-source APIs for synchronized telemetry from stereo RGB, stereo DVS, depth, IMU, GPS, ground-truth segmentation, ego-motion, and vehicle control signals (Soliman et al., 2022). It includes 34 datasets for calibration and SLAM under clear, moderate, and dynamic weather, introduces Checkerboard (7×7) and AprilGrid (6×6) targets into CARLA maps, and validates simulated IMU noise through Allan variance analysis over 168 hours of data. ORB-SLAM3 systems and BASALT VIO were evaluated using Absolute Trajectory Error and Relative Pose Error, with ORB-SLAM3 generally outperforming BASALT on ATE and RPE in a variety of settings, while both suffered degradation under highly dynamic weather (Soliman et al., 2022).

CARLA-GeAR addressed adversarial robustness rather than nominal accuracy. Built on the CARLA Python API, it automatically generates photo-realistic synthetic datasets for semantic segmentation, 2D object detection, 3D stereo object detection, and monocular depth estimation, with adversarial patches attached to billboards or the back of a truck (Nesti et al., 2022). The generator supports clean and adversarial twin datasets, standard formats such as Cityscapes, COCO, and KITTI, and evaluation through task metrics including mIoU, COCO mAP, RMSE, and KITTI AP, as well as AUROC for attack detection. Its emphasis is systematic, repeatable comparison of defense and detection methods under physical patch attacks rather than scenario diversity alone (Nesti et al., 2022).

ANTI-CARLA moved from robustness measurement to adversarial scenario search. It extends CARLA with a scenario description language for challenging weather, sensor faults, traffic, infrastructure, and route geometry, and uses passive samplers such as random, grid, and Halton sequences together with active samplers including Random Neighborhood Search and Guided Bayesian Optimization (Ramakrishna et al., 2022). Test severity is aggregated through a weighted infraction score,

$$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$7

The central empirical result is diagnostic: although Learning By Cheating reached 100% in the CARLA benchmark, ANTI-CARLA found failures in up to 27% of adversarially generated test cases in Town3 when using Random Neighborhood Search (Ramakrishna et al., 2022). This directly challenges the sufficiency of static benchmark suites for safety validation.

5. Multi-agent and embodied-intelligence extensions

CARLA-Air generalizes the benchmark space from ground vehicles to a unified air-ground environment. It integrates CARLA and AirSim inside a single Unreal Engine process, preserves both native Python APIs and ROS 2 interfaces, and synchronously captures up to 18 sensor modalities across aerial and ground platforms at each tick (Zeng et al., 30 Mar 2026). The platform supports air-ground cooperation, embodied navigation and vision-language action, multi-modal perception and dataset construction, and reinforcement-learning-based policy training. Its benchmark workflows include precision landing on a moving car, dual-view VLN/VLA data generation, synchronized multi-modal dataset collection, cross-view perception, and RL environments tested through 357 reset cycles with zero crashes or memory leaks during 3-hour continuous runs (Zeng et al., 30 Mar 2026). The emphasis on single-process execution and strict sensor synchronization reframes CARLA benchmarking as a spatial-temporal consistency problem, not only a route-completion problem.

A deployment-oriented variant of CARLA benchmarking appears in autonomous racing stacks implemented through ROS 2. One such system uses a 360° LiDAR, stereo camera, GNSS, and IMU in CARLA, reports reliable cone detection up to 35 m with LiDAR and depth estimation up to 20 m with the stereo camera, and quantifies system timing through 12–14 ms camera inference latency, 37.3 ms LiDAR processing latency, pose estimation at up to 100 Hz from an EKF, and mapping updates at 9.5 Hz (Abdo et al., 14 Nov 2025). Planning is evaluated with midpoint generation and Pure Pursuit,

$$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$8

while validation is carried out in CARLA before transfer to hardware (Abdo et al., 14 Nov 2025). This line of work is less about public benchmark suites than about cycle-for-cycle full-stack validation, but it uses the same CARLA benchmarking logic: realistic sensing, timing, and closed-loop control under controlled conditions.

6. Distinct benchmark uses of the CARLA acronym outside the simulator literature

Outside autonomous driving simulation, CARLA is also the name of a Python library for benchmarking counterfactual explanation and algorithmic recourse methods (Pawelczyk et al., 2021). That CARLA supports 11 popular methods—AR, AR-LIME, CEM, DICE, GS, Wachter et al., CEM-VAE, CLUE, FACE-KNN, FACE-EPS, and REVISE—across Adult, Give Me Some Credit, and COMPAS, with logistic regression and multi-layer perceptron backends and wrappers for PyTorch and TensorFlow. Its six integrated evaluation measures are costs $$\text{Success Rate} = \frac{\sum_{i=1}^N \mathbb{I}[\text{goal reached in episode } i]}{N} \times 100\%,$$9, constraint violation, yNN, redundancy, success rate, and computation time. The benchmark’s main finding is that there is no universal winner: independence-based methods often produce sparser counterfactuals, while dependence-based methods often achieve higher neighborhood support (Pawelczyk et al., 2021).

A further, unrelated use appears in activity-based transport modeling, where CARLA denotes spatially Constrained Anchor-based Recursive Location Assignment (Petre et al., 19 Sep 2025). Its empirical benchmarks use a sample of 1,000 individuals from a German Human Travel Survey and target locations in Hanover derived from OpenStreetMap. Compared with the Hörl relaxation-discretization approach, CARLA achieved lower mean total distance deviation at every runtime budget, and in one detailed comparison on 693 feasible individuals, runtime was 45 seconds for CARLA versus 89 seconds for Hörl, with CARLA winning in 630 cases, Hörl in 1 case, and 62 ties (Petre et al., 19 Sep 2025). For the term “CARLA benchmarks,” this means acronym disambiguation is essential: in current arXiv usage, the phrase can refer either to benchmark suites built on the CARLA simulator or to benchmark frameworks in entirely different research areas.

Across these literatures, the unifying feature is not a single metric or task but a benchmarking philosophy: configurable environments, explicit protocols, and quantitative comparison under controlled variation. In the CARLA simulator literature specifically, the trajectory has moved from route-based urban navigation toward digital twins, dense scene annotation, adversarial search, human-interaction realism, and cross-domain embodied intelligence. This suggests that “CARLA benchmarks” now denotes an ecosystem of evaluation methodologies rather than a single benchmark suite.