CARLA-Drive: Autonomous Driving Research Platform
- CARLA-Drive is an autonomous driving research ecosystem built on the CARLA simulator, offering a configurable urban testing ground with open-source tools.
- It integrates detailed sensor setups, synthetic datasets, and benchmark protocols to rigorously train and validate perception, planning, and control systems.
- The platform has evolved to support diverse methodologies including modular pipelines, end-to-end learning, and reinforcement learning with real-to-simulation digital twin evaluations.
CARLA-Drive denotes autonomous driving research conducted in the CARLA simulator: the use of CARLA as an open-source, urban, configurable, and instrumented environment for developing, training, and validating perception, planning, and control systems under controlled weather, lighting, traffic, and sensor configurations (Dosovitskiy et al., 2017). In practice, the term encompasses a broader ecosystem of benchmark protocols, modular and end-to-end driving stacks, reinforcement-learning methods, scenario-authoring tools, synthetic datasets, and digital-twin evaluation frameworks built around CARLA’s client–server simulation core (Neis et al., 29 Aug 2025).
1. Origins and research scope
CARLA was introduced as a purpose-built research simulator rather than a repurposed game engine. Its original design combined open-source simulator code, open digital assets, configurable sensors, rich ground truth, and explicit evaluation protocols intended to support training, prototyping, and validation of autonomous urban driving systems (Dosovitskiy et al., 2017). That design directly addressed limitations of earlier platforms. Racing simulators such as TORCS lacked intersections, pedestrians, cross traffic, and traffic rules, while commercial open-world games such as GTA V offered visual realism but not full state access, sensor configurability, or explicit benchmarking support (Dosovitskiy et al., 2017).
Within that framework, CARLA-Drive is not limited to a single methodological school. From the outset, CARLA supported both classic modular stacks and end-to-end learning. The canonical comparison in the original simulator paper evaluated three approaches on the same platform: a classic modular pipeline, conditional imitation learning, and reinforcement learning, all under shared tasks and metrics (Dosovitskiy et al., 2017). Later work expanded the meaning of CARLA-Drive beyond this initial triad. CARJAN added semantic multi-agent scenario generation and behavior-tree execution on top of CARLA (Neis et al., 29 Aug 2025), while other work used CARLA for conflict-situation research in shared autonomy (Mihaylova et al., 4 Mar 2025), synthetic multimodal data generation (Ramesh et al., 2024), large-scale Leaderboard 2.0 datasets (Gorgulu et al., 26 Feb 2026), nuScenes-style BEV perception benchmarks (Qiao et al., 12 Nov 2025), and real-to-simulation digital-twin evaluation at real intersections (Yu et al., 28 Sep 2025).
A persistent feature across these uses is the treatment of CARLA as a full experimental platform rather than merely a renderer. CARLA-Drive typically means controlled closed-loop experimentation in which the researcher can inspect not only images and controls, but also infractions, dynamic-agent states, traffic-light states, semantic labels, and scenario structure.
2. Simulator architecture and sensing
The foundational CARLA architecture is client–server. The server runs on Unreal Engine 4, simulates physics for vehicles and pedestrians, renders 3D scenes, and maintains the dynamic world state. The client uses a Python API over sockets to send vehicle-control commands and meta-commands and to receive sensor streams, measurements, and environment metadata (Dosovitskiy et al., 2017). This division is central to CARLA-Drive workflows because it decouples simulation from the learning framework and allows external integration through Python.
The original simulator shipped with two towns and a deliberately explicit train/test split. Town 1, used for training, contains about 2.9 km of drivable roads; Town 2, used for testing, contains about 1.4 km and different layout, buildings, and textures (Dosovitskiy et al., 2017). The environment includes static 3D models such as buildings, vegetation, sidewalks, fences, poles, walls, signs, and traffic lights, together with dynamic actors including 16 animated vehicle models and 50 animated pedestrian models with randomized appearance (Dosovitskiy et al., 2017). Non-player traffic follows basic behavior models: vehicles perform lane following, traffic-light handling, and intersection decisions, while pedestrians navigate via a cost-based map, prefer sidewalks and crossings, but can cross roads elsewhere (Dosovitskiy et al., 2017).
Sensor support in the original release focused on cameras, but with unusually strong configurability. CARLA supported RGB cameras plus pseudo-cameras for ground-truth depth and semantic segmentation; each camera could be configured by position, orientation, field of view, resolution, and depth of field (Dosovitskiy et al., 2017). The 12 semantic classes were road, lane-marking, traffic sign, sidewalk, fence, pole, wall, building, vegetation, vehicle, pedestrian, and other (Dosovitskiy et al., 2017). Beyond images, the simulator exposed ego pose, orientation, speed, 3D acceleration, cumulative collision damage by object type, sidewalk and wrong-lane overlap, traffic-light positions and states, speed-limit signs, and the positions, orientations, and bounding boxes of all vehicles and pedestrians (Dosovitskiy et al., 2017). This instrumentation is one of the main reasons CARLA-Drive became useful for both modular AV stacks and RL reward design.
Environmental variation was also explicit. CARLA offered two lighting regimes, midday and sunset, combined with nine weather presets varying cloudiness, rain intensity, puddles, fog, and ambient light, yielding 18 weather–lighting combinations (Dosovitskiy et al., 2017). The agent interface exposed steering in , throttle in , brake in , hand brake, and reverse gear, together with meta-commands for resetting episodes, spawning traffic, selecting weather, and defining sensors (Dosovitskiy et al., 2017).
3. Canonical benchmark design
The initial CARLA-Drive benchmark defined four goal-directed navigation tasks of increasing difficulty and evaluated all methods without task-specific fine-tuning (Dosovitskiy et al., 2017).
| Task | Dynamic objects | Average path length |
|---|---|---|
| Straight | None | 200 m / 100 m in Town 1 / Town 2 |
| One turn | None | 400 m / 170 m |
| Navigation | None | 770 m / 360 m |
| Navigation with dynamic obstacles | Moving cars and pedestrians | Same setting as Navigation with traffic |
Training used Town 1 and four training-weather conditions: Clear day, Clear sunset, Daytime rain, and Daytime after rain. Testing considered four evaluation regimes: training conditions, new town, new weather, and new town plus new weather. The held-out test weather set contained Cloudy daytime and Soft rain at sunset (Dosovitskiy et al., 2017). Each combination of task, town, and weather used 25 episodes. Episodes spawned the car and goal at random locations and ended when the goal was reached or when a time budget based on the optimal path at 10 km/h expired. Infractions were recorded but did not terminate episodes (Dosovitskiy et al., 2017). High-level route guidance was intentionally weak: a topological planner based on A* over the road graph supplied only the next-intersection command—go straight, turn left, or turn right—rather than a dense metric trajectory (Dosovitskiy et al., 2017).
This benchmark produced two enduring findings. First, urban driving in CARLA remained difficult even under training conditions. Under Town 1 and training weather, the modular pipeline, imitation learning, and reinforcement learning achieved 98%, 95%, and 89% on Straight, but only 77%, 83%, and 7% on Navigation with dynamic obstacles (Dosovitskiy et al., 2017). Second, generalization to a new town was substantially harder than generalization to new weather. On Town 1 with held-out weather, the modular pipeline achieved 94% on Navigation and 89% on Navigation with dynamic obstacles, whereas on Town 2 with seen weather it dropped to 24% on both tasks; conditional imitation learning showed the same asymmetry, with 84% and 82% on new weather versus 40% and 38% on new town (Dosovitskiy et al., 2017). This suggests that appearance shifts in illumination and precipitation were less damaging than structural shifts in map layout, urban texture, and intersection configuration.
CARLA also exposed a second layer of evaluation through infractions, reported as average distance between infractions. In Navigation with dynamic obstacles under training conditions, the imitation-learning agent achieved 33.4 km between opposite-lane infractions, compared with 10.2 km for the modular pipeline and 0.18 km for reinforcement learning; conversely, the modular pipeline achieved 16.4 km between collisions with vehicles and 10.0 km between collisions with static objects, outperforming imitation learning on those collision categories (Dosovitskiy et al., 2017). A common misunderstanding is that CARLA-Drive began as a simple success-rate benchmark; in fact, the original protocol already coupled route completion to lane-keeping, sidewalk overlap, and collision structure.
4. Driving system paradigms
The original CARLA comparison defined three archetypes that continued to shape subsequent work. The modular pipeline used RefineNet-based semantic segmentation, an AlexNet-based intersection classifier, a rule-based local planner with states such as road-following, left-turn, right-turn, intersection-forward, and hazard-stop, and three PID controllers for steering, throttle, and brake (Dosovitskiy et al., 2017). Conditional imitation learning mapped a forward RGB camera, current speed, and a high-level command into steering, throttle, and brake through a branched network with four command-specific control heads, trained on about 14 hours of expert data from both automated and human driving (Dosovitskiy et al., 2017). The reinforcement-learning baseline used A3C with two stacked RGB frames, speed, distance to goal, collision damage, and command information, plus a reward combining goal progress, speed change, collision damage increments, sidewalk overlap, and opposite-lane overlap (Dosovitskiy et al., 2017).
Later CARLA-Drive work diversified this design space. One highway-oriented DQN formulation trained in Town04 used a low-dimensional 5×5 observation containing the ego vehicle and four nearby vehicles in normalized lane coordinates, together with five discrete actions—[LANE](https://www.emergentmind.com/topics/latent-autoregressive-network-lane) LEFT, IDLE, LANE RIGHT, FASTER, and SLOWER. Across Town01–Town07 and Town10, the trained agent reduced aggregate collision rate from 96% for an untrained baseline to 68%, increased overall average speed from 6.10 km/h to 8.71 km/h, and improved average time-to-failure from 241 to 373 timesteps (Hossain, 2023). A separate safe-navigation DQN system reduced state dimensionality even more aggressively by extracting four scalars—lateral distance to road center, heading difference, distance to the closest obstacle from segmentation and depth cameras, and speed—and reported 94% success over 100 runs on four trajectories with 35 vehicles and 80 pedestrians (Nehme et al., 2023).
Other work preserved modularity while strengthening planning. PaaS used reactive Frenet-frame planning, heuristic trajectory costs, and a controllable driving-style factor; in the CARLA Autonomous Driving Leaderboard MAPS track it ranked third of nine submissions with a driving score of 48.24 and the strongest infraction penalty, 0.84 (Truong et al., 2023). The Team LRM-B entry in the first CARLA Autonomous Driving Challenge combined point-cloud obstacle detection, YOLOv3-based traffic-sign and traffic-light detection with depth, short-term risk assessment, an MDP for decision-making, and MPC for control, winning three of four tracks and ranking second in the remaining one (Rosero et al., 2020). These systems show that CARLA-Drive remained a fertile environment for classical software architectures rather than being monopolized by end-to-end learning.
Reinforcement learning in CARLA also evolved beyond early A3C and DQN baselines. WAD combined detection-guided latent-state extraction with TD3 and SAC and reported 100% success on all driving tasks in the original CARLA benchmark together with 82% on the NoCrash benchmark, surpassing the prior state of the art by more than +30% on NoCrash (Sharma et al., 2021). Think2Drive moved to model-based RL in CARLA Leaderboard v2, using privileged BEV masks and a recurrent state-space world model; on CARLA v2 test routes it reported 98.6% route completion and a Weighted Driving Score of 91.7 (Li et al., 2024). Imagine-2-Drive replaced one-step latent dynamics with a diffusion-based world model and a diffusion policy actor using only front-camera RGB input, reporting 83.33% Success Rate, 82.13% Route Completion, and 0.70 infractions per km in CARLA (Garg et al., 2024). Raw2Drive then introduced a dual-stream world-model RL framework in which a privileged world model guides a raw-sensor world model through latent-state alignment and head guidance; on Bench2Drive it reported a Driving Score of 71.36 and a Success Rate of 50.24, positioning itself as the only RL-based end-to-end method on CARLA Leaderboard 2.0 and Bench2Drive (Yang et al., 22 May 2025).
A plausible implication is that CARLA-Drive has bifurcated into at least three methodological regimes: modular planning stacks with explicit geometry and rules, privileged-information world-model planners that treat CARLA as a neural-simulation environment, and raw-sensor end-to-end systems that attempt to preserve deployment realism while borrowing privileged structure during training.
5. Scenario generation and data resources
As CARLA-Drive matured, scenario authoring and dataset construction became major research themes. CARJAN integrated CARLA with the AJAN multi-agent framework, storing scenarios as RDF quads in TriG, exposing a browser-based grid editor, and specifying agent behavior with SPARQL Behavior Trees. It supports drag-and-drop authoring of roads, intersections, sidewalks, and no-go areas; path definition via waypoints connected by cubic Bézier curves; decision boxes for event-triggered interactions; and live monitoring of agent behavior-tree execution in CARLA 0.9.15 (Neis et al., 29 Aug 2025). This shifted CARLA-Drive toward semantically explicit, explainable multi-agent simulation rather than imperative Python scenario scripts.
Another line of work focused on edge cases and shared autonomy. A conflict-situation toolkit in CARLA 0.9.15 implemented five high-urgency scenarios drawn from takeover-request literature: total sensor failure via Gaussian image noise, an on-ramp lane-change conflict, a road-narrowing conflict caused by a parked vehicle, danger-zone obstacles blocking the lane, and loss of lane-reference signals through missing or degraded lane markings and adverse weather (Mihaylova et al., 4 Mar 2025). The toolkit used XML scenario descriptions, a controller abstraction, and threshold-based takeover logic to study situation awareness and human–automation interaction in conflict states (Mihaylova et al., 4 Mar 2025).
Synthetic data generation also expanded dramatically. SCaRL built a synchronized multimodal dataset directly on CARLA with 140,000 frames, each containing 6 RGB cameras, 6 semantic/instance cameras, 6 depth cameras, 6 coherent lidars, and 6 MIMO FMCW radars. It is described as the first dataset to include synthetic synchronized data from coherent lidar and MIMO radar sensors, including range–Doppler–azimuth/elevation representations and raw radar outputs (Ramesh et al., 2024). TaCarla, collected in CARLA 0.9.15 on Leaderboard 2.0 training and validation routes, contains over 2.85 million frames at 10 Hz and supports planning, dynamic-object detection, lane-divider detection, centerline detection, traffic-light recognition, prediction tasks, and visual-language action models, while also providing numerical rarity scores for text scene descriptions (Gorgulu et al., 26 Feb 2026). nuCarla approached the same problem from a perception-benchmark angle, constructing a nuScenes-compatible BEV dataset with 1,000 scenarios, a six-camera 360° rig, balanced class distributions, and direct compatibility with nuScenes-format BEV detectors (Qiao et al., 12 Nov 2025).
Benchmark realism also shifted from synthetic towns to digital twins. DriveE2E extracted 800 dynamic traffic scenarios from more than 100 hours of infrastructure-camera video, reconstructed 15 real intersections as CARLA digital twins with matched static assets and environmental conditions, and used log-replayed non-ego trajectories for closed-loop evaluation of an ego policy (Yu et al., 28 Sep 2025). This is technically distinct from the original CARLA-Drive assumption of procedurally spawned traffic: the ego remains closed-loop, but the traffic participants replay real-world motion traces inside CARLA (Yu et al., 28 Sep 2025).
6. Evaluation debates, robustness, and domain expansion
A recurring debate in CARLA-Drive concerns what should count as meaningful progress. One misconception is that the field is fundamentally about route-completion percentages in closed-loop evaluation. In fact, several later works argued that route-level metrics alone can obscure important structure. Think2Drive introduced the Weighted Driving Score, normalizing infraction counts by the number of scenarios rather than route length, because the official CARLA v2 Driving Score applies multiplicative penalties over long routes and can understate per-scenario competence (Li et al., 2024). DriveE2E made a related argument from a different direction: manually configured synthetic traffic can diverge from real-world intersection behavior, so a benchmark grounded in real trajectories and digital twins may better expose failure modes that ordinary CARLA route sets miss (Yu et al., 28 Sep 2025).
A second misconception is that CARLA-Drive is equivalent to camera-only end-to-end learning. The historical record is broader. The original platform explicitly supported pseudo-sensors, full-state measurements, modular planning, and controlled sensor placement (Dosovitskiy et al., 2017). Challenge architectures retained MDP and MPC modules (Rosero et al., 2020), while newer end-to-end methods frequently rely on privileged structure during training, even when raw sensors are used at inference. Raw2Drive is exemplary: it is raw-sensor end-to-end at test time, but its training pipeline depends on a privileged world model and guided alignment (Yang et al., 22 May 2025). This suggests that the boundary between “modular” and “end-to-end” in CARLA-Drive is often methodological rather than absolute.
A third issue is robustness to instruction, scenario, and data distribution shifts. The larger ecosystem represented in the supplied corpus implies several pressure points. New towns remained harder than new weather in the original benchmark (Dosovitskiy et al., 2017). Bench2Drive and Leaderboard 2.0 emphasized long-tail corner cases and exposed the difficulty of urban lane invasion, merging, blocked intersections, and emergency interactions (Li et al., 2024). Dataset papers such as TaCarla and nuCarla argued that perception quality and sensor configuration remain major bottlenecks for end-to-end systems, especially under closed-loop deployment (Gorgulu et al., 26 Feb 2026, Qiao et al., 12 Nov 2025). A plausible implication is that CARLA-Drive increasingly depends on the co-design of simulator realism, dataset structure, perception representation, and evaluation protocol rather than on isolated improvements to control policy alone.
Finally, the scope of CARLA-Drive has broadened beyond passenger-car urban driving. A CARLA-based simulation of electrically driven forklifts used OpenDRIVE warehouse maps, custom vehicle blueprints, A* routing, trajectory replay from real fleets, and a physical battery model to simulate state of charge as a function of load and activity, then applied the resulting platform to traffic-density analysis and charging-station placement (Claus et al., 19 Sep 2025). This does not redefine the core meaning of CARLA-Drive, but it shows that CARLA’s role as a configurable driving-systems laboratory now extends into intralogistics, energy-aware fleet operation, and industrial traffic analysis.
In this expanded sense, CARLA-Drive is best understood as an evolving research domain organized around a single simulator substrate but multiple epistemic styles: controlled urban benchmarking, modular AV architecture, scenario-centric safety analysis, multimodal synthetic data generation, privileged and raw-sensor world-model RL, and real-to-simulation digital-twin evaluation.