---
title: CARLA-Based Dataset for Autonomous Research
url: https://www.emergentmind.com/topics/carla-based-dataset
type: topic
---

# CARLA-Based Dataset for Autonomous Research

A CARLA-based dataset is a multimodal, temporally aligned collection of sensor data generated within the CARLA open-source automotive simulator. These datasets are engineered for research in autonomous driving, robotics, computer vision, adversarial robustness, simulation-to-real (sim2real) transfer, and related disciplines. Typical data modalities encompass RGB and depth imagery, LiDAR and Radar point clouds, GNSS/IMU logs, ground-truth object labels (semantic, instance, or detection), and emerging sensor types such as event-based vision streams. The scale, structure, annotation protocols, and extensibility of CARLA-based datasets are determined by the synthesis pipeline, simulation parameters, and research objectives of the dataset developers.

## 1. Architectural Principles and Data Generation Pipelines

CARLA-based datasets are generated by interfacing with the CARLA Python API, which exposes fine-grained control over scene construction, sensor rig configuration, environment parameters (weather, lighting), agent spawning, and ground-truth data extraction. Recent frameworks automate dataset creation via GUI-driven tools, configuration file templates, or scenario scripting.

For example, the Car-STAGE framework centralizes configuration through a graphical interface, translating user-defined criteria (selected maps $M$, sensor set $S$, environmental conditions $E$, lighting $L$, actor counts, frame rate, episode length) into automated, synchronous data capture [2503.03100]. The pipeline is modular and multithreaded: a main thread orchestrates the CARLA server and simulation ticks, a queue worker thread tags and batches incoming frames, and multiple executor threads process and write each sensor's raw data to pre-allocated memory-mapped files. This design ensures frame-level sensor synchronization and efficient pipeline utilization.

An example workflow proceeds as:
1. User configures scenario via GUI;
2. Car-STAGE launches CARLA, spawns traffic and sensors, and enters a synchronous acquisition loop at the specified FPS;
3. Each sensor's thread writes raw binary to a direct-access mmap region;
4. After all episodes, worker pools convert binary blobs to canonical machine learning formats (PNG, PCD, CSV, etc.).

Example pseudocode for the launch and acquisition steps is included in [2503.03100, Section 1.c].

Other frameworks instantiate complex scenario trees (long-tail maneuvers [2602.23499]), closed-loop perception/planning feedback [2511.13744], or adversarial manipulations (e.g., physically injected patch attacks on billboards [2206.04365]).

## 2. Sensor Suites, Environmental Diversity, and Data Modalities

CARLA-based datasets leverage the simulator's rich sensor abstraction—including, but not limited to, the following modalities:
- **Vision:** Pinhole RGB, depth, semantic/instance segmentation, optical flow, event-based DVS;
- **Range:** LiDAR (RayCast, semantic, coherent, Doppler-enabled), MIMO FMCW radar (Range-Doppler, Range-Azimuth-Elevation cubes);
- **Localization:** GNSS, IMU (accelerometer, gyroscope);
- **Hybrid:** Bird's Eye View projections (e.g., nuCarla BEV), panoptic segmentation.

Sensor parameters (intrinsics/extrinsics, FOV, baseline, mounting geometry) are precisely logged per session—for example, nuCarla matches the nuScenes suite (six 1600×900 RGB cameras at canonical locations, calibrated intrinsics/extrinsics, ego pose logs) [2511.13744], TaCarla mirrors this with LiDAR and radar integration [2602.23499], and SCaRL provides six rigidly attached suites of RGB, semantic, depth, LiDAR, and radar, perfectly synchronized in ego-vehicle and world frames [2405.17030].

Environmental diversity is engineered via randomization or structured factorial design:
- **Weather:** presets (Clear, Cloudy, Rain, Fog, Snow) and controlled interpolation (domain-shift sweeps in SEVD [2404.10540]).
- **Lighting/Daytime:** random seeds or sliders spanning noon, sunset, dusk, night.
- **Traffic/Actors:** deterministic or random actor spawning, diverse vehicles, nuanced behaviors (car-following, gap-acceptance [2601.12119]).
- **Scenario Complexity:** benchmarked challenge scenarios, e.g., long-tail events (emergency yielding, construction, cut-ins [2602.23499]), roundabout merging/yielding (factorial LOS x weather in CARLA-Round [2601.12119]), closed-loop feedback simulations.

## 3. Data Representation, Formats, and Annotation Schemas

Standardized outputs and annotation conventions enable reproducibility and cross-benchmarks. Sensor data is typically exported in:
- **Images:** PNG (8/16-bit for vision/range), JPEG (for downstream compressed storage), resolvable up to 2160×1440 (e.g., SkyScenes aerial scenes [2312.06719]).
- **Point Clouds:** PCD (ASCII or binary), .bin (float32 per point: $[x, y, z, \mathrm{intensity}]$). LiDAR and radar outputs sometimes include per-point semantic, instance, velocity information [2405.17030].
- **Tabular:** CSV logs (GNSS, IMU, annotation JSONs, trajectory files).
- **Specialized:** numpy .npz arrays for event streams or raw radar cubes (used in SEVD and SCaRL).
- **Directory Structure:** Hierarchical, by modality/sensor/date/scene, with canonical naming (camera_front/00000.png, lidar/00000.bin, etc.) [2503.03100].

Annotation schemas are explicit and scenario-specific:
- **Object Detection:** 2D/3D bounding boxes, projected via sensor transforms, with per-actor class, id, orientation, occlusion/visibility tags. Various standard formats are supported—COCO JSON (2D), KITTI, nuscenes-devkit-compatible JSON (3D, BEV). Occlusion filtering addresses the "ghost box" problem by cross-referencing segmentation pixels within detection boxes [2509.16773].
- **Segmentation:** Dense per-pixel labels, instance ID masks, panoptic codes; 23–36 class taxonomies harmonized with Cityscapes/KITTI [2204.09788, 2111.11348].
- **Trajectories:** Agent-centric state logs (position, velocity, heading, behavioral flags: yielding, merging, occluded), sampled at fixed rates (e.g., 10 Hz in CARLA-Round [2601.12119]).
- **Event/Optical Flow:** Event tuples $(x, y, p, t)$, timestamped with microsecond precision, temporally aligned with ground truth grayscale/flow [2412.09209].

Formats and schemas are usually published alongside code for conversion and data loading (e.g., MMDetection3D adapter for nuCarla [2511.13744]).

## 4. Performance Benchmarks, Baselines, and Analysis

CARLA-based datasets routinely publish baseline results using established architectures for each supported perception or planning task. Evaluation protocols are congruent with established standards from the relevant vision/robotics subfields.

- **Detection:** mAP (mean Average Precision) at varying IoU thresholds (0.5 for COCO/KITTI, 0.7 for stricter BEV), NDS (nuScenes detection score), per-class, per-scenario breakdowns [2511.13744, 2602.23499, 2405.17030].
- **Segmentation:** mIoU across all classes, per-class IoU, Panoptic Quality (PQ), class-weighted or instance-weighted accuracy [2312.06719, 2204.09788, 2111.11348].
- **Planning/Trajectory Prediction:** Average Displacement Error (ADE), Final Displacement Error (FDE), Action/Heading Error (AHE/FHE) for multi-step prediction, zero-shot transfer to real-world datasets [2601.12119, 2602.23499].
- **Closed-Loop Control:** Driving Score, Route Completion, Penalty metrics (CARLA Leaderboard framework in TaCarla [2602.23499]).
- **Adversarial Robustness:** AUROC for attack/defense discrimination, accuracy drop from clean to patched scenes, recoverability by defense methods (e.g., Z-Mask, LGS) [2206.04365].
- **Sim-to-Real Transfer:** Segmentation/odometry error deltas before and after fine-tuning, mIoU and ATE for synthetic vs. real test data [2111.11348, 2109.00892].

Results yield quantitative insight into factor impacts (e.g., traffic density's monotonic effect on prediction error [2601.12119]), model generalization, and the efficacy of transfer or domain adaptation methods.

## 5. Dataset Scale, Organization, Reproducibility, and Access

CARLA-based datasets vary widely in scale, with typical datasets comprising tens of thousands to millions of frames or time steps:
- **Large-scale:** TaCarla (2.85 million frames, 79 h drive time, full scenario stratification) [2602.23499]; SELMA (20M sensor samples, 30,909 unique waypoints over 216 scene/view combinations) [2204.09788]; SCaRL (140,000 frames × 6 sensor suites) [2405.17030].
- **Moderate-scale:** nuCarla (40,000 samples, 459,632 objects, fully compatible with nuScenes devkit) [2511.13744]; Car-STAGE (e.g., 10 runs × 60 s × 30 FPS = 18,000 frames) [2503.03100].
- **Specialized:** Adver-City (24,000 frames, 890,000+ 3D bounding box annotations under adverse conditions) [2410.06380]; SKoPe3D (25,000 images, 4.9M 3D keypoint annotations [2309.01324]; Paris–CARLA-3D (700M points synthetic, 60M points real) [2111.11348].

Datasets are commonly released with full scenario scripts, YAML configs, code for both reproduction and data loading, and detailed system requirements (e.g., CARLA version, OS, hardware). Many pipelines publish open-source repositories for the full generation/processing/benchmarking cycle [2503.03100, 2511.13744, 2206.04365, 2405.17030, 2412.09209]. Reproducibility steps include random seed logging, per-frame sensor/resource usage metadata, and mapping baked-in scenario randomness to configuration state.

## 6. Advances in Synchronized Multimodal, Adversarial, and Photorealistic Data

CARLA-based datasets continue to drive advances in three principal areas:
- **Multimodal Synchronization:** High-frequency, precisely synchronized acquisition of RGB, depth, semantic, LiDAR, radar (including Range-Doppler-Azimuth-Elevation representations) and event-based streams, often under full or partial sensor-fusion calibration [2405.17030, 2404.10540]. Emerging datasets incorporate coherent LiDAR and MIMO radar, enabling research into sensor-fusion models and non-vision detection.
- **Adversarial Example Benchmarking:** Physical patch-injection frameworks (e.g., CARLA-GeAR [2206.04365], adversarial mesh streaming [2207.10719]) provide standardized evaluation environments for adversarial attacks and defense strategies, integrating mesh-aware, differentiable patch optimization and in situ environmental transforms (lighting, occlusion, motion).
- **Sim2Real Alignment:** Photorealistic enhancement tools such as CARLA2Real utilize GAN-based, G-buffer-conditioned style transfer to generate pairs of synthetic and "enhanced" images, reducing domain gaps at both the appearance and feature-distribution levels. These approaches are validated by mIoU improvements in segmentation tasks and feature cosine-similarity metrics versus real datasets (Cityscapes, KITTI) [2410.18238].

## 7. Canonical Use Cases and Extensions

CARLA-based datasets serve a broad set of research and development purposes:
- **Autonomous driving perception and planning:** End-to-end closed-loop learning (nuCarla, TaCarla), scenario-driven prediction (CARLA-Round), rare-event handling.
- **Sim2Real Transfer and Curriculum Learning:** Pre-train on synthetic data, fine-tune on limited real data, and evaluate generalization on held-out real-world benchmarks (KITTI, rounD, UAVid, etc.).
- **Robustness to Adverse or Rare Events:** Evaluate models under systematically varied and adversarial conditions (weather, patch attacks, density).
- **Benchmarks for Multi-modal Sensor Fusion:** Develop and compare sensor-fusion (camera–LiDAR–radar) architectures for detection, segmentation, and tracking.
- **New Sensing Modalities:** Event camera datasets for optical flow and traffic analysis [2404.10540, 2412.09209].

A plausible implication is that the proliferation of scalable, richly annotated, open CARLA-based datasets, combined with robust pipelines for factorized scenario control, multithreaded acquisition, and postprocessing, will continue to provide indispensable resources for the quantitative development, benchmarking, and real-world translation of autonomous systems research [2503.03100, 2511.13744, 2206.04365, 2404.10540, 2412.09209, 2602.23499, 2601.12119, 2111.11348, 2410.06380, 2405.17030].

Source: https://www.emergentmind.com/topics/carla-based-dataset