AeroScene: UAV Datasets & 3D Scene Synthesis

Updated 3 July 2026

AeroScene is a comprehensive resource integrating densely annotated UAV datasets with multi-modal data (RGB, depth, semantic) for aerial scene understanding.
It leverages scalable simulation pipelines and hierarchical denoising diffusion models to generate physically plausible 3D environments, validated by improved UAV benchmark mIoU metrics.
The framework supports controlled variation studies and end-to-end simulation for autonomous UAV tasks, demonstrating robust sim-to-real transfer in diverse weather and layouts.

AeroScene is a collective designation for a set of datasets and generative frameworks developed primarily for aerial scene understanding, synthetic scene synthesis, and photorealistic simulation in the context of unmanned aerial vehicle (UAV) perception, robotics, and computer vision. The term encompasses multiple prominent works, notably synthetic image datasets for dense semantic perception, progressive generative models for 3D scene creation, and specialized benchmarks for hyperspectral semantic segmentation. Each AeroScene contribution targets critical bottlenecks in aerial robotics, including the lack of densely annotated datasets under diverse conditions, scalable high-fidelity 3D scene synthesis for simulation and navigation, and improved physical realism and semantic richness in synthetic environments.

1. Synthetic Aerial Scene Datasets and Annotations

AeroScene includes a large-scale, densely annotated dataset designed for UAV-perspective scene understanding, as articulated in the derivative work on SkyScenes (Khose et al., 2023). This resource offers 33,600 high-resolution (2160 × 1440) images sampled across eight virtual towns (four urban, four rural), capturing a wide spectrum of scene layouts in terms of road network density, building typology, vegetation coverage, and object distribution. Environmental variation is introduced by capturing each camera viewpoint under five distinct weather and illumination settings: ClearNoon, ClearSunset, MidRainNoon, CloudyNoon, and ClearNight.

AeroScene supports three complementary per-pixel annotation modalities:

Semantic segmentation: 28-class dense labeling, spanning static infrastructure, nature, dynamic actors, and catch-all labels.
Instance segmentation: Unique ID assignment to each object instance, with specific challenges in oblique views due to occlusions and the small size of tail-class objects.
Depth maps: Logarithmic-format depth from CARLA’s simulator, allowing ground-truth, centimeter-scale precision, critical for depth-aware perception.

These features enable both pixel-level learning and robust diagnostic analysis under controlled, repeatable conditions.

2. Data Generation, Modalities, and Benchmarking

AeroScene leverages the CARLA simulator (v0.9.14) for scalable, reproducible dataset generation. Sensor configurations include RGB, semantic, instance, and depth channels with matching camera parameters (FoV=110°, resolution=2160×1440). The data-generation pipeline features waypoint-driven traversal at approximately 5 m intervals, procedural dynamic actor spawning (notably for pedestrians), and systematic variation in UAV altitude (three nominal heights with Gaussian jitter) and pitch (four settings with jitter).

Key quantitative findings demonstrate AeroScene’s value as a pre-training corpus:

Zero-shot Sim→Real transfer: DAFormer–MiT-B5 models trained on AeroScene consistently outperform previous synthetic datasets (e.g., SynDrone) on real-world UAV benchmarks, as shown by mIoU metrics:
- UAVid: 47.09% (AeroScene) vs. 42.31% (SynDrone)
- AeroScapes: 40.72% vs. 30.53%
- ICG Drone: 25.91% vs. 15.92%
Few-shot data augmentation: In regimes with only 10% of UAVid labels, AeroScene pre-training or joint-training increases DeepLabv2 mIoU from ~42% (real only) to ~50%.

These results highlight the dataset’s significance for addressing real-world generalization, especially when dense real annotations are limited (Khose et al., 2023).

The meticulously structured metadata enables controlled studies, where only one covariate (e.g., pitch, height, weather, or layout) is varied at a time:

Height and pitch: Best mIoU is achieved for oblique angles (θ=45°, 60°), with degraded performance at nadir (θ=90°) or forward (θ=0°).
Weather and time-of-day: Models trained on a specific condition (e.g., ClearNoon) generalize poorly to different conditions; the best performance is always in-matched.
Layout dependency: Models trained on rural layouts perform markedly worse on urban scenes and vice versa, underscoring the significance of domain diversity.

AeroScene also supports multimodal learning strategies. Fusing RGB and depth input in a multi-modal segmentation model (M3L-MiT-B5) improves semantic segmentation mIoU from 80.01% (RGB only) to 83.22% (RGB + D). Depth input is especially beneficial for instance segmentation in scenes with complex geometry or overlapping objects (Khose et al., 2023).

4. Hierarchical Generative Scene Synthesis

AeroScene denotes a progressive 3D scene synthesis framework that employs hierarchical denoising diffusion probabilistic models (DDPMs) for automatic, large-scale generation of physics-ready, semantically diverse, and physically plausible virtual environments for aerial robotics simulation (Vu et al., 24 Mar 2026). Unlike traditional flat-set DDPMs, the AeroScene generator operates on a two-branch hierarchical tokenization:

Coarse-scale tokens represent major scene elements; processed via a 3D-CNN.
Fine-scale tokens encode detailed objects and local structures; processed via a spatial adjacency graph GNN.

Cross-scale progressive attention alternates between global structure (top-down) and local refinement (bottom-up), yielding scene layouts with strong physical and semantic consistency.

Physical validity is enforced via three differentiable guidance objectives:

Collision avoidance: Penalizing excessive bounding box intersection.
Coarse-to-fine consistency: Ensuring fine objects are appropriately placed relative to the coarse context.
Semantic constraints: Imposing empirical priors on the relative distribution of categories.

In experiments, AeroScene outperforms prior works (ATISS, Diffusion-SDF, DiffuScene, PhyScene) on metrics including FID (27.3), KID (0.021), collision rate (6.2%), coarse-to-fine consistency, and semantic plausibility. An ablation shows substantial increases in collision rate without explicit avoidance guidance (Vu et al., 24 Mar 2026).

5. End-to-End Simulation and Downstream Evaluation

The AeroScene synthesis pipeline directly produces NVIDIA Isaac Sim–compatible assets: object transforms result in collision meshes and physical parameters, semantic categories map to materials and sensor properties, and annotated landing zones are exported for control evaluation. A dataset of 1,016 such scenes (160,010 objects, 23 coarse/47 fine semantic categories, 54 landing areas per scene) provides the foundation for benchmarking autonomous UAV tasks.

Downstream validation involves two-phase drone missions per scene:

Long-range navigation via graph-based planning and geometric control.
Landing/perching using point-cloud–based touchdown region selection and precision descent.

Evaluated with physical drones (3DR Iris, AscTec Hummingbird) and 300 trajectories per scene, environments generated by AeroScene yield a 91% mission success rate, exceeding static handcrafted simulators in realism and challenge (Vu et al., 24 Mar 2026).

AeroScene techniques and benchmarks relate closely to other prominent works:

SkyScenes: The direct antecedent of the AeroScene image dataset, setting standards for densely annotated, multimodal aerial image data under controlled variation. Its findings on sim-to-real transfer and the impact of depth are central to AeroScene's value proposition (Khose et al., 2023).
Skyeyes: Provides a blueprint for geometry-conditioned ground-to-aerial synthesis using Surface-Aligned Gaussian Splatting (SuGaR), diffusion-based appearance control, and temporal view-consistency; this enables the real-time generation of photorealistic ground-level imagery from aerial inputs, critical for urban simulation and autonomous driving research. AeroScene can be implemented by adapting this pipeline for UAV-centric data (Gao et al., 2024).
AeroRIT: The AeroScene label is also associated with the largest urban hyperspectral semantic segmentation dataset, collected at ~0.4 m GSD with 372 contiguous VNIR bands. Baselines include SegNet, U-Net, and Res-U-Net architectures, with enhancements via squeeze-and-excitation blocks and self-supervised encoder initialization. The dataset enables vehicle detection/tracking, spectral unmixing for occlusions, and robust benchmarking for hyperspectral airborne sensing (Rangnekar et al., 2019).

7. Applications, Limitations, and Future Directions

AeroScene datasets and generative frameworks support diverse applications:

Aerial surveillance: Detection and tracking of people and vehicles over changing urban/rural scenes and atmospheric conditions.
Autonomous navigation: End-to-end policy training and benchmarking on high-fidelity, physically plausible virtual worlds.
Precision mapping: Semantic height-layer extraction and detailed infrastructure modeling using multi-modal and hierarchical data.
Cross-domain image synthesis: Real-time populating of 3D worlds with geometrically consistent ground views for simulation, gaming, or visualization.

Limitations remain in photorealism, sensor physics modeling (material reflectance, noise), dynamic object integration, and generalization to complex real-world domains. Future research is focused on the inclusion of dynamic or temporal elements, domain adaptation for bridging simulated and real data, multi-modal fusion (LiDAR, IR), more granular level-of-detail synthesis, and extension to broader classes of embodied agents (Khose et al., 2023, Gao et al., 2024, Vu et al., 24 Mar 2026, Rangnekar et al., 2019).