AirSim360: A Panoramic Simulation Platform within Drone View (2512.02009v1)

Abstract: The field of 360-degree omnidirectional understanding has been receiving increasing attention for advancing spatial intelligence. However, the lack of large-scale and diverse data remains a major limitation. In this work, we propose AirSim360, a simulation platform for omnidirectional data from aerial viewpoints, enabling wide-ranging scene sampling with drones. Specifically, AirSim360 focuses on three key aspects: a render-aligned data and labeling paradigm for pixel-level geometric, semantic, and entity-level understanding; an interactive pedestrian-aware system for modeling human behavior; and an automated trajectory generation paradigm to support navigation tasks. Furthermore, we collect more than 60K panoramic samples and conduct extensive experiments across various tasks to demonstrate the effectiveness of our simulator. Unlike existing simulators, our work is the first to systematically model the 4D real world under an omnidirectional setting. The entire platform, including the toolkit, plugins, and collected datasets, will be made publicly available at https://insta360-research-team.github.io/AirSim360-website.

• The paper introduces a panoramic UAV simulation platform that combines six-camera equirectangular rendering with pixel-level semantic, geometric, and instance annotations.
• It employs an interactive pedestrian-aware system and automated trajectory planning to enhance dynamic scene modeling, achieving a 45% boost in per-camera frame rates.
• Extensive evaluations demonstrate significant improvements in depth estimation, segmentation, and vision-language navigation, supporting scalable omnidirectional AI research.

AirSim360: A Panoramic UAV Simulation Platform for Omnidirectional Perception

Introduction

AirSim360 is an omnidirectional simulation platform specifically designed to enable large-scale data collection and evaluation for aerial systems equipped with 360-degree vision. Utilizing the advanced rendering features of Unreal Engine (UE) 5, the platform addresses the scarcity and annotation inefficiency of panoramic datasets in computer vision, with particular emphasis on pixel-level semantic, geometric, and instance-level understanding. Unlike conventional simulators limited to perspective cameras and static scenes, AirSim360 provides dynamic simulation environments incorporating scene diversity, interactive pedestrians, automated UAV trajectory planning, and a comprehensive annotation pipeline, consolidating a closed-loop system capable of supporting a rich spectrum of embodied AI tasks.

Figure 1: The left panel depicts the core interaction architecture of Airsim360, comprising the flight control module, rendering engine, and inference engine. The middle and right panels showcase three purpose-built data generation modules: the Interactive Pedestrian-Aware System, Automated Trajectory Generator, and Data Collection Toolkit.

System Architecture and Data Generation

The AirSim360 platform extends UE5 with custom dynamics and communication modules, facilitating direct and flexible control of UAVs and seamless integration with external perception, control, or vision-language-action (VLA) models. The architecture is organized around three core modules:

1. Render-Aligned Data and Label Generation: AirSim360 employs an array of six pinhole cameras with idealized, distortion-free optics to construct panoramic images using equirectangular projection (ERP). Synchronous GPU-side stitching leverages the Render Hardware Interface, ensuring high-fidelity composite images and pixel-wise ground truth (semantic and entity segmentation, depth, and human keypoints), while mitigating the bottlenecks of sequential capture or CPU-based post-processing.

   Figure 2: The equirectangular projection serves as a standard representation for panoramic imagery, generated from six calibrated pinhole cameras stitched into a continuous panoramic image.

2. Interactive Pedestrian-Aware System (IPAS): The IPAS module synthesizes large numbers of autonomous, realistic human actors capable of meaningful behaviors and social interactions, implemented via hierarchical Behavior Trees and State Machines. The system ensures consistent temporal annotation of 3D keypoints, enabling complex human-centric tasks such as monocular 3D localization, behavior understanding, and multi-agent navigation.

   Figure 3: Conceptual diagram of the IPAS logic, showing the interplay between the Behavior Tree, State Machine, and the message-passing system for autonomous agent interaction.

3. Automated Trajectory Generation: AirSim360 supports efficient automated and user-driven UAV data acquisition, with Minimum Snap trajectory planning translating sparse user waypoints into dynamically feasible, smooth flight paths. Kinematic constraints are respected, and resulting trajectories are directly transferable to physical UAVs.

The simulation pipeline enables seamless capture of panoramic RGB, depth, semantic and instance labels, and high-frequency human pose data synchronized along the trajectory.

Datasets and Annotation Paradigms

Leveraging the scalable simulation engine, AirSim360 provides Omni360-X, a comprehensive panoramic dataset. It offers:

• Omni360-Scene: 60K+ images across four urban scenarios, densely annotated for depth, semantic categories (20–29 per scene), and entity segmentation, supporting scene parsing and pixel-level recognition tasks.
• Omni360-Human: 100K samples with diverse pedestrian behaviors, consistent 3D keypoint labels, and varied spatial configurations, facilitating monocular localization and behavior analysis.
• Omni360-WayPoint: 100K+ UAV waypoints sampled along physics-constrained trajectories in various kinematic regimes, enhancing research in control, trajectory prediction, and system identification.

  Figure 4: Visualization of Omni360-X; a) shows render-aligned data and labels, b) demonstrates panoramic pedestrian-aware annotations, c) illustrates trajectory synthesis from sparse waypoints.

Key annotation advances include full entity segmentation (beyond label count restrictions inherent to classic stencil buffers), slant-range-consistent omnidirectional depth, hierarchy-enforced semantic categories, and synchronous multi-sensor rendering.

Experimental Evaluation

Ablations and System Performance

Benchmarks on a moderate workstation (RTX 4060 Ti, i7-14700) demonstrate that the GPU-side pipeline and asynchronous event-driven sensor synchronization provide a 45% increase in per-camera frame rate and overall higher data throughput versus sequential or blueprint-based approaches. The custom C++ RHI interface improves the six-camera panoramic rendering speed from 14 to 18 FPS under full sensor load.

Perception Tasks

Monocular Pedestrian Distance Estimation

Synthesized data from Omni360-Human, when used to augment real-world pedestrian datasets (e.g., nuScenes), produce consistent improvements for monocular pedestrian distance estimation. For example, average angular error is reduced from 21.2° to 17.0°, and Euclidean error from 0.484 m to 0.458 m over real test sets, with FreeMan showing marked improvements (17.0° to 11.6° angular error).

Panoramic Depth Estimation

UniK3D depth estimation models trained on Omni360 outperform those trained on Deep360 in both out-of-domain (SphereCraft) and cross-domain evaluations. For SphereCraft, Omni360 achieves AbsRel of 5.44 (vs. 8.26 from Deep360), RMSE of 0.0435 (vs. 0.0566), and $\delta_1$ of 0.399 (vs. 0.349). These results highlight the enhanced transferability and robustness of omnidirectional, physically consistent data from AirSim360.

Segmentation

Combining Omni360-Scene with prior panoramic segmentation sets (WildPASS) produces substantial gains in both semantic segmentation (mIoU: 58.0 → 67.4) and entity segmentation (mAP: 24.6 → 38.9) on standard panoramic benchmarks, demonstrating the benefits of scaling and diversity for pixel-wise scene parsing.

Vision-Language Navigation (VLN)

The panoramic-VLN benchmark introduced in AirSim360 confirms that panoramic UAV perception facilitates near-optimal goal reach without the need for exploratory maneuvers, validating the "You Only Move Once" (YOMO) paradigm for targets visible in the omnidirectional field.

Implications and Future Directions

Practically, AirSim360 enables expedited research and benchmarking in panoramic scene parsing, omnidirectional navigation, multi-modal fusion, vision-language instruction following, and human-UAV interaction. The closed-loop simulation model—incorporating dynamic human actors, photorealistic rendering, and scalable, annotation-rich data capture—lowers the domain gap for sim-to-real transfer. Theoretically, the platform advances the study of embodied intelligence under full spatial coverage, supporting AI systems that require comprehensive situational awareness and spatial reasoning from aerial vantage points.

With the public release of code, datasets, and toolkits, subsequent improvements in generative models, multi-agent simulation, and cross-modal fusion are expected to accelerate. Potential directions include:

• Scaling up scenario complexity (more diverse agents, environments, lighting/weather conditions)
• High-fidelity simulation for reinforcement learning and real-world robotic transfer
• Integration of real-world sensor models (distortion, blur, noise)
• Robust benchmarking for 4D spatial-temporal tasks, including future-prediction, crowd modeling, and multi-agent coordination in panoramic space

Conclusion

AirSim360 establishes a foundational platform for panoramic UAV research, offering an integrated simulation pipeline for scalable, omnidirectional data generation, human-in-the-loop interaction, and physically plausible flight control. Quantitative benchmarks across multiple perception and reasoning tasks confirm the platform's ability to deliver diverse, high-quality synthetic data that is both efficient for large-scale collection and effective for real-world transfer. The release of AirSim360 catalyzes further advances in panoramic embodied AI and serves as a blueprint for future simulation-driven large-scale aerial intelligence research.