Cosys-AirSim: Cinematic UAV Simulation

• Cosys-AirSim is a simulation platform that integrates realistic flight dynamics with a thin-lens cinematic camera for autonomous aerial filming.
• It employs a programmable API and real-time scene control to simulate depth-of-field, bokeh, and complex shot planning.
• The system supports vision-based target tracking, trajectory optimization, and multi-agent shot choreography for UAV control research.

Cosys-AirSim is an advanced simulation and control platform for autonomous aerial cinematography, unifying physically accurate flight dynamics with photorealistic, controllable cinematic camera models and programmable scenario orchestration. Designed for both robotics and film research, Cosys-AirSim (in the literature typically referred to as CinemAirSim and implemented as an AirSim extension) enables rigorous development and evaluation of autonomous cinematography methods, with direct support for real-world camera optics, multi-UAV shot planning, visualization of artistic effects, and export of flight plans for deployment on actual drones. The foundation of Cosys-AirSim is AirSim’s photorealistic Unreal Engine simulation core, augmented with thin-lens optical models and a comprehensive API for real-time camera and lighting control (Pueyo et al., 2020). The environment supports vision-based target localization, real-time mapping, cost-based trajectory optimization under safety and aesthetics constraints, and scripting of complex multi-agent scenarios (Bonatti et al., 2019, Zhang et al., 2020).

1. Motivation and Theoretical Background

Traditional UAV simulators for robotics and control research, such as AirSim and FlightGoggles, employ simple pinhole camera models that maintain all objects in focus and restrict camera properties to fixed intrinsic parameters. In contrast, real-world cinematography relies on deliberate adjustment of camera properties—lens focal length, aperture (f-number), focus distance, shutter speed, and lighting—to shape visual perception using depth-of-field, bokeh, and exposure manipulation.

Two primary deficiencies are identified in the robotics simulation landscape:

• Lack of Cinematic Parameterization: The inability to change lens, aperture, focus, or lighting in simulation hinders research on the interaction between UAV control and artistic shot composition.
• Inadequacy for Cinematic Pipeline Prototyping: Simulators that lack realistic optical effects cannot be used to develop or test autonomous camera-control policies or to generate datasets that accurately reflect operational camera artifacts.

Cosys-AirSim resolves these deficiencies by integrating a physical thin-lens camera with full parameter control, enabling exploration of the entire cinematographic parameter space within a 6-DoF dynamic simulation (Pueyo et al., 2020).

2. Cinematic Camera Model and Mathematical Foundations

The core innovation in Cosys-AirSim’s camera model is replacement of the pinhole projection with a thin-lens model, implemented via Unreal Engine’s UCineCameraComponent. This model supports depth-of-field and blur consistent with cinematographic optics. The following relations underpin the simulation:

• Thin-Lens Focus Equation:

$\frac{1}{f} = \frac{1}{z} + \frac{1}{Z}$

where $f$ is the focal length, $z$ is the lens-to-sensor distance, and $Z$ is the subject’s distance.

• Circle of Confusion (CoC) Diameter:

$c = \frac{|z - Z|}{z} \cdot \frac{f}{N}$

with $N$ as the f-number. CoC defines the degree of blur for out-of-focus objects, modulated by the aperture size (smaller $N$ yields more pronounced bokeh).

• Projection Matrix (3D to Image Plane):

$$\mathbf{x} \sim K \begin{bmatrix} I & \mathbf{0} \end{bmatrix} \mathbf{X}$$

where $K$ is the pinhole-like intrinsic matrix, but due to the thin-lens, points at depth $X \neq Z$ are distributed over a disk of radius $f$0 on the sensor, generating focus and blur.

This modeling supports simulation of realistic rack focus, focus pulls, and artistic optical signatures absent from prior simulators (Pueyo et al., 2020).

3. System Architecture and API Extensions

Cosys-AirSim extends both server- and client-side APIs to permit real-time and programmatic manipulation of all major camera and lighting properties. The API is accessible via both C++ and Python, exposing the following elements:

API Method	Function	Example Parameter(s)
simSetCameraFilmback(camera, w, h)	Set physical sensor size	width_mm, height_mm
simSetCameraLens(camera, lens_profile)	Select detailed lens model	lens_profile_name
simSetCameraAperture(camera, f_number)	Adjust aperture for DoF	float f_number
simSetCameraFocusDistance(camera, focus_m)	Move focal plane	float focus_meters
simSetCameraShutterSpeed(camera, seconds)	Control exposure, enable motion blur	float seconds
simSetLightIntensity(light, intensity_cd)	Vary scene lighting	float intensity_cd

Example usage (Python client):

$f$6

The environment architecture supports seamless integration with asset libraries and complex outdoor/urban scenes via Unreal Engine 4, leveraging photogrammetrically reconstructed mesh import and extensive scenario scripting (Zhang et al., 2020).

4. Shot Planning, Target Tracking, and Control Algorithms

Cosys-AirSim incorporates a modular architecture that enables autonomous or semi-automatic aerial shot planning in complex, dynamic environments (Bonatti et al., 2019, Zhang et al., 2020). This encompasses:

• Vision-Based Target Localization: Monocular, gimbal-stabilized imaging provides 2D detection and tracking (MobileNet-SSD + Kernelized Correlation Filter) fused through raybackprojection (using known camera intrinsics and extrinsics) and constant-velocity Kalman Filtering for trajectory prediction. Heading regression is performed via compact CNN with a loss enforcing temporal smoothness.
• Incremental Signed Distance Map for Safety: Environment occupancy and truncated signed-distance fields (TSDF) are maintained in a voxel grid (typical: 250×250×100 at 1 m resolution), updated online via LiDAR, supporting fine-grained dynamic obstacle avoidance.
• Trajectory Optimization: Described as receding-horizon optimization over a cost function,

$f$1

where $f$2 penalizes jerk, $f$3 enforces shot framing, $f$4 reduces occlusion, and $f$5 penalizes collisions. Covariant gradient descent (CHOMP-style) with precomputed Hessian is used, re-planning at ~5 Hz with typical closed-loop latency of 300–800 ms (Bonatti et al., 2019).

• Shot Parameterization: Pre-set shot types (e.g., Orbit, Chase, Establish) are supported as UI primitives, with keyframe, Catmull-Rom, and spherical linear interpolation for camera path and orientation editing (Zhang et al., 2020).

5. Representative Demonstrations and Experimental Evaluation

Benchmark and illustrative experiments validate the full system and the unique capabilities enabled by camera realism:

• Lens Transition without Platform Motion: Dynamic between 12 mm and 300 mm focal lengths yields zoom effects along fixed UAV trajectory, enabling complex in-shot focal manipulation (Pueyo et al., 2020).
• Rack Focus and Depth-of-Field Control: Autonomous focus sweeps from foreground to distant elements, producing canonical “rack focus” visual effects not possible with pinhole-only models.
• Autofocus Tracking: Live actor-following with feedback-looped focus distance ensures subject sharpness with defocused backdrop as distance varies 10–40 m, showing real-time linkage of UAV pose, sensor data, and camera optics.
• Multi-Drone and Artistic Framing: Planning across agents with cross-focusing and framing constraints—suitable for multi-actor shots—demonstrates the simulator’s extensibility and potential for coordination research.
• Quantitative Results: In vision-based localization, walking-path length error ≈1.7 m over 40 m and circle diameter error ≈0.4 m on 18.3 m radius scenarios are reported. Planner achieves average planning time of 32–69 ms depending on map modality, with 8% cost degradation online versus GT (Bonatti et al., 2019).

6. Scenario Authoring, Asset Integration, and Flight Plan Export

The pipeline for generating realistic scenarios couples photogrammetric environmental capture (virtual globe data or real drone photogrammetry), 3D mesh post-processing, and Unreal Engine import. Asset and coordinate normalization is handled via explicit transformation between georeferenced ENU and UE4’s scene graph. The scenario manager supports both event-driven scripting (e.g., on-collision triggers, time-based events) and state machine logic for assets, enabling complex choreography for drone/actors/crowd (Zhang et al., 2020).

Flight plans can be exported for actual UAV deployment: supported formats include MAVLink waypoint files and DJI JSON scripts. Conversion between simulator world and geographic coordinates is managed explicitly, supporting direct pipeline transfer from simulation to hardware.

7. Research Opportunities and Future Directions

Cosys-AirSim catalyzes several research threads (Pueyo et al., 2020, Bonatti et al., 2019):

• Joint Camera-Trajectory Optimization: Simultaneous planning over UAV motion and all lens/intrinsic parameters to maximize a composite cinematic objective.
• Learning-Based Cinematography: RL agents able to select not just viewpoints but full photographic parameterization for reward functions grounded in “cinematic quality.”
• Multi-Agent Shot Choreography: Coordinated depth-of-field, occlusion handling, framing, and cross-drone focus transitions.
• Robust Perception under Cinematic Effects: Systematic benchmarking of visual algorithm performance (e.g., SLAM, tracking) under real-world photographic effects such as focus blur and motion blur.
• Optics System Identification: Dataset generation under varied lens/aperture regimes for calibration and modeling of physical camera systems.
• Automated Environment Creation: Integration of automated mesh and environment generation pipelines from selected GPS polygons.
• HIL (Hardware-in-the-Loop) and VR Integration: Support for hardware-in-the-loop autopilot testing and immersive VR interfaces for operator training are identified as ongoing directions (Zhang et al., 2020).

By closing the loop between sophisticated camera simulation, realistic robotic dynamics, and scenario authoring, Cosys-AirSim positions itself as a foundational tool for experimental robotics, virtual cinematography, and applied computer vision research (Pueyo et al., 2020, Bonatti et al., 2019, Zhang et al., 2020).