FalconGym 2.0: Real-Time UAV Simulation

• FalconGym 2.0 is a photorealistic simulation framework designed for visual aerial navigation using editable Gaussian Splatting for rapid, high-fidelity scene synthesis.
• It implements Performance-Guided Refinement (PGR) to target challenging navigation cases, enhancing policy robustness and cross-domain generalization.
• The framework supports closed-loop learning on simulated and hardware UAV/quadrotor systems, enabling zero-shot sim-to-real transfer with high success rates.

FalconGym 2.0 is a photorealistic simulation framework for visual aerial navigation, based on editable Gaussian Splatting and designed for the development and evaluation of robust vision-based control policies. Addressing overfitting and generalization failures endemic to existing simulation platforms, FalconGym 2.0 provides millisecond-scale, programmatically generated static and dynamic tracks, and supports closed-loop learning and evaluation on both simulated and hardware UAV/quadrotor systems. Its core algorithm, Performance-Guided Refinement (PGR), focuses learning on the most challenging navigation cases, yielding policies that achieve high cross-domain generalization, robustness to scene perturbations, and zero-shot sim-to-real transfer capability (Miao et al., 2 Oct 2025).

1. Architecture and Editable Scene Representation

FalconGym 2.0 is constructed on a Gaussian Splatting (GSplat) paradigm, in which the environment is represented as a set of $N$ anisotropic 3D Gaussians:

$S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$

with mean $\mu_j \in \mathbb{R}^3$, covariance $\Sigma_j \in \mathbb{R}^{3 \times 3}$ parameterized by an orientation $R_j \in \mathrm{SO}(3)$ and per-axis scales $s_j \in \mathbb{R}_{>0}^3$ via $\Sigma_j = R_j \operatorname{diag}(s_j^2) R_j^\top$, color $c_j \in \mathbb{R}^3$, and opacity $\alpha_j \in [0,1]$. Images are rendered by projecting Gaussians onto the image plane and compositing their contributions in depth order, providing photorealistic RGB frames at interactive rates (on the order of tens of milliseconds per $256 \times 256$ image), significantly improving upon NeRF’s real-time limitations and enabling efficient closed-loop simulation.

The Edit API exposes seven primitive operations—add, translate, rotate, scale, duplicate, delete, lighting adjustment—on arbitrary subsets of Gaussians, each implemented as tensor operations. This allows fast (≈0.004 s per call on RTX 4090) programmatic construction of new scene and track geometries by combining, transforming, or removing gate and obstacle models. Both static and dynamic tracks, including moving gates, can be synthesized in milliseconds, supporting scalable generation of diverse training and evaluation scenarios.

GSplat Scene Properties	Specification	Notes
No. of Splats ($S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$0)	$S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$1 million	Photorealistic fidelity
Edit Latency	$S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$2 s	Per API call, RTX 4090 GPU
Rendering Resolution	$S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$3, $S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$4	UAV, quadrotor variants

2. Performance-Guided Refinement (PGR) Algorithm

The Performance-Guided Refinement (PGR) strategy formalizes visual policy learning as a min–max objective over the set of all feasible two-gate segments $S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$5:

$S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$6

where $S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$7 is the policy, $S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$8 is the closed-loop rollout, and the per-segment loss

$S = \{ (\mu_j, \Sigma_j, c_j, \alpha_j) \}_{j=1}^N$9

quantifies both failures and, upon successful completion, gate-centering error ($\mu_j \in \mathbb{R}^3$0: crossing position; $\mu_j \in \mathbb{R}^3$1: gate center).

The PGR procedure iteratively samples grids over $\mu_j \in \mathbb{R}^3$2, rolling out a state-based expert to collect (image, action) data, trains the visual policy on this set, computes per-grid losses over a validation set, weights grid sampling towards harder segments (using a convex combination with uniform to avoid mode collapse), and repeats over $\mu_j \in \mathbb{R}^3$3 refinement iterations (using $\mu_j \in \mathbb{R}^3$4 grids for Dubins-airplane and $\mu_j \in \mathbb{R}^3$5 for quadrotor, $\mu_j \in \mathbb{R}^3$6 mixing).

A summary of the pseudocode and sampling methodology is provided in Algorithm 1 of (Miao et al., 2 Oct 2025), ensuring systematic focus on maximally challenging and error-prone regions of the domain.

3. Simulation, Rendering, and Imitation Learning Pipeline

At each simulation timestep, FalconGym 2.0 executes: (1) dynamics propagation ($\mu_j \in \mathbb{R}^3$7) via either the Dubins-airplane model or a 12-state quadrotor system, (2) frame rendering via the GSplat pipeline (NeRFStudio’s Splatfacto backend), (3) U-Net-based segmentation to extract a gate mask from RGB input, and (4) action selection via either a lightweight MLP (three layers, 128 units) or a classical geometric controller, the latter being employed for hardware quadrotor deployment for higher reliability.

Imitation learning is performed using closed-loop state-based expert rollouts, with datasets typically comprising $\mu_j \in \mathbb{R}^3$8 two-gate segments and $\mu_j \in \mathbb{R}^3$9 million image–action pairs per PGR iteration. Training, scene generation, and editing all execute on a single RTX 4090 GPU; onboard inference for the quadrotor is performed at 8 Hz closed-loop using Adreno 650 GPU on the VOXL 2 compute board.

Component	Architecture/Config
U-Net	5-level encoder–decoder, 32→256 channels
Controller MLP	3 layers, 128 units (UAV)
Image Resolution	$\Sigma_j \in \mathbb{R}^{3 \times 3}$0 (UAV), $\Sigma_j \in \mathbb{R}^{3 \times 3}$1 (quad)
Onboard Hardware	ModalAI Starling 2, VOXL 2, Adreno 650

4. Empirical Performance and Benchmarks

Two central case studies demonstrate FalconGym 2.0’s efficacy:

Fixed-Wing UAV (Dubins Airplane, Arena $\Sigma_j \in \mathbb{R}^{3 \times 3}$2 m)

• State: $\Sigma_j \in \mathbb{R}^{3 \times 3}$3, controls $\Sigma_j \in \mathbb{R}^{3 \times 3}$4, 7 m/s speed.
• Evaluation on three novel tracks—Spatial-S, Random, Moving (dynamic gates at 2 m/s)—with metrics being Success Rate (SR) and Mean Gate Error (MGE).

Key findings:

• Baseline A (NeRF+dual ViT imitation) and B (actor-critic with Swin Transformer) achieve 100% SR on their respective training tracks, but 0–50% SR on unseen tracks.
• FalconGym 2.0 with uniform sampling (no PGR) achieves 100% SR and MGE of 33–47 cm. With PGR, 100% SR is maintained, and MGE is reduced to 15–21 cm.
• Under gate-pose perturbations (random shifts up to 50 cm), FalconGym 2.0's performance closely tracks the expert, losing less than 5% SR (cf. baselines rapidly degrading for $\Sigma_j \in \mathbb{R}^{3 \times 3}$5 cm).

Quadrotor (12-State Model, Arena $\Sigma_j \in \mathbb{R}^{3 \times 3}$6 m)

• Control: body-frame linear velocities + yaw rate, 1 m/s speed.
• Tracks: Left-Turn, Random, Moving (0.25 m/s), same metrics as above.

Key findings:

• Baselines overfit—100% SR on Left-Turn, but 0–67% SR elsewhere.
• FalconGym 2.0 (no PGR): 100% SR, MGE 11–24 cm (simulation); 93–100% SR, MGE 16–21 cm (hardware).
• With PGR: MGE slightly improves to 9–20 cm (sim), 12–18 cm (real); SR on Random in hardware increases to 100%.
• On-board hardware experiments (VOXL 2, 8 Hz, 30 flights & 70 gate crossings): PGR-trained policy crosses 69/70 gates (98.6% SR), with no additional fine-tuning or domain randomization.

5. Generalization and Sim-to-Real Transfer

Policies trained in FalconGym 2.0 generalize zero-shot to unseen track topologies across both case studies, consistently achieving 100% SR without per-track retraining. Robustness tests using randomized gate-pose perturbations confirm sustained high success rates, even as scene parameters deviate from training setpoints. The U-Net-based perception stack and lightweight controller trained in simulation transfer to real-world quadrotor hardware without further adaptation, successfully crossing 98.6% of gates in multi-environment field trials. No domain randomization or real-world retraining beyond gate geometry edits is required for this sim-to-real transfer.

6. Limitations and Prospective Developments

Current limitations include: (1) GSplat’s fidelity for dynamic/moving gates can suffer under severe lighting changes, occasionally introducing rendering artifacts; (2) scaling to complex outdoor or large indoor domains may necessitate tens of millions of Gaussians, possibly affecting interactive editability; (3) the gating approach requires the next gate to be within the field-of-view at transition, restricting applicability to occlusion-rich or heavily cluttered environments.

Future directions outlined in (Miao et al., 2 Oct 2025) include: integrating 4D GSplat representations to support dynamic backgrounds, leveraging reinforcement learning to directly optimize behavioral and efficiency metrics, condensing the two-stage network (U-Net + controller) into a single lightweight high-frequency policy, and evaluating both high-speed ($\Sigma_j \in \mathbb{R}^{3 \times 3}$7 m/s) regimes and advanced aerodynamics for fixed-wing sim-to-real transfer.

In sum, FalconGym 2.0’s programmatic, editable simulation and PGR methodology facilitate rigorous evaluation and training of visual policies that demonstrate strong generalization, robustness, and hardware transfer in aerial navigation contexts (Miao et al., 2 Oct 2025).