SkyDreamer: Autonomous Drone Racing

• SkyDreamer is an end-to-end vision-based autonomous drone racing system that employs model-based reinforcement learning with privileged-state decoding to achieve champion-level performance.
• It utilizes a recurrent state-space model to process pixel-level inputs and IMU data, mapping them directly to low-level motor commands with rapid onboard inference.
• Empirical results highlight its robustness in sim-to-real transfer, precise parameter estimation, and agile performance in complex aerobatic maneuvers on an NVIDIA Jetson Orin NX.

SkyDreamer is an end-to-end vision-based autonomous drone racing (ADR) system that employs a model-based reinforcement learning (MBRL) approach rooted in privileged-state decoding. SkyDreamer is the first ADR policy to simultaneously achieve full sim-to-real transfer, champion-level performance, and entirely onboard execution, mapping pixel-level inputs directly to low-level motor commands. The system leverages a world model that functions as an implicit state and parameter estimator, enhancing both interpretability and sim-to-real robustness, while bypassing the need for extrinsic camera calibration and enabling rapid deployment across disparate hardware platforms (Verraest et al., 16 Oct 2025).

1. Model Architecture and Training Objectives

SkyDreamer’s core is a recurrent state-space model (RSSM) closely related to Dreamer V3, augmented with privileged-state decoding. At each timestep $t$, the model processes:

• A deterministic hidden state $h_t \in \mathbb{R}^H$ updated by a GRU,
• A stochastic latent state $z_t \in \mathbb{R}^Z$ inferred from observations,
• A dynamics predictor to imagine future latents.

Input and Encoding

Inputs consist solely of a $64 \times 64$ binary gate segmentation mask, IMU body-rate measurements $(\hat\Omega)$, motor RPMs $(\hat\omega)$, and a flight-plan vector $(\mathbf f)$ encoding the relative geometry of upcoming gates. The encoder $q^e_\theta(z_t|h_t,o_t)$, constructed as a CNN followed by an MLP, infers $z_t$ from these observations.

Decoding and Training

During training, the world model includes decoders to reconstruct:

• Privileged states and parameters (ground-truth pose, velocity, orientation, motor parameters, camera extrinsics) via $p^i_\theta(i_t|h_t,z_t)$,
• Scalar reward ($h_t \in \mathbb{R}^H$0),
• Continue probability ($h_t \in \mathbb{R}^H$1).

The training objective maximizes an ELBO summed over timesteps:

$h_t \in \mathbb{R}^H$2

Privileged-state decoding, rather than visual image reconstruction, supplies the primary self-supervision signal during training.

2. Reinforcement Learning in Latent Space

Upon completing world model training, policy optimization operates entirely via imagined trajectories in latent space. The actor $h_t \in \mathbb{R}^H$3 and critic $h_t \in \mathbb{R}^H$4 are trained by sampling sequences from the dynamics predictor $h_t \in \mathbb{R}^H$5 for a fixed horizon ($h_t \in \mathbb{R}^H$6 timesteps, approximately 0.18 s). The policy is optimized to maximize

$h_t \in \mathbb{R}^H$7

using a REINFORCE estimator with rewards derived from the predicted reward head $h_t \in \mathbb{R}^H$8. To prevent bang–bang actuation, a smoothness regularizer on the policy mean $h_t \in \mathbb{R}^H$9 penalizes large changes in consecutive actions:

$z_t \in \mathbb{R}^Z$0

3. Interpretability and Implicit State/Parameter Estimation

A privileged-state decoder $z_t \in \mathbb{R}^Z$1 learns to recover global and gate-relative positions $z_t \in \mathbb{R}^Z$2, velocities $z_t \in \mathbb{R}^Z$3, orientation $z_t \in \mathbb{R}^Z$4, body-rates $z_t \in \mathbb{R}^Z$5, propeller speeds $z_t \in \mathbb{R}^Z$6, camera extrinsics $z_t \in \mathbb{R}^Z$7, and mechanical parameters $z_t \in \mathbb{R}^Z$8 from $z_t \in \mathbb{R}^Z$9. These estimates, available during test-time for logging but not control, allow direct inspection of the agent’s internal state and understanding of its parameter identification. For instance, the maximum attainable motor RPM $64 \times 64$0 rapidly converges within the first second of flight to within 5% of ground truth, and position and velocity converge to sub-decimeter errors within 10 simulation steps. Such features support precise debugging, interpretation, and assessment of the system's sim-to-real transfer capabilities.

4. Deployment Pipeline and Sim-to-Real Robustness

SkyDreamer runs fully onboard an NVIDIA Jetson Orin NX (16 GB), requiring no external computation or networked sensing. Its deployment pipeline integrates:

• 175° FOV rolling-shutter Arducam IMX219 at 90 Hz,
• Segmentation via GateNet U-Net in $64 \times 64$13 ms, incorporating StochGAN and random erosion augmentations for domain randomization,
• Inference (encoder, GRU, and policy) in 1.3 ms, yielding four low-level motor commands at 90 Hz.

The world model is trained with varied camera extrinsics but fixed intrinsics, decoding the extrinsics on-the-fly during flight and enabling immediate platform interchangeability—i.e., plug-and-play deployment with no recalibration or retraining.

Flight-plan logic leverages a vector input encoding the next three gates' relative yaws and positions, which is advanced when the decoded gate-relative x-coordinate $64 \times 64$2. This mechanism enables the policy to resolve visually ambiguous track segments, such as overlapping gates or mirrored structures.

Despite severe visual artifacts in the domain (e.g., poor-quality segmentation, segmentation mask blur, or false positives with dark MAVLab gates), SkyDreamer completed 25 consecutive laps without crash. It further demonstrated adaptation to real-world dynamical change—such as battery depletion (30% drop in maximum RPM, outside training distribution) by real-time estimation of thrust parameters and automatic reduction in aggressiveness during flight.

5. Experimental Results and Performance Metrics

SkyDreamer attains champion-level agility in real-world racing scenarios. Empirical results include:

• Maximum speed up to 21 m/s and accelerations reaching 6 g,
• Small-track inverted loop: mean lap time $64 \times 64$3, 100% completion across 25 laps,
• Ladder-inverted-loop: mean $64 \times 64$4, 100% completion,
• MAVLab inverted loop: mean $64 \times 64$5, 100% completion.

Complex aerobatic maneuvers—tight ladder passes (within 1 m funnels), split-S, and inverted loops (radius $64 \times 64$6 1.5 m)—were performed without explicit perception-based rewards; the policy oriented its camera toward gates emergently. Simulation studies revealed that mechanical parameters and states converge rapidly: $64 \times 64$7 and thrust coefficient within 50–100 steps ($64 \times 64$80.6–1.1 s), positions within 0.1 m, and velocity within 0.5 m/s over 10 steps. Some parameters, such as motor drag or single-motor constants, exhibited drift over long horizons.

A comparative analysis with previous end-to-end ADR methods demonstrated that SkyDreamer is the first to achieve full onboard execution, champion-level thrust-to-weight ratio, nontrivial visual sim-to-real transfer, interpretable state decoding, and resilience to visual ambiguity, all in conjunction.

The following table summarizes representative flight results:

Track Condition	Mean Lap Time (s)	Success Rate (%)	Peak Speed (m/s)
Big Track (large hall)	N/A	N/A	21
Small-track Inverted Loop	3.25 ± 0.22	100	N/A
Ladder-inverted-loop	3.62 ± 0.06	100	N/A
MAVLab Inverted Loop	2.97 ± 0.08	100	N/A

6. Limitations and Future Extensions

Identified limitations include:

• Parameter estimates may exhibit drift over extended mission durations, with outcomes dependent on the training run.
• Decoded state estimates display high-frequency jitter due to the model’s discrete latent representations.
• The system is susceptible to segmentation false positives in the absence of explicit physical gate-blocking strategies.
• Training remains computationally intensive, typically requiring approximately 50 hours on an NVIDIA A100 GPU.

Planned future directions include expanding the input modality to richer pixel observations (e.g., depth sensors), enabling generalization to arbitrary, unseen, or unstructured tracks, and integrating racing with obstacle avoidance tasks. These efforts aim to extend the applicability of interpretable end-to-end vision-based world-model reinforcement learning to a broader set of agile autonomous flight settings (Verraest et al., 16 Oct 2025).