MonoRace: Autonomous Drone Racing System

• MonoRace is a resource-minimal autonomous drone racing system that uses a single monocular, rolling-shutter camera and IMU for real-time state estimation and direct control.
• It employs a U-Net based GateNet for segmentation, robust EKF integration with PnP and IMU data, and self-supervised calibration to maintain precision without external sensors.
• Validated in high-speed competitions, MonoRace achieves speeds up to 100 km/h, outperforming human-champion pilots and setting new benchmarks in autonomous drone racing.

MonoRace is an onboard autonomous drone racing system that achieves championship-level performance using only a minimal sensor suite—specifically, a single monocular, rolling-shutter camera and an onboard IMU—eschewing external motion-capture infrastructure or stereo vision. In competition environments, MonoRace demonstrates the ability to generalize robustly, achieving speeds up to 100 km/h and outperforming world-champion pilots. The system innovates in real-time state estimation, segmentation-based perception, direct end-to-end motor control via neural networks, and self-supervised calibration, all while operating fully on lightweight embedded hardware (Bahnam et al., 21 Jan 2026).

1. System and Hardware Architecture

MonoRace is composed of a lightweight carbon-fiber quadrotor platform equipped solely with:

• A single forward-looking rolling-shutter CMOS camera (155° × 115° FOV, 820 × 616 @ 90 Hz), mounted with arbitrary orientation.
• An onboard IMU (accelerometers at 1 kHz, gyroscopes at 2 kHz) on the autopilot board.
• An NVIDIA Jetson Orin NX computer (for perception and GateNet inference).
• An STM32H743 (480 MHz) flight controller that executes the EKF and the Guidance-and-Control CNet at 500 Hz. The perception and control data flows as follows:

1. Camera frames are undistorted and adaptively cropped to 384 × 384.
2. GateNet, a U-Net-based segmentation model, runs on the Orin.
3. The QuAdGate method produces edge and intersection corner fittings, yielding 2D–3D correspondences.
4. A PnP solver with multi-gate and de-rotated fallback modes estimates relative pose.
5. The EKF fuses visual and IMU data at 500 Hz, replacing saturated IMU values with model predictions as needed.
6. The Guidance-and-Control CNet directly computes motor commands based on the EKF state and gate geometry.

2. Perception: GateNet Segmentation

GateNet is a U-Net-style segmentation neural network optimized for robust gate detection under limited compute. Its architecture features:

• Input: 384 × 384 RGB (or grayscale) image crop.
• Encoder: inc-64/f → down1-128/f → down2-256/f → down3-512/f → down4-512/f (f = 4 channel reduction).
• Decoder with skip-connections: up4-512/f → up3-256/f → up2-128/f → up1-64/f → outc-1.
• Convolutions: 3×3 kernels, batch-norm, ReLU; upsampling via 2×2 transpose convolution.
• Multi-scale supervision: outputs at several resolutions, but real-time operation uses only the full-resolution output.

Supervision is via multi-scale Dice and binary cross-entropy, with the total loss

$$\mathcal{L}_{\rm total} = 4\,\mathcal{L}_0 + 2\,\mathcal{L}_1 + \mathcal{L}_2 + \mathcal{L}_3 + \mathcal{L}_4,$$

where

$$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$

Train data: 3500 synthetic and 500 real images, with synthetic data generated through compositing varied, warped, and photometrically distorted gates on randomized backgrounds. On-the-fly augmentations include affine transforms, HSV jitter, Gaussian noise, lens distortion, and motion-blur kernels replicating rolling-shutter effects.

3. State Estimation: IMU, PnP, and EKF Integration

The state estimation pipeline fuses visual pose updates from PnP with continuous IMU data using an EKF:

• Camera projection (PnP-invariant, ignoring rolling shutter for pose):

$\mathbf{x} = \Pi(K\,[R|t]\,X)$

where $K$ is the intrinsics matrix and $\Pi([u,v,w]^T) = (u/w, v/w)$.

• IMU-driven dynamics:

$$\dot{p} = v,\quad \dot{v} = R(q)a_{\rm meas} + g,\quad \dot{q} = \tfrac12\,q \otimes [0,p,q,r]^T$$

with biases evolving as $\dot{b}_a = w_{ba}$, $\dot{b}_\omega = w_{b\omega}$.

• When $$||a_{\rm meas}-a_{\rm model}|| > 22\,\mathrm{m/s}^2$$, the system substitutes $a_{\rm model}$ (from the quadcopter model) in state updates.

Discrete EKF prediction runs at 1 kHz: $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$0 EKF updates are triggered by PnP outputs: $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$1 where $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$2 extracts position and quaternion. To reject outliers, only PnP updates where

$$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$3

($$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$4 = number of detected corners) are accepted.

The EKF employs multi-rate fusion with camera–IMU latency compensation by fusing measurements at image time and re-propagating to the flight-controller clock.

4. Offline Camera-IMU Extrinsics Optimization

MonoRace dispenses with external ground-truth for camera–IMU extrinsics, instead leveraging the known gate geometry:

• For each frame, the method computes a reprojected gate mask $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$5 from EKF state and candidate extrinsics $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$6.
• The cost is the negative mean intersection-over-union (IoU) with the GateNet segmentation $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$7 over $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$8 frames: $$\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).$$9
• Bayesian optimization over only 40 evaluations per log achieves sub-degree recovery (0.1°–0.7° error). This self-supervised procedure is validated in both simulation-in-the-loop (with ground-truth extrinsics) and on real data, providing accuracy on par with marker-based calibration methods.

5. End-to-End Guidance and Control

The Guidance-and-Control CNet eschews explicit inner control loops, generating direct motor commands in real-time:

• Inputs (≈24 dimensions): current relative position/velocity to gate, Euler angles and rates, next-gate relative yaw and position.
• Architecture: 3 hidden ReLU layers of 64 units; output: four scalars $\mathbf{x} = \Pi(K\,[R|t]\,X)$0 for direct motor actuation.
• Inference runs at 500 Hz on the STM32, with ≈2 ms total latency.

Policy is trained in simulation (using the quadcopter model) with domain randomization ±30–50 % on thrust/drag coefficients, inertia, time constants, and motor bounds. The PPO-based reward function incorporates progress, minimal rate usage, gate offset penalties, perception quality, actuation smoothness, and crash avoidance: $\mathbf{x} = \Pi(K\,[R|t]\,X)$1 This regimen yields policies that are robust to substantial simulation–reality discrepancies, as verified by close agreement between simulated and measured lap times.

6. Quantitative Results, Performance, and Robustness

MonoRace set a new benchmark at the 2025 Abu Dhabi Autonomous Drone Racing Competition (A2RL):

• Fastest single-drone Grand Challenge time: 16.56 s (two laps, 11 gates), faster than three FPV world-champions.
• Peak real-world speed: 100 km/h (28.23 m/s), compared to prior stereo-VIO systems (22 m/s).
• Won three direct knockout races against human champions.

Ablation analyses indicate:

• Simulated rollouts under 30% domain randomization yield ≈90% success rates, which closely match real-world success rates.
• IMU saturation handling (model-based acceleration replacement) increased EKF success rate from 50% to 100% in key scenarios.
• The EKF with KF outlier rejection and fallback to gate mask remained robust with up to 50% camera frame loss due to EMI; even with 75% corruption, at least one lap was completed before crash.
• Multi-gate PnP fusion reduced position RMSE from >0.5 m (beyond 5 m) to <0.2 m and heading errors from >5° to <2°.

The measured reality gap is low: real lap times match simulation means within ≈0.5 s, and the policy remained stable for forces/moments outside the randomization envelope. This suggests the MonoRace policy generalizes robustly beyond the simulated training set.

7. Significance and Implications

MonoRace demonstrates that a resource-minimal, fully onboard perception and control stack—devoid of external localization infrastructure and relying only on a monocular camera and IMU—can achieve and surpass human-champion performance in high-speed, real-world autonomous drone racing. The architecture integrates robust neural segmentation, geometric-consistent state estimation, self-supervised calibration, and end-to-end direct motor control. This validation provides a foundation for future research aiming at deploying autonomous aerial systems in complex, dynamic tasks without reliance on heavy or external sensing modalities (Bahnam et al., 21 Jan 2026).