---
title: 'MonoRace: Autonomous Drone Racing System'
url: https://www.emergentmind.com/topics/monorace
type: topic
---

# MonoRace: Autonomous Drone Racing System

MonoRace is an onboard autonomous drone racing system that achieves championship-level performance using only a minimal sensor suite—specifically, a single monocular, rolling-shutter camera and an onboard IMU—eschewing external motion-capture infrastructure or stereo vision. In competition environments, MonoRace demonstrates the ability to generalize robustly, achieving speeds up to 100 km/h and outperforming world-champion pilots. The system innovates in real-time state estimation, segmentation-based perception, direct end-to-end motor control via neural networks, and self-supervised calibration, all while operating fully on lightweight embedded hardware [2601.15222].

## 1. System and Hardware Architecture

MonoRace is composed of a lightweight carbon-fiber quadrotor platform equipped solely with:
- A single forward-looking rolling-shutter CMOS camera (155° × 115° FOV, 820 × 616 @ 90 Hz), mounted with arbitrary orientation.
- An onboard IMU (accelerometers at 1 kHz, gyroscopes at 2 kHz) on the autopilot board.
- An NVIDIA Jetson Orin NX computer (for perception and GateNet inference).
- An STM32H743 (480 MHz) flight controller that executes the EKF and the Guidance-and-Control CNet at 500 Hz.
The perception and control data flows as follows:
1. Camera frames are undistorted and adaptively cropped to 384 × 384.
2. GateNet, a U-Net-based segmentation model, runs on the Orin.
3. The QuAdGate method produces edge and intersection corner fittings, yielding 2D–3D correspondences.
4. A PnP solver with multi-gate and de-rotated fallback modes estimates relative pose.
5. The EKF fuses visual and IMU data at 500 Hz, replacing saturated IMU values with model predictions as needed.
6. The Guidance-and-Control CNet directly computes motor commands based on the EKF state and gate geometry.

## 2. Perception: GateNet Segmentation

GateNet is a U-Net-style segmentation neural network optimized for robust gate detection under limited compute. Its architecture features:
- Input: 384 × 384 RGB (or grayscale) image crop.
- Encoder: inc-64/f → down1-128/f → down2-256/f → down3-512/f → down4-512/f (f = 4 channel reduction).
- Decoder with skip-connections: up4-512/f → up3-256/f → up2-128/f → up1-64/f → outc-1.
- Convolutions: 3×3 kernels, batch-norm, ReLU; upsampling via 2×2 transpose convolution.
- Multi-scale supervision: outputs at several resolutions, but real-time operation uses only the full-resolution output.

Supervision is via multi-scale Dice and binary cross-entropy, with the total loss
\[
\mathcal{L}_{\rm total} = 4\,\mathcal{L}_0 + 2\,\mathcal{L}_1 + \mathcal{L}_2 + \mathcal{L}_3 + \mathcal{L}_4,
\]
where 
\[
\mathcal{L}_i = \text{Dice}(y_i, \hat y_i) + 2\,\text{BCE}(y_i, \hat y_i).
\]
Train data: 3500 synthetic and 500 real images, with synthetic data generated through compositing varied, warped, and photometrically distorted gates on randomized backgrounds. On-the-fly augmentations include affine transforms, HSV jitter, Gaussian noise, lens distortion, and motion-blur kernels replicating rolling-shutter effects.

## 3. State Estimation: IMU, PnP, and EKF Integration

The state estimation pipeline fuses visual pose updates from PnP with continuous IMU data using an EKF:
- Camera projection (PnP-invariant, ignoring rolling shutter for pose):
  \[
  \mathbf{x} = \Pi(K\,[R|t]\,X)
  \]
  where $K$ is the intrinsics matrix and $\Pi([u,v,w]^T) = (u/w, v/w)$.
- IMU-driven dynamics:
  \[
  \dot{p} = v,\quad \dot{v} = R(q)a_{\rm meas} + g,\quad \dot{q} = \tfrac12\,q \otimes [0,p,q,r]^T
  \]
  with biases evolving as $\dot{b}_a = w_{ba}$, $\dot{b}_\omega = w_{b\omega}$.
- When $||a_{\rm meas}-a_{\rm model}|| > 22\,\mathrm{m/s}^2$, the system substitutes $a_{\rm model}$ (from the quadcopter model) in state updates.

Discrete EKF prediction runs at 1 kHz:
\[
x_{k+1} = x_k + f_c(x_k, u_k, 0)\Delta t;\quad 
P_{k+1} = F_k P_k F_k^T + L_k Q_k L_k^T
\]
EKF updates are triggered by PnP outputs:
\[
K_k = P_k H_k^T (H_k P_k H_k^T + R_k)^{-1};
\quad x_k \leftarrow x_k + K_k(z_k - h(x_k));
\quad P_k \leftarrow (I-K_k H_k)P_k
\]
where $h(x)$ extracts position and quaternion. To reject outliers, only PnP updates where
\[
||p_{\rm KF}-p_{\rm PnP}||^2 < 16\,N_c^2\,\mathrm{trace}(P_{\rm pos})
\]
($N_c$ = number of detected corners) are accepted.

The EKF employs multi-rate fusion with camera–IMU latency compensation by fusing measurements at image time and re-propagating to the flight-controller clock.

## 4. Offline Camera-IMU Extrinsics Optimization

MonoRace dispenses with external ground-truth for camera–IMU extrinsics, instead leveraging the known gate geometry:
- For each frame, the method computes a reprojected gate mask $M_i(\Theta)$ from EKF state and candidate extrinsics $\Theta$.
- The cost is the negative mean intersection-over-union (IoU) with the GateNet segmentation $S_i$ over $N$ frames:
\[
J(\Theta) = -\frac{1}{N}\sum_{i=1}^N \mathrm{IoU}(S_i, M_i(\Theta))
\]
- Bayesian optimization over only 40 evaluations per log achieves sub-degree recovery (0.1°–0.7° error).
This self-supervised procedure is validated in both simulation-in-the-loop (with ground-truth extrinsics) and on real data, providing accuracy on par with marker-based calibration methods.

## 5. End-to-End Guidance and Control

The Guidance-and-Control CNet eschews explicit inner control loops, generating direct motor commands in real-time:
- Inputs (≈24 dimensions): current relative position/velocity to gate, Euler angles and rates, next-gate relative yaw and position.
- Architecture: 3 hidden ReLU layers of 64 units; output: four scalars $u_1,\dots,u_4\in[-1,1]$ for direct motor actuation.
- Inference runs at 500 Hz on the STM32, with ≈2 ms total latency.

Policy is trained in simulation (using the quadcopter model) with domain randomization ±30–50 % on thrust/drag coefficients, inertia, time constants, and motor bounds. The PPO-based reward function incorporates progress, minimal rate usage, gate offset penalties, perception quality, actuation smoothness, and crash avoidance:
\[
r_k = r_{\rm prog} - p_{\rm rate} - p_{\rm offset} - p_{\rm perc} - p_{\Delta u} - p_u - p_{\rm crash}
\]
This regimen yields policies that are robust to substantial simulation–reality discrepancies, as verified by close agreement between simulated and measured lap times.

## 6. Quantitative Results, Performance, and Robustness

MonoRace set a new benchmark at the 2025 Abu Dhabi Autonomous Drone Racing Competition (A2RL):
- Fastest single-drone Grand Challenge time: 16.56 s (two laps, 11 gates), faster than three FPV world-champions.
- Peak real-world speed: 100 km/h (28.23 m/s), compared to prior stereo-VIO systems (22 m/s).
- Won three direct knockout races against human champions.

Ablation analyses indicate:
- Simulated rollouts under 30% domain randomization yield ≈90% success rates, which closely match real-world success rates.
- IMU saturation handling (model-based acceleration replacement) increased EKF success rate from 50% to 100% in key scenarios.
- The EKF with KF outlier rejection and fallback to gate mask remained robust with up to 50% camera frame loss due to EMI; even with 75% corruption, at least one lap was completed before crash.
- Multi-gate PnP fusion reduced position RMSE from >0.5 m (beyond 5 m) to <0.2 m and heading errors from >5° to <2°.

The measured reality gap is low: real lap times match simulation means within ≈0.5 s, and the policy remained stable for forces/moments outside the randomization envelope. This suggests the MonoRace policy generalizes robustly beyond the simulated training set.

## 7. Significance and Implications

MonoRace demonstrates that a resource-minimal, fully onboard perception and control stack—devoid of external localization infrastructure and relying only on a monocular camera and IMU—can achieve and surpass human-champion performance in high-speed, real-world autonomous drone racing. The architecture integrates robust neural segmentation, geometric-consistent state estimation, self-supervised calibration, and end-to-end direct motor control. This validation provides a foundation for future research aiming at deploying autonomous aerial systems in complex, dynamic tasks without reliance on heavy or external sensing modalities [2601.15222].

Source: https://www.emergentmind.com/topics/monorace