Deep Drone Racing: From Simulation to Reality with Domain Randomization (1905.09727v2)

Abstract: Dynamically changing environments, unreliable state estimation, and operation under severe resource constraints are fundamental challenges that limit the deployment of small autonomous drones. We address these challenges in the context of autonomous, vision-based drone racing in dynamic environments. A racing drone must traverse a track with possibly moving gates at high speed. We enable this functionality by combining the performance of a state-of-the-art planning and control system with the perceptual awareness of a convolutional neural network (CNN). The resulting modular system is both platform- and domain-independent: it is trained in simulation and deployed on a physical quadrotor without any fine-tuning. The abundance of simulated data, generated via domain randomization, makes our system robust to changes of illumination and gate appearance. To the best of our knowledge, our approach is the first to demonstrate zero-shot sim-to-real transfer on the task of agile drone flight. We extensively test the precision and robustness of our system, both in simulation and on a physical platform, and show significant improvements over the state of the art.

• The paper introduces a modular system coupling CNN-based perception for waypoint prediction with a dedicated control module for agile drone racing.
• It employs domain randomization to achieve seamless sim-to-real transfer, enabling speeds up to 9 m/s and robust performance under partial occlusions.
• Extensive evaluations in simulation and real-world tests show superior performance over end-to-end learning and traditional VIO tracking methods.

Deep Drone Racing: From Simulation to Reality with Domain Randomization

This paper presents a modular system designed to address the fundamental challenges of deploying small autonomous drones in dynamic environments for vision-based drone racing. The system is noteworthy for its combination of a state-of-the-art planning and control system with the perceptual capabilities of a convolutional neural network (CNN). This amalgamation enables the autonomous drone to navigate complex, dynamic tracks at high speeds, successfully transferring its training from a simulation environment to a real-world setting through domain randomization techniques.

Overview

The crux of the proposed solution lies in its modular architecture, which separates perception from control. The perception system is a CNN that predicts the next waypoint and desired speed from the visual input. This decoupling allows the network to focus solely on processing visual information, bypassing the requirement to stabilize the inherently unstable quadrotor platform—a task managed by the dedicated control system. Consequently, the complex task of drone racing is effectively broken down into image mapping and high-speed trajectory planning, with each module optimized independently, enhancing overall system robustness and agility.

Evaluation

The authors conduct extensive evaluations in both simulation and real-world environments, substantiating the performance of the system. In simulation, the system consistently outperforms end-to-end learning baselines while demonstrating an ability to navigate tracks with dynamically moving gates—a feat unmanageable for traditional VIO-based tracking due to state estimation drift. They further exploit domain randomization during training in simulation to enhance generalization to a variety of environmental changes, such as lighting or gate appearance, significantly improving its robustness.

As demonstrated in real-world experiments, the proposed model exhibits the capability to transfer learned policies from simulation into practice seamlessly. The researchers showcase that with domain randomization, the system performs comparably to models trained with real-world data under known conditions and can even surpass these models in previously unseen environmental settings, such as changes in illumination.

Numerical Results

The paper highlights how the system can achieve speeds as high as 9 m/s on complex tracks and maintain its robustness against occlusions of up to 50% of the target gate in cluttered environments. Even with zero-shot sim-to-real transfer, the system maintains a high task completion rate of 75% under difficult lighting conditions, where directly trained models falter.

Implications and Future Directions

The success of domain randomization in facilitating sim-to-real transfer has broader implications for autonomous systems, demonstrating the potential of simulation training to generalize real-world applications robustly. As modular systems such as this evolve, we anticipate advancements in adaptive navigation strategies and the incorporation of reinforcement learning methodologies to further enhance agility and navigation precision in dynamic and uncertain environments.

The paper ultimately pushes forward the boundaries of autonomous drone racing, offering insights into tackling similar challenges across other domains involving real-time perception and control, including rescue missions, exploration, and industrial inspections. The approach outlined here exemplifies an adept balance between leveraging learned models for perceptual inputs and conventional controllers for precision and stability, a potent paradigm in the advancement of intelligent autonomous systems.