Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs (2103.10873v1)

Abstract: Artificial intelligence-powered pocket-sized air robots have the potential to revolutionize the Internet-of-Things ecosystem, acting as autonomous, unobtrusive, and ubiquitous smart sensors. With a few cm$^{2}$ form-factor, nano-sized unmanned aerial vehicles (UAVs) are the natural befit for indoor human-drone interaction missions, as the pose estimation task we address in this work. However, this scenario is challenged by the nano-UAVs' limited payload and computational power that severely relegates the onboard brain to the sub-100 mW microcontroller unit-class. Our work stands at the intersection of the novel parallel ultra-low-power (PULP) architectural paradigm and our general development methodology for deep neural network (DNN) visual pipelines, i.e., covering from perception to control. Addressing the DNN model design, from training and dataset augmentation to 8-bit quantization and deployment, we demonstrate how a PULP-based processor, aboard a nano-UAV, is sufficient for the real-time execution (up to 135 frame/s) of our novel DNN, called PULP-Frontnet. We showcase how, scaling our model's memory and computational requirement, we can significantly improve the onboard inference (top energy efficiency of 0.43 mJ/frame) with no compromise in the quality-of-result vs. a resource-unconstrained baseline (i.e., full-precision DNN). Field experiments demonstrate a closed-loop top-notch autonomous navigation capability, with a heavily resource-constrained 27-gram Crazyflie 2.1 nano-quadrotor. Compared against the control performance achieved using an ideal sensing setup, onboard relative pose inference yields excellent drone behavior in terms of median absolute errors, such as positional (onboard: 41 cm, ideal: 26 cm) and angular (onboard: 3.7$^{\circ}$, ideal: 4.1$^{\circ}$).

• The paper introduces PULP-Frontnet, a CNN designed for efficient onboard human-drone pose estimation on nano-UAVs, achieving inference speeds up to 135 frames per second.
• The paper employs an innovative data collection method with synchronized motion capture and extensive augmentation to enhance network robustness under varied drone dynamics.
• The paper integrates an 8-bit quantization strategy and decentralized control framework, enabling energy-efficient operation at 0.43 mJ/frame and a peak power of 86.6 mW.

Overview of "Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs"

The research paper presents a novel approach to autonomous Human-Drone Interaction (HDI) using ultra-low power nano-sized Unmanned Aerial Vehicles (UAVs). The authors focus on deploying a Convolutional Neural Network (CNN) named PULP-Frontnet aboard the nanodrone Crazyflie 2.1, powered by a Parallel Ultra-Low Power (PULP) architecture-based GAP8 System-on-Chip (SoC). This paper highlights a fully autonomous framework that encompasses vision-based pose estimation and real-time control, enabled by onboard processing capabilities within stringent power and computational constraints.

Technical Contributions

1. PULP-Frontnet Architecture: The paper introduces PULP-Frontnet, a specialized CNN designed for the inference of a human subject's pose relative to the drone. Three variants of the network are presented, each tailored for different computational and memory efficiency trade-offs, achieving performance within the hardware capabilities of the GAP8 SoC.
2. Data Collection and Augmentation: The research includes an innovative dataset collection process facilitated by a synchronized motion capture system. The dataset not only provides diverse human poses but also undergoes extensive augmentation to simulate in-field scenarios, such as varied drone pitch during missions, enhancing the network's robustness.
3. Quantization and Deployment: The authors employ an 8-bit quantization strategy to optimize the network for deployment on the GAP8 SoC, utilizing tools like NEMO for training and DORY for efficient deployment. This approach ensures that the network can operate within the resource constraints of nano-UAVs without significant loss in prediction accuracy.
4. Onboard Control Framework: A multi-level control strategy is implemented, with the GAP8 performing real-time inference and a host microcontroller unit managing the drone's kinematics. This decentralized structure effectively handles the drone's navigation and interactions with human subjects in dynamic environments.

Numerical Results

The paper provides comprehensive performance metrics for the various PULP-Frontnet configurations, including energy efficiency of as low as \SI{0.43}{\milli\joule/frame}, onboard inference throughput of up to \SI{135}{frame/\second}, and a peak power consumption of \SI{86.6}{\milli\watt}. These results reflect the feasibility of deploying advanced AI applications on power-constrained platforms.

Implications and Future Directions

The demonstrated capabilities of PULP-Frontnet on nano-UAVs have significant implications for ubiquitous IoT applications, making possible new use cases in smart environments where mobility and unobtrusiveness are critical. This work showcases how integrating AI with edge-computing paradigms can facilitate autonomous operations in resource-constrained settings. Future research might explore expanding the sensory capabilities of such platforms and enhancing multi-UAV cooperative tasks, leveraging the potential of on-device resilience and improved network architectures.

In conclusion, the paper presents a compelling contribution to the field of small-scale robotics and AI, demonstrating an effective solution to the problem of autonomous navigation and interaction in nano-UAV platforms with limited resources.