DroneARchery: Human-Drone Interaction through Augmented Reality with Haptic Feedback and Multi-UAV Collision Avoidance Driven by Deep Reinforcement Learning (2210.07730v1)

Abstract: We propose a novel concept of augmented reality (AR) human-drone interaction driven by RL-based swarm behavior to achieve intuitive and immersive control of a swarm formation of unmanned aerial vehicles. The DroneARchery system developed by us allows the user to quickly deploy a swarm of drones, generating flight paths simulating archery. The haptic interface LinkGlide delivers a tactile stimulus of the bowstring tension to the forearm to increase the precision of aiming. The swarm of released drones dynamically avoids collisions between each other, the drone following the user, and external obstacles with behavior control based on deep reinforcement learning. The developed concept was tested in the scenario with a human, where the user shoots from a virtual bow with a real drone to hit the target. The human operator observes the ballistic trajectory of the drone in an AR and achieves a realistic and highly recognizable experience of the bowstring tension through the haptic display. The experimental results revealed that the system improves trajectory prediction accuracy by 63.3% through applying AR technology and conveying haptic feedback of pulling force. DroneARchery users highlighted the naturalness (4.3 out of 5 point Likert scale) and increased confidence (4.7 out of 5) when controlling the drone. We have designed the tactile patterns to present four sliding distances (tension) and three applied force levels (stiffness) of the haptic display. Users demonstrated the ability to distinguish tactile patterns produced by the haptic display representing varying bowstring tension(average recognition rate is of 72.8%) and stiffness (average recognition rate is of 94.2%). The novelty of the research is the development of an AR-based approach for drone control that does not require special skills and training from the operator.

• The paper introduces DroneARchery, which fuses augmented reality, haptic feedback, and deep reinforcement learning to improve drone trajectory accuracy by 63%.
• It employs an Oculus Quest 2 with Passthrough AR and a wearable haptic device to simulate a virtual archery mechanism for precise drone control.
• The deep reinforcement learning–driven multi-UAV collision avoidance system enhances operational safety, indicating broad applicability in diverse fields.

An Analysis of DroneARchery: Human-Drone Interaction through Augmented Reality with Haptic Feedback

The paper "DroneARchery: Human-Drone Interaction through Augmented Reality with Haptic Feedback and Multi-UAV Collision Avoidance Driven by Deep Reinforcement Learning" presents an innovative approach in the field of human-drone interaction (HDI). This paper introduces a system that leverages augmented reality (AR) and haptic feedback to enable users to control a swarm of unmanned aerial vehicles (UAVs) in a promising, intuitive, and immersive manner. The system, aptly named DroneARchery, is designed to facilitate user engagement and enhance the precision of drone operations through a virtual archery metaphor, where drones are deployed to simulate the release of arrows.

System Overview and Methodology

DroneARchery integrates several key technologies to achieve its objectives. The system operation relies on the following components:

1. Augmented Reality Interface: Users wear an Oculus Quest 2 VR headset utilizing the Passthrough AR functionality, allowing them to perceive virtual elements and interactions overlaid on the real world. The AR environment projects the ballistic trajectory of drones, enhancing user awareness and control precision.
2. Haptic Feedback Device: A wearable tactile interface known as LinkGlide provides haptic feedback akin to the tension of a bowstring. This device, attached to the forearm, delivers linear position and force feedback, improving the user's ability to estimate and adjust the virtual bow tension accurately.
3. Multi-UAV Collision Avoidance: The system employs Deep Reinforcement Learning (DRL) to handle collision avoidance scenarios in a swarm of drones, especially when movement trajectories intersect with dynamic obstacles. This layer ensures real-time responsivity of the swarm to unpredictable environmental conditions.

Experimental Evaluation

The research delineates multiple experiments aimed at assessing the system’s efficiency. Notably, a user paper was conducted to evaluate haptic feedback and its impact on human users' ability to predict drone trajectories. Results indicated that the system improved trajectory prediction accuracy by 63.3%, with user feedback corroborating the enhanced naturalness (4.3/5) and confidence (4.7/5) during drone control.

Moreover, participants exhibited an average recognition rate of 72.8% and 94.2% for perceiving varying bowstring tension and stiffness, respectively, demonstrating the haptic display's potential utility in delivering effective tactile stimuli.

Implications and Future Prospects

The applications of DroneARchery extend beyond recreational uses to potential deployments in industries such as search and rescue, agricultural monitoring, and infrastructure maintenance, where swift and accurate UAV deployment is critical. The paper suggests that DroneARchery can streamline operations requiring multi-agent systems by offering intuitive control and interaction paradigms that reduce learning curves and the need for extensive user training.

Conclusion

This paper advances the field of HDI by integrating AR and tactile feedback to enhance UAV swarm interaction. The combination of these technologies with DRL-based swarm control posits a significant step forward in making advanced drone control accessible and applicable across various domains. While further investigation is warranted to optimize system scalability and integration with real-world applications, DroneARchery represents a substantial contribution to the ongoing discourse on intuitive human-drone symbiosis. Future research might explore deeper integration of machine learning techniques for even more nuanced control dynamics in unmanned aerial systems.