Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 178 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 38 tok/s Pro
GPT-5 High 40 tok/s Pro
GPT-4o 56 tok/s Pro
Kimi K2 191 tok/s Pro
GPT OSS 120B 445 tok/s Pro
Claude Sonnet 4.5 36 tok/s Pro
2000 character limit reached

HGIC: A Hand Gesture Based Interactive Control System for Efficient and Scalable Multi-UAV Operations (2403.05478v1)

Published 8 Mar 2024 in cs.RO

Abstract: As technological advancements continue to expand the capabilities of multi unmanned-aerial-vehicle systems (mUAV), human operators face challenges in scalability and efficiency due to the complex cognitive load and operations associated with motion adjustments and team coordination. Such cognitive demands limit the feasible size of mUAV teams and necessitate extensive operator training, impeding broader adoption. This paper developed a Hand Gesture Based Interactive Control (HGIC), a novel interface system that utilize computer vision techniques to intuitively translate hand gestures into modular commands for robot teaming. Through learning control models, these commands enable efficient and scalable mUAV motion control and adjustments. HGIC eliminates the need for specialized hardware and offers two key benefits: 1) Minimal training requirements through natural gestures; and 2) Enhanced scalability and efficiency via adaptable commands. By reducing the cognitive burden on operators, HGIC opens the door for more effective large-scale mUAV applications in complex, dynamic, and uncertain scenarios. HGIC will be open-sourced after the paper being published online for the research community, aiming to drive forward innovations in human-mUAV interactions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Y. Zhou, B. Rao, and W. Wang, “Uav swarm intelligence: Recent advances and future trends,” IEEE Access, vol. 8, pp. 183856–183878, 2020.
  2. E. T. Alotaibi, S. S. Alqefari, and A. Koubaa, “Lsar: Multi-uav collaboration for search and rescue missions,” IEEE Access, vol. 7, pp. 55817–55832, 2019.
  3. M. Aljehani and M. Inoue, “Performance evaluation of multi-uav system in post-disaster application: Validated by hitl simulator,” IEEE Access, vol. 7, pp. 64386–64400, 2019.
  4. Springer International Publishing, 2020.
  5. M. L. Cummings and P. J. Mitchell, “Predicting controller capacity in supervisory control of multiple uavs,” IEEE Trans. Syst. Man. Cybern. - Part A: Systems and Humans, vol. 38, no. 2, pp. 451–460, 2008.
  6. A. Paas, E. B. J. Coffey, G. Beltrame, and D. St-Onge, “Towards evaluating the impact of swarm robotic control strategy on operators’ cognitive load,” in IEEE RO-MAN, pp. 217–223, 2022.
  7. A. Kolling, P. Walker, N. Chakraborty, K. Sycara, and M. Lewis, “Human interaction with robot swarms: A survey,” IEEE Transactions on Human-Machine Systems, vol. 46, no. 1, pp. 9–26, 2016.
  8. J. R. Cauchard, J. L. E, K. Y. Zhai, and J. A. Landay, “Drone & me: An exploration into natural human-drone interaction,” in Proceedings of the 2015 ACM UbiComp, UbiComp ’15, p. 361–365, 2015.
  9. F. Gao, M. L. Cummings, and L. F. Bertuccelli, “Teamwork in controlling multiple robots,” in Proceedings of the seventh annual ACM/IEEE Int. Conf. Hum.-Robot Interact., pp. 81–88, 2012.
  10. C. Huang, W. Luo, and R. Liu, “Meta preference learning for fast user adaptation in human-supervisory multi-robot deployments,” in IEEE/RSJ IROS, pp. 5851–5856, IEEE, 2021.
  11. J. Moon, S. Papaioannou, C. Laoudias, P. Kolios, and S. Kim, “Deep reinforcement learning multi-uav trajectory control for target tracking,” IEEE Internet Things J., vol. 8, no. 20, pp. 15441–15455, 2021.
  12. J. Nagi, A. Giusti, L. M. Gambardella, and G. A. Di Caro, “Human-swarm interaction using spatial gestures,” in IEEE/RSJ IROS, pp. 3834–3841, 2014.
  13. J. Alonso-Mora, S. Haegeli Lohaus, P. Leemann, R. Siegwart, and P. Beardsley, “Gesture based human - multi-robot swarm interaction and its application to an interactive display,” in 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 5948–5953, 2015.
  14. M. Macchini, L. De Matteïs, F. Schiano, and D. Floreano, “Personalized human-swarm interaction through hand motion,” IEEE RAL, vol. 6, no. 4, pp. 8341–8348, 2021.
  15. K. Pfeil, S. L. Koh, and J. LaViola, “Exploring 3d gesture metaphors for interaction with unmanned aerial vehicles,” in Proceedings of the 2013 ACM IUI, pp. 257–266, 2013.
  16. E. Tsykunov, R. Agishev, R. Ibrahimov, L. Labazanova, A. Tleugazy, and D. Tsetserukou, “Swarmtouch: Guiding a swarm of micro-quadrotors with impedance control using a wearable tactile interface,” IEEE Transactions on Haptics, vol. 12, no. 3, pp. 363–374, 2019.
  17. H. I. Son, A. Franchi, L. L. Chuang, J. Kim, H. H. Bulthoff, and P. Robuffo Giordano, “Human-centered design and evaluation of haptic cueing for teleoperation of multiple mobile robots,” IEEE Transactions on Cybernetics, vol. 43, no. 2, pp. 597–609, 2013.
  18. S. K. Cho, H. Z. Jin, J. M. Lee, and B. Yao, “Teleoperation of a mobile robot using a force-reflection joystick with sensing mechanism of rotating magnetic field,” IEEE/ASME Trans. Mechatron., vol. 15, no. 1, pp. 17–26, 2010.
  19. M. Quigley, M. Goodrich, and R. Beard, “Semi-autonomous human-uav interfaces for fixed-wing mini-uavs,” in IEEE/RSJ IROS, vol. 3, pp. 2457–2462 vol.3, 2004.
  20. S. Pourmehr, V. M. Monajjemi, R. Vaughan, and G. Mori, ““you two! take off!”: Creating, modifying and commanding groups of robots using face engagement and indirect speech in voice commands,” in IEEE/RSJ IROS, pp. 137–142, 2013.
  21. M. Landau and S. Van Delden, “A system architecture for hands-free uav drone control using intuitive voice commands,” in Proceedings of the companion of the 2017 ACM/IEEE Int. Conf. Hum.-Robot Interact., pp. 181–182, 2017.
  22. C. Pfeiffer and D. Scaramuzza, “Human-piloted drone racing: Visual processing and control,” IEEE RAL, vol. 6, no. 2, pp. 3467–3474, 2021.
  23. M. Yu, Y. Lin, D. Schmidt, X. Wang, and Y. Wang, “Human-robot interaction based on gaze gestures for the drone teleoperation,” Journal of Eye Movement Research, vol. 7, no. 4, pp. 1–14, 2014.
  24. S. Chen, M. J. O’Brien, F. Talbot, J. Williams, B. Tidd, A. Pitt, and R. C. Arkin, “Multi-modal user interface for multi-robot control in underground environments,” in IEEE/RSJ IROS, pp. 9995–10002, 2022.
  25. I. Maza, F. Caballero, R. Molina, N. Peña, and A. Ollero, “Multimodal interface technologies for uav ground control stations: a comparative analysis,” in Selected papers from the 2nd International Symposium on UAVs, Reno, Nevada, USA June 8–10, 2009, pp. 371–391, 2010.
  26. K. Matsumoto, E. Langbehn, T. Narumi, and F. Steinicke, “Detection thresholds for vertical gains in vr and drone-based telepresence systems,” in 2020 IEEE VR, pp. 101–107, 2020.
  27. L. Chen, K. Takashima, K. Fujita, and Y. Kitamura, “Pinpointfly: An egocentric position-control drone interface using mobile ar,” in Conf. Hum. Factors Comput. Syst. - Proc., pp. 1–13, 2021.
  28. C. W. Reynolds, “Flocks, herds and schools: A distributed behavioral model,” in Annual conference on computer graphics and interactive techniques, pp. 25–34, 1987.
  29. T. Balch and R. Arkin, “Behavior-based formation control for multirobot teams,” IEEE Transactions on Robotics and Automation, vol. 14, no. 6, pp. 926–939, 1998.
  30. G. Vásárhelyi, C. Virágh, G. Somorjai, T. Nepusz, A. E. Eiben, and T. Vicsek, “Optimized flocking of autonomous drones in confined environments,” Science Robotics, vol. 3, no. 20, p. eaat3536, 2018.
  31. V. Grabe, M. Riedel, H. H. Bülthoff, P. R. Giordano, and A. Franchi, “The telekyb framework for a modular and extendible ros-based quadrotor control,” in 2013 ECMR, pp. 19–25, 2013.
  32. Z. Liu, X. Wang, L. Shen, S. Zhao, Y. Cong, J. Li, D. Yin, S. Jia, and X. Xiang, “Mission-oriented miniature fixed-wing uav swarms: A multilayered and distributed architecture,” IEEE Trans. Syst. Man Cybern.: Syst., vol. 52, no. 3, pp. 1588–1602, 2022.
  33. F. Schilling, J. Lecoeur, F. Schiano, and D. Floreano, “Learning vision-based cohesive flight in drone swarms,” arXiv preprint arXiv:1809.00543, 2018.
  34. S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in Field and Service Robotics, 2017.
  35. C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee, et al., “Mediapipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019.
  36. A. Vakunov, “Mediapipe hands: On-device real-time hand tracking,” 2020.
  37. G. Sung, K. Sokal, E. Uboweja, V. Bazarevsky, J. Baccash, E. G. Bazavan, C.-L. Chang, and M. Grundmann, “On-device real-time hand gesture recognition,” arXiv preprint arXiv:2111.00038, 2021.
  38. Q. De Smedt, H. Wannous, and J.-P. Vandeborre, “Skeleton-based dynamic hand gesture recognition,” in Proceedings of the IEEE CVPR Workshops, June 2016.
  39. G. Devineau, “Deep learning for hand gesture recognition on skeletal data,” in 13th IEEE FG, pp. 106–113, 2018.
  40. J. Cortes, S. Martinez, T. Karatas, and F. Bullo, “Coverage control for mobile sensing networks,” IEEE Transactions on Robotics and Automation, vol. 20, no. 2, pp. 243–255, 2004.
  41. “User Study Details.” Google Forms, 2024. https://docs.google.com/forms/d/18___Vcp74rXcCjstG_RhDNhUHbuFZbfOF37_K1JApl4/prefill.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up