Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Cooperative Flight Control Using Visual-Attention (2212.11084v2)

Published 21 Dec 2022 in cs.RO and cs.AI

Abstract: The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. M. A. J. T. Deveans, L. T. C. R. H. Kewley, and P. D, “Overcoming Information Overload in the Cockpit Sponsored by Program Executive Office - Soldier,” Operations Research, no. July, 2009.
  2. J. Lee and N. Moray, “Trust, control strategies and allocation of function in human-machine systems,” Ergonomics, vol. 35, no. 10, pp. 1243–1270, 1992.
  3. D. Sofge, W. Lawless, and R. Mittu, “Ai bookie: Will a self-authorizing ai-based system take control from a human operator?” AI Magazine, vol. 40, no. 3, pp. 79–84, 2019.
  4. H. Chao, Y. Cao, and Y. Q. Chen, “Autopilots for small fixed-wing unmanned air vehicles: A survey,” Proceedings of the 2007 IEEE International Conference on Mechatronics and Automation, ICMA 2007, pp. 3144–3149, 2007.
  5. M. Dikmen and C. Burns, “Trust in autonomous vehicles: The case of tesla autopilot and summon,” in 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017, pp. 1093–1098.
  6. J. Jackson, D. Koch, T. Henrichsen, and T. McLain, “ROSflight: A lean open-source research autopilot,” IEEE International Conference on Intelligent Robots and Systems, pp. 1173–1179, 2020.
  7. W. Schwarting, J. Alonso-Mora, and D. Rus, “Planning and Decision-Making for Autonomous Vehicles,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, pp. 187–210, 2018.
  8. M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 2669–2678, 2018.
  9. C. Vorbach, R. Hasani, A. Amini, M. Lechner, and D. Rus, “Causal navigation by continuous-time neural networks,” Advances in Neural Information Processing Systems, vol. 34, pp. 12 425–12 440, 2021.
  10. H. Zhu, R. Xu, and M. L. Cummings, “Quantitative operator strategy comparisons across human supervisory control scenarios,” IEEE International Conference on Intelligent Robots and Systems, pp. 10 968–10 974, 2020.
  11. W. Schwarting, J. Alonso-Mora, L. Pauli, S. Karaman, and D. Rus, “Parallel autonomy in automated vehicles: Safe motion generation with minimal intervention,” Proceedings - IEEE International Conference on Robotics and Automation, pp. 1928–1935, 2017.
  12. S. Nikolaidis, Y. X. Zhu, D. Hsu, and S. Srinivasa, “Human-Robot Mutual Adaptation in Shared Autonomy,” ACM/IEEE International Conference on Human-Robot Interaction, vol. Part F127194, pp. 294–302, 2017.
  13. S. Javdani, S. S. Srinivasa, and J. A. Bagnell, “Shared autonomy via hindsight optimization,” Robotics: Science and Systems, vol. 11, 2015.
  14. D. Gopinath, S. Jain, and B. D. Argall, “Human-in-the-loop optimization of shared autonomy in assistive robotics,” IEEE Robotics and Automation Letters, vol. 2, no. 1, pp. 247–254, 2017.
  15. M. Carvalho, J. Avelino, A. Bernardino, R. Ventura, and P. Moreno, “Human-Robot greeting: tracking human greeting mental states and acting accordingly,” 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), no. i, pp. 1935–1941, 2021.
  16. S. J. Anderson, S. B. Karumanchi, K. Iagnemma, and J. M. Walker, “The intelligent copilot: A constraint-based approach to shared-adaptive control of ground vehicles,” IEEE Intelligent Transportation Systems Magazine, vol. 5, no. 2, pp. 45–54, 2013.
  17. A. W. Knapp, “AirGuardian : A Parallel Autonomy Approach to Self-Flying Planes by,” Master’s thesis, Massachusetts Institute of Technology, 2020.
  18. J. Kim and J. Canny, “Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention,” Proceedings of the IEEE International Conference on Computer Vision, vol. 2017-Octob, pp. 2961–2969, 2017.
  19. N. Wessendorp, R. Dinaux, J. Dupeyroux, and G. C. H. E. de Croon, “Obstacle Avoidance onboard MAVs using a FMCW Radar,” 2017 IEEE International Conference on Computer Vision (ICCV), pp. 117–122, 2021.
  20. S. Xu, D. Honegger, M. Pollefeys, and L. Heng, “Real-time 3D navigation for autonomous vision-guided MAVs,” IEEE International Conference on Intelligent Robots and Systems, vol. 2015-December, pp. 53–59, 2015.
  21. Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
  22. K. Xu, J. L. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” 32nd International Conference on Machine Learning, ICML 2015, vol. 3, pp. 2048–2057, 2015.
  23. M. Bojarski, A. Choromanska, K. Choromanski, B. Firner, L. Jackel, U. Muller, and K. Zieba, “VisualBackProp: efficient visualization of CNNs,” 2016. [Online]. Available: http://arxiv.org/abs/1611.05418
  24. S. Yang, W. Wang, C. Liu, and W. Deng, “Scene understanding in deep learning-based end-to-end controllers for autonomous vehicles,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 1, pp. 53–63, 2019.
  25. R. M. Aronson, T. Santini, T. C. Kübler, E. Kasneci, S. Srinivasa, and H. Admoni, “Eye-Hand Behavior in Human-Robot Shared Manipulation,” ACM/IEEE International Conference on Human-Robot Interaction, pp. 4–13, 2018.
  26. B. Schölkopf, “Causality for machine learning,” arXiv preprint arXiv:1911.10500, 2019.
  27. R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu, “Liquid Time-constant Networks,” 2020. [Online]. Available: http://arxiv.org/abs/2006.04439
  28. M. Lechner, R. Hasani, A. Amini, T. A. Henzinger, D. Rus, and R. Grosu, “Neural circuit policies enabling auditable autonomy,” Nature Machine Intelligence, vol. 2, no. 10, pp. 642–652, 2020. [Online]. Available: http://dx.doi.org/10.1038/s42256-020-00237-3
  29. M. Chahine, R. Hasani, P. Kao, A. Ray, R. Shubert, M. Lechner, A. Amini, and D. Rus, “Robust flight navigation out of distribution with liquid neural networks,” Science Robotics, vol. 8, no. 77, p. eadc8892, 2023. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.adc8892
  30. R. Hasani, M. Lechner, A. Amini, L. Liebenwein, M. Tschaikowski, G. Teschl, and D. Rus, “Closed-form continuous-depth models,” arXiv preprint arXiv:2106.13898, 2021.
  31. R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu, “Liquid time-constant networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 9, 2021, pp. 7657–7666.
  32. P. Tylkin, T.-H. Wang, T. Seyde, K. Palko, R. Allen, D. Rus, and A. Amini, “Autonomous Flight Arcade Challenge: Single- and Multi-Agent Learning Environments for Aerial Vehicles,” in International Conference on Autonomous Agents and Multiagent Systems, Auckland, US, 2022.
  33. B. Amos and J. Z. Kolter, “OptNet: Differentiable optimization as a layer in neural networks,” in Proceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 70.   PMLR, 2017, pp. 136–145.
  34. S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 11 1997. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now