2000 character limit reached
SafeEmbodAI: a Safety Framework for Mobile Robots in Embodied AI Systems (2409.01630v1)
Published 3 Sep 2024 in cs.RO, cs.AI, and cs.ET
Abstract: Embodied AI systems, including AI-powered robots that autonomously interact with the physical world, stand to be significantly advanced by LLMs, which enable robots to better understand complex language commands and perform advanced tasks with enhanced comprehension and adaptability, highlighting their potential to improve embodied AI capabilities. However, this advancement also introduces safety challenges, particularly in robotic navigation tasks. Improper safety management can lead to failures in complex environments and make the system vulnerable to malicious command injections, resulting in unsafe behaviours such as detours or collisions. To address these issues, we propose \textit{SafeEmbodAI}, a safety framework for integrating mobile robots into embodied AI systems. \textit{SafeEmbodAI} incorporates secure prompting, state management, and safety validation mechanisms to secure and assist LLMs in reasoning through multi-modal data and validating responses. We designed a metric to evaluate mission-oriented exploration, and evaluations in simulated environments demonstrate that our framework effectively mitigates threats from malicious commands and improves performance in various environment settings, ensuring the safety of embodied AI systems. Notably, In complex environments with mixed obstacles, our method demonstrates a significant performance increase of 267\% compared to the baseline in attack scenarios, highlighting its robustness in challenging conditions.
- J. Duan, S. Yu, H. L. Tan, H. Zhu, and C. Tan, “A Survey of Embodied AI: from Simulators to Research Tasks,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 6, no. 2, pp. 230–244, 2022.
- R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, A. Kapoor, K. Hausman, B. Ichter, D. Driess, J. Wu, C. Lu, and M. Schwager, “Foundation Models in Robotics: Applications, Challenges, and the Future,” 2023.
- Y. Hu et. al., “Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis,” 2023.
- A. Botta, S. Rotbei, S. Zinno, and G. Ventre, “Cyber Security of Robots: a Comprehensive Survey,” Intelligent Systems with Applications, p. 200237, 2023.
- R. Raval, A. Maskus, B. Saltmiras, M. Dunn, P. J. Hawrylak, and J. Hale, “Competitive Learning Environment for Cyber-Physical System Security Experimentation,” in 2018 1st international conference on data intelligence and security (ICDIS). IEEE, 2018, pp. 211–218.
- S. Longari, J. Jannone, M. Polino, M. Carminati, A. Zanchettin, M. Tanelli, and S. Zanero, “Janus: A Trusted Execution Environment Approach for Attack Detection in Industrial Robot Controllers,” IEEE Transactions on Emerging Topics in Computing, 2024.
- H. Kim, R. Bandyopadhyay, M. Ozmen, Z. Celik, A. Bianchi, Y. Kim, and D. Xu, “A Systematic Study of Physical Sensor Attack Hardness,” in 2024 IEEE Symposium on Security and Privacy (SP). Los Alamitos, CA, USA: IEEE Computer Society, may 2024, pp. 146–146.
- Y. Xu, X. Han, G. Deng, J. Li, Y. Liu, and T. Zhang, “SoK: Rethinking Sensor Spoofing Attacks Against Robotic Vehicles from a Systematic View,” in 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 2023, pp. 1082–1100.
- L. Zhou and V. Kumar, “Robust Multi-Robot Active Target Tracking Against Sensing and Communication Attacks,” IEEE Transactions on Robotics, 2023.
- S. Rivera, S. Lagraa, A. K. Iannillo, and R. State, “Auto-Encoding Robot State Against Sensor Spoofing Attacks,” in 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 2019, pp. 252–257.
- P. Kapoor, A. Vora, and K.-D. Kang, “Detecting and Mitigating Spoofing Attack Against an Automotive Radar,” in 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall). IEEE, 2018, pp. 1–6.
- Z. Han, J. Long, W. Wang, and L. Wang, “Adaptive Tracking Control of Two-Wheeled Mobile Robots under Denial-of-Service Attacks,” ISA transactions, vol. 141, pp. 365–376, 2023.
- W. Zhan, Z. Miao, Y. Chen, Z.-G. Wu, and Y. Wang, “Event-Triggered Finite-Time Formation Control for Networked Nonholonomic Mobile Robots under Denial-of-Service Attacks,” IEEE Transactions on Network Science and Engineering, 2023.
- Y.-S. Hsiao, Z. Wan, T. Jia, R. Ghosal, A. Mahmoud, A. Raychowdhury, D. Brooks, G.-Y. Wei, and V. J. Reddi, “Silent Data Corruption in Robot Operating System: A Case for End-to-End System-Level Fault Analysis Using Autonomous UAVs,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023.
- Y. Zhang and S. Li, “Kinematic Control of Serial Manipulators under False Data Injection Attack,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 4, pp. 1009–1019, 2023.
- M. U. Hadi, R. Qureshi, A. Shah, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wu, S. Mirjalili et al., “A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage,” Authorea Preprints, 2023.
- Y. Lu, “Artificial Intelligence: a Survey on Evolution, Models, Applications and Future Trends,” Journal of Management Analytics, vol. 6, no. 1, pp. 1–29, 2019.
- Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A Survey on Large Language Model (LLM) Security and Privacy: the Good, the Bad, and the Ugly,” High-Confidence Computing, vol. 4, no. 2, p. 100211, 2024.
- F. Wu, N. Zhang, S. Jha, P. McDaniel, and C. Xiao, “A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems,” 2024.
- Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey,” 2024.
- W. Fan, Y. Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models,” 2024.
- R. Jiao, S. Xie, J. Yue, T. Sato, L. Wang, Y. Wang, Q. A. Chen, and Q. Zhu, “Exploring Backdoor Attacks Against Large Language Model-based Decision Making,” 2024.
- P. He, H. Xu, Y. Xing, H. Liu, M. Yamada, and J. Tang, “Data Poisoning for In-context Learning,” 2024.
- Q. Zhang, B. Zeng, C. Zhou, G. Go, H. Shi, and Y. Jiang, “Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications,” 2024.
- R. Pedro, D. Castro, P. Carreira, and N. Santos, “From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?” 2023.
- F. Perez and I. Ribeiro, “Ignore Previous Prompt: Attack Techniques for Language Models,” 2022.
- K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not What You’ve Signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection,” 2023.
- X. Liu, Z. Yu, Y. Zhang, N. Zhang, and C. Xiao, “Automatic and Universal Prompt Injection Attacks Against Large Language Models,” 2024.
- A. Salem, A. Paverd, and B. Köpf, “Maatphor: Automated Variant Analysis for Prompt Injection Attacks,” 2023.
- Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al., “The Rise and Potential of Large Language Model Based Agents: a Survey,” URL https://arxiv. org/abs/2309.07864, 2023.
- C. Xiong, X. Qi, P.-Y. Chen, and T.-Y. Ho, “Defensive Prompt Patch: a Robust and Interpretable Defense of LLMs Against Jailbreak Attacks,” arXiv preprint arXiv:2405.20099, 2024.
- OpenAI, “OpenAI Platform Documentation: Overview,” https://platform.openai.com/docs/overview, 2024, accessed: 2024-07-09.
- LangChain, “Memory Management for Chatbots,” https://python.langchain.com/v0.1/docs/use_cases/chatbots/memory_management/, 2024, accessed: 2024-07-11.
- S. Shahriar, B. Lund, N. R. Mannuru, M. A. Arshad, K. Hayawi, R. V. K. Bevara, A. Mannuru, and L. Batool, “Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency,” 2024.
- J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022.
- Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework,” arXiv preprint arXiv:2308.08155, 2023.
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models are Zero-Shot Reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022.
- Y. Ding, W. Fan, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models,” arXiv preprint arXiv:2405.06211, 2024.
- Y. Shentu, P. Wu, A. Rajeswaran, and P. Abbeel, “From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control,” 2024.
- L. Xia, C. Li, C. Zhang, S. Liu, and P. Zheng, “Leveraging Error-Assisted Fine-Tuning Large Language Models for Manufacturing Excellence,” Robotics and Computer-Integrated Manufacturing, vol. 88, p. 102728, 2024.
- W. Wang, L. Mao, R. Wang, and B.-C. Min, “SRLM: Human-in-Loop Interactive Social Robot Navigation with Large Language Model and Deep Reinforcement Learning,” 2024.
- H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y. Du, Y. Hong, and C. Gan, “3D-VLA: A 3D Vision-Language-Action Generative World Model,” arXiv preprint arXiv:2403.09631, 2024.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.