Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications (2404.06921v1)

Published 10 Apr 2024 in cs.CL and cs.AI

Abstract: LLMs are evolving beyond their classical role of providing information within dialogue systems to actively engaging with tools and performing actions on real-world applications and services. Today, humans verify the correctness and appropriateness of the LLM-generated outputs (e.g., code, functions, or actions) before putting them into real-world execution. This poses significant challenges as code comprehension is well known to be notoriously difficult. In this paper, we study how humans can efficiently collaborate with, delegate to, and supervise autonomous LLMs in the future. We argue that in many cases, "post-facto validation" - verifying the correctness of a proposed action after seeing the output - is much easier than the aforementioned "pre-facto validation" setting. The core concept behind enabling a post-facto validation system is the integration of an intuitive undo feature, and establishing a damage confinement for the LLM-generated actions as effective strategies to mitigate the associated risks. Using this, a human can now either revert the effect of an LLM-generated output or be confident that the potential risk is bounded. We believe this is critical to unlock the potential for LLM agents to interact with applications and services with limited (post-facto) human involvement. We describe the design and implementation of our open-source runtime for executing LLM actions, Gorilla Execution Engine (GoEX), and present open research questions towards realizing the goal of LLMs and applications interacting with each other with minimal human supervision. We release GoEX at https://github.com/ShishirPatil/gorilla/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. VLDB, 2015.
  3. Many-shot jailbreaking. 2024.
  4. AWS. Testing your AWS KMS API calls. https://docs.aws.amazon.com/kms/latest/developerguide/programming-dryrun.html.
  5. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022.
  6. Apache Flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering, 2015.
  7. A survey on evaluation of large language models. ACM TIST, 2023.
  8. StruQ: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363, 2024.
  9. Protection in the Hydra operating system. SOSP, 1975.
  10. Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In ACM AISec, 2023.
  11. Principles of transaction-oriented database recovery. ACM CSUR, 1983.
  12. Exploiting programmatic behavior of LLMs: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733, 2023.
  13. Kubernetes. kubectl usage conventions. https://kubernetes.io/docs/reference/kubectl/conventions/.
  14. Theodore A Linden. Operating system structures to support security and reliable software. CSUR, 1976.
  15. Prompt injection attack against LLM-integrated applications. arXiv preprint arXiv:2306.05499, 2023.
  16. Microsoft. Presidio - data protection and de-identification SDK. https://github.com/microsoft/presidio.
  17. OpenAI. ChatGPT plugins, 2023. https://openai.com/blog/chatgpt-plugins.
  18. TALM: Tool augmented language models, 2022.
  19. Generative agents: Interactive simulacra of human behavior. In UIST, 2023.
  20. Gorilla: Large language model connected with massive APIs. arXiv preprint arXiv:2305.15334, 2023.
  21. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527, 2022.
  22. Jatmo: Prompt injection defense by task-specific finetuning. arXiv preprint arXiv:2312.17673, 2023.
  23. ToolLLM: Facilitating large language models to master 16000+ real-world APIs, 2023.
  24. Ros: an open-source robot operating system. In ICRA workshop on open source software, volume 3, pp.  5. Kobe, Japan, 2009.
  25. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2024.
  26. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922, 2023.
  27. Site isolation: Process separation for web sites within the browser. In USENIX Security, 2019.
  28. Code Llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
  29. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  30. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of LLMs through a global scale prompt hacking competition. arXiv preprint arXiv:2311.16119, 2023.
  31. Xuchen Suo. Signed-Prompt: A new approach to prevent prompt injection attacks against LLM-integrated applications. arXiv preprint arXiv:2401.07612, 2024.
  32. Tensor trust: Interpretable prompt injection attacks from an online game. arXiv preprint arXiv:2311.01011, 2023.
  33. DecodingTrust: A comprehensive assessment of trustworthiness in GPT models. arXiv preprint arXiv:2306.11698, 2023a.
  34. Voyager: An open-ended embodied agent with large language models, 2023b.
  35. The Cambridge CAP computer and its operating system. 1979.
  36. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023.
  37. SecGPT: An execution isolation architecture for llm-based systems. arXiv preprint arXiv:2403.04960, 2024.
  38. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
  39. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197, 2023.
  40. Assessing prompt injection risks in 200+ custom gpts. arXiv preprint arXiv:2311.11538, 2023.
  41. Siren’s song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
  42. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (10)
  1. Shishir G. Patil (8 papers)
  2. Tianjun Zhang (38 papers)
  3. Vivian Fang (5 papers)
  4. Noppapon C. (1 paper)
  5. Roy Huang (1 paper)
  6. Aaron Hao (1 paper)
  7. Martin Casado (2 papers)
  8. Joseph E. Gonzalez (167 papers)
  9. Raluca Ada Popa (20 papers)
  10. Ion Stoica (177 papers)
Citations (4)
View on Semantic Scholar

Summary

Enabling Autonomous Actions in LLMs: A Study on Post-Facto Validation and the Gorilla Execution Engine

Introduction to Autonomous LLM Systems

The ability of LLMs to transition from passive information provision to active engagement with real-world applications marks a significant evolution in the field of artificial intelligence. This shift towards autonomy in LLM systems necessitates a reevaluation of how these models interact with external applications and perform tasks without extensive human oversight. The paper by Patil et al. explores this evolution in depth, identifying the inherent trustworthiness challenges and proposing innovative solutions to address them.

Challenges in LLM-Powered Applications

The exploration of autonomous LLM systems unveils several trust and safety concerns:

  • Unpredictability: LLM-based applications, due to their inherent stochastic nature, exhibit unpredictable behavior, raising concerns over their reliability and the appropriateness of the actions they might take.
  • Unreliability: The difficulty in thoroughly testing LLMs to ensure error-free performance underlines the challenges in integrating these models into existing trustworthy systems.
  • Delayed and Aggregate Feedback Loops: Traditional feedback mechanisms, essential for iterative development and refinement, are less effective. LLM-powered systems may not provide immediate or individual action-based feedback, complicating error identification and system improvement efforts.
  • Integration Challenges: Integrating LLMs with existing systems challenges current paradigms of software testing, notably unit and integration testing, due to the dynamic outputs of LLMs.

Post-Facto LLM Validation: A Novel Approach

To counter these challenges, the paper introduces "post-facto LLM validation," a method that relies on evaluating the outcomes of LLM-generated actions rather than pre-validating each potential action. This approach places humans as the final arbiters of an LLM's output, allowing for a more feasible and efficient means of supervision. To mitigate the risks associated with executing unintended actions, the authors propose two core concepts: undo and damage confinement, offering strategies to revert or limit the impact of any actions taken by LLMs.

Undo Mechanism

The undo mechanism allows for actions taken by LLM systems to be reversible, affording a layer of safety by enabling the system to revert to a prior state post-action execution. This concept necessitates maintaining multiple versions of the system state, raising considerations regarding the complexity and resource implications of such an approach.

Damage Confinement

In scenarios where undoing an action is not feasible, the concept of damage confinement introduces a means to bound the potential risks associated with an action. This approach allows developers and users to define their risk tolerance, effectively confining the 'blast radius' of any unintended consequences.

Gorilla Execution Engine (GoEx)

As a practical step towards realizing the vision of autonomous LLM-powered applications, the paper presents the Gorilla Execution Engine (GoEx). GoEx is an open-source runtime designed to safely execute actions generated by LLMs, incorporating the principles of undo and damage confinement. The engine supports various actions, including RESTful API calls, database operations, and filesystem interactions, each tailored with unique handling mechanisms to ensure safety and adherence to the proposed validation approach.

Implications and Future Directions

The conceptualization and implementation of GoEx hint at a future where LLMs can autonomously interact with applications and services, driving innovation while addressing the critical need for trustworthiness and safety. This paper not only lays the groundwork for further exploration into autonomous LLM systems but also opens up a discussion on the importance of designing LLM-friendly APIs and the potential need for new software development paradigms tailored to the unique challenges of integrating LLMs into our digital infrastructure.

Conclusion

The transition towards autonomous LLM-powered systems poses significant challenges in ensuring their safe and reliable operation. The introduction of post-facto LLM validation, alongside the development of the Gorilla Execution Engine, represents a forward-thinking approach to enabling LLMs to take actions autonomously while maintaining human oversight. As the field of artificial intelligence continues to evolve, the concepts and solutions proposed in this paper provide a valuable foundation for addressing the complex interplay between autonomy, trustworthiness, and user safety in the next generation of LLM applications.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube

HackerNews

  1. Autonomous LLM agents with human-out-of-loop (14 points, 8 comments)