Papers
Topics
Authors
Recent
Search
2000 character limit reached

LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Language

Published 26 Feb 2024 in cs.SE, cs.AI, cs.CL, and cs.PL | (2402.16929v2)

Abstract: LLMs have demonstrated commendable performance across diverse domains. Nevertheless, formulating high-quality prompts to instruct LLMs proficiently poses a challenge for non-AI experts. Existing research in prompt engineering suggests somewhat scattered optimization principles and designs empirically dependent prompt optimizers. Unfortunately, these endeavors lack a structured design template, incurring high learning costs and resulting in low reusability. In addition, it is not conducive to the iterative updating of prompts. Inspired by structured reusable programming languages, we propose LangGPT, a dual-layer prompt design framework as the programming language for LLMs. LangGPT has an easy-to-learn normative structure and provides an extended structure for migration and reuse. Experiments illustrate that LangGPT significantly enhances the performance of LLMs. Moreover, the case study shows that LangGPT leads LLMs to generate higher-quality responses. Furthermore, we analyzed the ease of use and reusability of LangGPT through a user survey in our online community.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. 01-ai. Building the Next Generation of Open-Source and Bilingual LLMs. https://github.com/01-ai/Yi, 2023. original-date: 2023-11-03T16:08:37Z.
  2. GPT-4 Technical Report, December 2023. arXiv:2303.08774 [cs].
  3. Alfred V Aho. Compilers: principles, techniques and tools. Pearson Education India, 2007.
  4. Nabil Alouani. Prompt Engineering Could Be the Hottest Programming Language of 2024 — Here’s Why. https://towardsdatascience.com, December 2023.
  5. Gemini: A Family of Highly Capable Multimodal Models, December 2023. arXiv:2312.11805 [cs].
  6. Qwen Technical Report, September 2023. arXiv:2309.16609 [cs].
  7. Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4, December 2023. arXiv:2312.16171 [cs].
  8. A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair, April 2023. arXiv:2304.08191 [cs].
  9. Chakray. Programming Languages: Types and Features, December 2018.
  10. Harrison Chase. LangChain: Building applications with LLMs through composability. https://github.com/langchain-ai/langchain, October 2022.
  11. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review, October 2023. arXiv:2310.14735 [cs] version: 2.
  12. Black-Box Prompt Optimization: Aligning Large Language Models without Model Training, November 2023. arXiv:2311.04155 [cs].
  13. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Annual Meeting of the Association for Computational Linguistics, 2021.
  14. Mihail Eric. A complete introduction to prompt engineering for large language models. https://www.mihaileric.com, October 2022.
  15. Towards an Automatic Prompt Optimization Framework for AI Image Generation. In HCI International 2023 Posters, Communications in Computer and Information Science, pages 405–410, Cham, 2023. Springer Nature Switzerland.
  16. An Introduction to Language. Cengage Learning, 2018.
  17. Nagaraju Gajula. A Guide to Prompt Engineering in Large Language Models. https://www.latentview.com, August 2023.
  18. GeeksforGeeks. Natural Language Processing(NLP) VS Programming Language. https://www.geeksforgeeks.org, December 2023. Section: Python.
  19. John Gruber. Markdown: Syntax. http://daringfireball.net/projects/markdown, 2012.
  20. Modern compiler design. Springer Science & Business Media, 2012.
  21. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers, September 2023. arXiv:2309.08532 [cs].
  22. Optimizing Prompts for Text-to-Image Generation, December 2022. arXiv:2212.09611 [cs].
  23. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework, November 2023. arXiv:2308.00352 [cs].
  24. Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt Optimization for Few-shot Learning, August 2023. arXiv:2308.07272 [cs].
  25. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6), 2023.
  26. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, pages 1–23, New York, NY, USA, April 2022. Association for Computing Machinery.
  27. Calibrating LLM-Based Evaluator, September 2023. arXiv:2309.13308 [cs].
  28. Mark Lutz. Programming Python: Powerful Object-Oriented Programming. ”O’Reilly Media, Inc.”, December 2010.
  29. Bertalan Meskó. Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial. Journal of Medical Internet Research, 25(1):e50638, October 2023.
  30. Shritam Kumar Mund. The AI War: Mastering the Art of Prompt Engineering in the Era of Large Language Models. https://ai.plainenglish.io, February 2023.
  31. Matt Nigh. ChatGPT3-Free-Prompt-List: A free guide for learning to create ChatGPT3 Prompts. https://github.com/mattnigh/ChatGPT3-Free-Prompt-List, 2023.
  32. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, pages 1–22, New York, NY, USA, October 2023. Association for Computing Machinery.
  33. Foundations of JSON Schema. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, pages 263–273, Republic and Canton of Geneva, CHE, April 2016. International World Wide Web Conferences Steering Committee.
  34. Programming languages: design and implementation. Technical report, Prentice-Hall Englewood Cliffs, NJ, 1984.
  35. Automatic Prompt Optimization with ”Gradient Descent” and Beam Search, October 2023. arXiv:2305.03495 [cs].
  36. Techgpt 2.0: Technology-oriented generative pretrained transformer 2.0. https://github.com/neukg/TechGPT-2.0, 2023.
  37. Tim Rentsch. Object oriented programming. ACM SIGPLAN Notices, 17(9):51–57, September 1982.
  38. Introducing ChatGPT. https://openai.com/blog/chatgpt, 2023.
  39. Robert W Sebesta. Concepts of programming languages. Pearson Education, Inc, 2012.
  40. Michael Sipser. Introduction to the Theory of Computation. ACM Sigact News, 27(1):27–29, 1996. Publisher: ACM New York, NY, USA.
  41. ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation, July 2021. arXiv:2107.02137 [cs].
  42. AutoHint: Automatic Prompt Optimization with Hint Generation, August 2023. arXiv:2307.07415 [cs].
  43. A Survey of Reasoning with Foundation Models, December 2023. arXiv:2312.11562 [cs].
  44. Evaluating Large Language Models on Controlled Generation Tasks, October 2023. arXiv:2310.14542 [cs].
  45. An Introduction to Large Language Models: Prompt Engineering and P-Tuning. https://developer.nvidia.com, April 2023.
  46. HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge, April 2023. arXiv:2304.06975 [cs].
  47. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization, December 2023. arXiv:2310.16427 [cs].
  48. Zhenxuan Wang. How to use prompt engineering with large language models. https://www.thoughtworks.com, August 2023.
  49. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023. arXiv:2201.11903 [cs].
  50. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation, October 2023. arXiv:2308.08155 [cs].
  51. The Rise and Potential of Large Language Model Based Agents: A Survey, September 2023. arXiv:2309.07864 [cs].
  52. Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf, September 2023. arXiv:2309.04658 [cs].
  53. Baichuan 2: Open Large-scale Language Models, September 2023. arXiv:2309.10305 [cs].
  54. Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction, April 2022. arXiv:2204.10360 [cs].
  55. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias, October 2023. arXiv:2306.15895 [cs].
  56. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, pages 1–21, New York, NY, USA, April 2023. Association for Computing Machinery.
  57. GLM-130B: An Open Bilingual Pre-trained Model, October 2023. arXiv:2210.02414 [cs].
  58. PICA: Unleashing The Emotional Power of Large Language Model. https://github.com/NEU-DataMining/PICA, July 2023.
Citations (3)
View on Semantic Scholar

Summary

  • The paper introduces LangGPT, a dual-layer framework that enhances prompt engineering by applying structured, programming language principles.
  • The framework utilizes inherent modules and customizable extension modules to ensure precision and reusability in designing LLM prompts.
  • The study demonstrates LangGPT's superior performance over baseline methods and highlights its role in fostering a community for sharing effective prompt designs.

LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Language

The paper "LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Language" (2402.16929) introduces LangGPT, a framework inspired by programming languages aimed at improving LLMs prompt design. The dual-layer structure of LangGPT supports modularity and reusability, enhancing the efficiency and quality of prompt engineering.

Framework Overview

Interaction of Natural and Programming Languages

LangGPT leverages the systematic nature of programming languages to enrich prompt design. Recognizing the analogies between programming languages and natural language prompts allows the framework to inherit modularity and precision, typical of programming constructs. Figure 1

Figure 1: An analogy between programming language and natural language prompt, depicting hierarchical structures.

LangGPT Dual-Layer Structure

The LangGPT framework is structured into inherent modules and extension modules:

  • Inherent Modules: These serve as standard components, analogous to classes in object-oriented programming. They are predefined for critical aspects like constraints, goals, and roles essential in various scenarios.
  • Extension Modules: When inherent modules don't fully meet specific scenario requirements, new modules can be crafted. This customizable nature ensures that prompt structures can be tailored efficiently to diverse LLM applications. Figure 2

    Figure 2: The design process for extension modules and custom elements showcases module definition and element customization.

Implementation Details

Module and Element Design

LangGPT adopts a dual-layer structure with modules representing broad aspects and elements detailing instructions. This setup parallels class and function dynamics in programming:

  • Modules represent macro-level requirements like objectives or constraints.
  • Elements function within modules as detailed tasks or properties that guide LLM behavior.

The framework operates on the principle that structured prompts lead to more accurate and tailored LLM outputs.

User Engagement and Community

LangGPT has fostered an online community to encourage prompt sharing and learning. This environment facilitates the exchange of effective prompts and design strategies, verified through a user survey assessing ease of use and reusability. Figure 3

Figure 3: User survey ratings on ease of use, revealing community feedback and the framework's accessibility.

Practical Applications

Experimentation and Baseline Comparisons

The framework was tested against baseline methods—Instruction-only and CRISPE. LangGPT demonstrated superior performance in generating quality responses across various LLMs, including ChatGPT-3.5 and Ernie Bot. Figure 4

Figure 4: A case of a boot-licker showcasing responses under different prompts from ChatGPT-3.5.

Prompt Generation by LLMs

LangGPT also facilitates LLMs to autonomously generate high-quality prompts through its structured framework, akin to code generation in a programming context. Figure 5

Figure 5: Example of ChatGPT-3.5 generating prompts with LangGPT, illustrating its capability to aid in prompt creation.

Conclusion

LangGPT represents a significant advancement in prompt engineering, borrowing systematic design principles from programming languages to enhance LLM interactions. Its dual-layer architecture ensures extensibility and reusability, promising to streamline prompt engineering tasks. Future work might focus on integrating tools and further refining token efficiency within the LangGPT framework.

File Document Download Save Streamline Icon: https://streamlinehq.com Markdown

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free