Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner's Guide (2402.14837v1)

Published 18 Feb 2024 in cs.CL, cs.AI, cs.HC, and cs.LG

Abstract: Due to rapid advancements in the development of LLMs, programming these models with prompts has recently gained significant attention. However, the sheer number of available prompt engineering techniques creates an overwhelming landscape for practitioners looking to utilize these tools. For the most efficient and effective use of LLMs, it is important to compile a comprehensive list of prompting techniques and establish a standardized, interdisciplinary categorization framework. In this survey, we examine some of the most well-known prompting techniques from both academic and practical viewpoints and classify them into seven distinct categories. We present an overview of each category, aiming to clarify their unique contributions and showcase their practical applications in real-world examples in order to equip fellow practitioners with a structured framework for understanding and categorizing prompting techniques tailored to their specific domains. We believe that this approach will help simplify the complex landscape of prompt engineering and enable more effective utilization of LLMs in various applications. By providing practitioners with a systematic approach to prompt categorization, we aim to assist in navigating the intricacies of effective prompt design for conversational pre-trained LLMs and inspire new possibilities in their respective fields.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Conversational health agents: A personalized LLM-powered agent framework. arXiv preprint arXiv:2310.02374, 2023.
  2. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441, 2022.
  3. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.
  4. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021.
  5. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023.
  6. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  7. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  8. E. Y. Chang. Prompting large language models with the socratic method. In 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), pages 0351–0360. IEEE, 2023.
  9. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023.
  10. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.
  11. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092, 2023.
  12. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474, 2023.
  13. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  14. Revisit input perturbation problems for llms: A unified robustness evaluation framework for noisy slot filling task. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 682–694. Springer, 2023.
  15. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
  16. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022.
  17. Self-explanation prompting improves dialogue understanding in large language models. arXiv preprint arXiv:2309.12940, 2023.
  18. On the effectiveness of creating conversational agent personalities through prompting. arXiv preprint arXiv:2310.11182, 2023.
  19. Zero-shot recommendations with pre-trained large language models for multimodal nudging. arXiv preprint arXiv:2309.01026, 2023.
  20. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022.
  21. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
  22. S. Kanti Karmaker Santu and D. Feng. TELeR: A general taxonomy of LLM prompts for benchmarking complex tasks. arXiv preprint arXiv:2305.11430, 2023.
  23. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
  24. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.
  25. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digital health, 2(2):e0000198, 2023.
  26. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14943–14952, 2023.
  27. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023a.
  28. icsberts: Optimizing pre-trained language models in intelligent customer service. Procedia Computer Science, 222:127–136, 2023b.
  29. FinGPT: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485, 2023c.
  30. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379, 2023.
  31. " oops, did i just say that?" testing and repairing unethical suggestions of large language models with suggest-critique-reflect process. arXiv preprint arXiv:2305.02626, 2023.
  32. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.
  33. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943, 2021.
  34. Auditing large language models: a three-layered approach. AI and Ethics, pages 1–31, 2023.
  35. Skeleton-of-thought: Large language models can do parallel decoding. arXiv preprint arXiv:2307.15337, 2023.
  36. C. O’neil. Weapons of math destruction: How big data increases inequality and threatens democracy. Crown, 2017.
  37. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  38. REFINER: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904, 2023.
  39. True few-shot learning with language models. Advances in neural information processing systems, 34:11054–11070, 2021.
  40. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019.
  41. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  42. L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7, 2021.
  43. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
  44. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633, 2021.
  45. T. Schick and H. Schütze. It’s not just size that matters: Small language models are also few-shot learners, 2021.
  46. Pattern-oriented software architecture, patterns for concurrent and networked objects, volume 2. John Wiley & Sons, 2013.
  47. Towards a catalog of prompt patterns to enhance the discipline of prompt engineering. https://www.dre.vanderbilt.edu/~schmidt/PDF/ADA-User-Journal.pdf, 2023.
  48. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324, 2023.
  49. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061, 2022.
  50. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020.
  51. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
  52. Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv preprint arXiv:2302.12822, 2023.
  53. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023.
  54. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
  55. K.-J. Tokayev. Ethical implications of large language models a multidimensional exploration of societal, economic, and technical concerns. International Journal of Social Analytics, 8(9):17–33, 2023.
  56. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  57. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  58. Chatgpt: five priorities for research. Nature, 614(7947):224–226, 2023.
  59. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  60. Iteratively prompt pre-trained language models for chain of thought. arXiv preprint arXiv:2203.08383, 2022a.
  61. Boosting language models reasoning with chain-of-knowledge prompting. arXiv preprint arXiv:2306.06427, 2023a.
  62. Knowledge-driven CoT: Exploring faithful reasoning in LLMs for knowledge-intensive question answering. arXiv preprint arXiv:2308.13259, 2023b.
  63. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022b.
  64. Y. Wang and Y. Zhao. Metacognitive prompting improves understanding in large language models. arXiv preprint arXiv:2308.05342, 2023.
  65. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a.
  66. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022b.
  67. Leveraging large language models to power chatbots for collecting user self-reported data. arXiv preprint arXiv:2301.05843, 2023.
  68. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint arXiv:2302.11382, 2023.
  69. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082, 2023.
  70. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023.
  71. Decomposition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633, 2023.
  72. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10, 2022.
  73. An LLM can fool itself: A prompt-based adversarial attack. arXiv preprint arXiv:2310.13345, 2023.
  74. RCOT: Detecting and rectifying factual inconsistency in reasoning by reversing chain-of-thought. arXiv preprint arXiv:2305.11499, 2023.
  75. Harnessing the power of LLMs in practice: A survey on ChatGPT and beyond. arXiv preprint arXiv:2304.13712, 2023a.
  76. ChatGPT is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306.11489, 2023b.
  77. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
  78. Complementary explanations for effective in-context learning. arXiv preprint arXiv:2211.13892, 2022.
  79. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959, 2023.
  80. Building trust in conversational ai: A comprehensive review and solution architecture for explainable, privacy-aware systems using llms and knowledge graph. arXiv preprint arXiv:2308.13534, 2023.
  81. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–21, 2023.
  82. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  83. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR, 2021.
  84. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797, 2023.
  85. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023.
  86. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
  87. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Oluwole Fagbohun (4 papers)
  2. Rachel M. Harrison (3 papers)
  3. Anton Dereventsov (17 papers)
Citations (1)
View on Semantic Scholar