Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization (2404.02319v2)

Published 2 Apr 2024 in cs.CL, cs.AI, and cs.LG

Abstract: In many modern LLM applications, such as retrieval augmented generation, prompts have become programs themselves. In these settings, prompt programs are repeatedly called with different user queries or data instances. A big practical challenge is optimizing such prompt programs. Recent work has mostly focused on either simple prompt programs or assumed that the general structure of a prompt program is fixed. We introduce SAMMO, a framework to perform symbolic prompt program search for compile-time optimizations of prompt programs. SAMMO represents prompt programs on a symbolic level which allows for a rich set of transformations that can be searched over during optimization. We show that SAMMO generalizes previous methods and improves the performance of complex prompts on (1) instruction tuning, (2) RAG pipeline tuning, and (3) prompt compression, across several different LLMs. We make all code available open-source at https://github.com/microsoft/sammo .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (30)
  1. Task-oriented dialogue as dataflow synthesis. TACL.
  2. Leveraging code to improve in-context learning for semantic parsing.
  3. Language models are few-shot learners. NeurIPS.
  4. Instructzero: Efficient instruction optimization for black-box large language models.
  5. Batch prompting: Efficient inference with large language model apis.
  6. Promptbreeder: Self-referential self-improvement via prompt evolution.
  7. Mixtral of experts.
  8. LLMLingua: Compressing prompts for accelerated inference of large language models. In EMNLP.
  9. Hoyoun Jung and Kyung-Joong Kim. 2023. Discrete prompt compression with reinforcement learning.
  10. Dspy: Compiling declarative language model calls into self-improving pipelines.
  11. The power of scale for parameter-efficient prompt tuning. In EMNLP.
  12. Compressing context to enhance inference efficiency of large language models. In EMNLP.
  13. Batchprompt: Accomplish more with less.
  14. Learning to compress prompts with gist tokens. In NeurIPS.
  15. GrIPS: Gradient-free, edit-based instruction search for prompting large language models. In EACL.
  16. Automatic prompt optimization with "gradient descent" and beam search.
  17. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In NAACL.
  18. Regularized evolution for image classifier architecture search. In AAAI.
  19. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.
  20. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TMLR.
  21. Black-box tuning for language-model-as-a-service. In ICML.
  22. Llama 2: Open foundation and fine-tuned chat models.
  23. Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks. In EMNLP.
  24. Building a semantic parser overnight. In ACL.
  25. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In EMNLP: Findings.
  26. Prompt engineering a prompt engineer.
  27. Did you read the instructions? Rethinking the effectiveness of task definitions in instruction learning. In ACL.
  28. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning.
  29. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In National Conference on Artificial Intelligence.
  30. Large language models are human-level prompt engineers.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Tobias Schnabel (21 papers)
  2. Jennifer Neville (57 papers)
Citations (2)
View on Semantic Scholar

Summary

Prompts as Programs: Enhancing LLM Prompt Optimization through Compile-Time Refinement

Introduction to SAMMO

Recent advances in LLMs have led to more elaborate prompts that contain intricate structures, akin to programming constructs. These complex prompts, termed metaprompt programs, allow for dynamic assembly of instructions, examples, and data formats which can be tailored according to the input at runtime. However, the optimization of such metaprompts, especially in a manner that is both efficient and effective, poses a significant challenge. Addressing this, the research introduces SAMMO (Structure-Aware Multi-objective Metaprompt Optimization), a framework built for compile-time optimization of metaprompt programs. SAMMO represents metaprompts as structured objects, enabling elaborate transformations and optimizations over traditional text-based approaches. Through experimental validation, SAMMO demonstrates substantial improvements in optimizing complex prompts across different use cases and LLMs.

Problem Definition and Notation

SAMMO formalizes the optimization of metaprompts as a search problem over a defined space of potential metaprompt configurations. It introduces a notation system where a metaprompt, π\pi, operates on input data XX, to produce a string for LLM processing. The optimization's objective is to discover an optimal metaprompt, π∗\pi^*, which minimizes loss across varied data distributions. Importantly, SAMMO differentiates itself by focusing on compile-time optimizations, executing modifications to the metaprompt structure and parameters before deployment, as opposed to run-time optimizations which require adjustments at the point of execution for each input.

Key Features of SAMMO

  • Programmatic Representation of Metaprompts: By structuring metaprompts as directed acyclic graphs, SAMMO provides a modular and comprehensive approach to prompt design, facilitating intricate editing and refinement.
  • Intelligent Search Algorithms: Utilizing a set of mutation operators, SAMMO employs genetic search algorithms to explore and select the most promising metaprompt configurations, considering both structural and content modifications.
  • Framework Flexibility: SAMMO’s framework supports an array of optimization scenarios, from instruction tuning and retrieval augmented generation optimization to prompt compression, making it scalable across different application needs.

Experimental Evaluation and Findings

SAMMO’s efficacy was explored through three core use-cases: instruction tuning, optimizing retrieval-augmented generation (RAG) pipelines, and prompt compression. Across these scenarios, SAMMO achieved significant improvements:

  • Instruction Tuning: SAMMO outperformed existing methods in instruction optimization, showing a 10-100\% increase in performance across multiple LLMs.
  • RAG Pipeline Tuning: In the scenario of retrieval-augmented generation, SAMMO demonstrated gains of 26-133\%, indicating its strong potential in fine-tuning complex data interaction processes within prompts.
  • Prompt Compression: SAMMO significantly reduced the computational cost of prompts without compromising on performance, achieving over 40\% compression across different models.

Implications and Future Directions

The introduction of SAMMO paves the way for more sophisticated prompt engineering methods that can adapt to the evolving complexity of LLM prompts. Its ability to generalize across various optimization techniques and demonstrate notable improvements irrespective of the underlying LLM suggests a promising avenue for research into compile-time optimization practices. Future work could explore extending SAMMO’s capabilities to accommodate real-time optimization needs and further refine its algorithmic efficiency for broader application scenarios.

Conclusion

SAMMO represents a significant step forward in the optimization of metaprompts for LLMs, offering a structured, efficient, and effective framework for prompt enhancement. Its programmatic approach to prompt construction and optimization stands to benefit a wide array of applications, from natural language processing tasks to more complex data interrogation and generation scenarios. As LLMs continue to advance, tools like SAMMO will be critical in harnessing their full potential through finely-tuned and efficiently-optimized prompts.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown