The Unreasonable Effectiveness of Eccentric Automatic Prompts (2402.10949v2)

Abstract: LLMs have demonstrated remarkable problem-solving and basic mathematics abilities. However, their efficacy is highly contingent on the formulation of the prompt. This study endeavors to quantify the influence of incorporating "positive thinking" into the system message of the prompt, then compare that to systematic prompt optimization. We assess the performance of 60 combinations of system message snippets, tested with and without Chain of Thought prompting, across three models with parameters ranging from 7 to 70 billion on the GSM8K dataset. Our findings reveal that results do not universally generalize across models. In most instances, the inclusion of "positive thinking" prompts positively affected model performance. Notably, however, Llama2-70B exhibited an exception when not utilizing Chain of Thought, as the optimal system message was found to be none at all. Given the combinatorial complexity, and thus computation time, of experimenting with hand-tuning prompts for large black-box models, we then compared the performance of the best "positive thinking" prompt against the output of systematic prompt optimization. We show that employing an automated prompt optimizer emerges as the most effective method for enhancing performance, even when working with smaller open-source models. Additionally, our findings reveal that the highest-scoring, automatically-optimized prompt exhibits a degree of peculiarity far beyond expectations.

• The paper reveals that systematic prompt optimization yields superior LLM performance compared to 'positive thinking' prompts.
• Experiments on the GSM8K dataset show that models of different sizes respond uniquely to varied prompt modifications.
• The study highlights the complex, model-specific nature of prompt engineering, advocating for algorithmic approaches in future research.

Exploring the Impact of "Positive Thinking" and Systematic Prompt Optimization on LLM Performance

Introduction

The impact of prompt formulation on the performance of LLMs has been a topic of substantial interest and research. This paper offers a comparative analysis of the effects of incorporating "positive thinking" into prompt formulation versus employing systematic prompt optimization techniques. Focusing on the GSM8K dataset, the authors test various combinations of system message modifications across LLMs of different sizes, ranging from 7 to 70 billion parameters. Notably, the paper challenges the generalizability of prompt modifications across models and datasets, emphasizing the nuanced nature of LLM performance enhancement.

Related Work

The research context is firmly rooted in the evolving discipline of prompt engineering, citing seminal works that establish the foundational significance of systematic prompt modifications. Previous studies have varied in their approaches to prompt engineering, ranging from simple modifications to complex, systematic optimization strategies. This paper builds on the growing body of literature by offering a structured analysis of both "positive thinking" prompts and automated prompt optimization, focusing specifically on their applicability and effectiveness across different LLMs.

Experimental Design

The research conducted a series of experiments utilizing the GSM8K dataset to evaluate the models' mathematical problem-solving capabilities. A controlled set of "positive thinking" prompts was tested against outputs produced through systematic prompt optimization. This methodological approach allows for a direct comparison of efficacy between conventional wisdom in prompt engineering and algorithmically driven optimization techniques.

Results and Discussion

The paper reveals that "positive thinking" prompts, while impactful in certain contexts, do not universally enhance performance across all models. Interestingly, the Llama2-70B model demonstrated a notable exception, performing optimally without any systematic message modification when not using Chain of Thought prompting. The findings significantly underline the complexity of prompt engineering, indicating that effective prompt formulation is highly model and context-specific.

Moreover, systematic prompt optimization was found superior in improving model performance, signifying the potential limitations of human intuition in crafting prompts for LLMs. The optimized prompts, characterized by their peculiar and unexpected nature, underscore the benefits of leveraging algorithmic methods for prompt engineering.

Implications for Future Research

This paper's insights into the nuanced efficacy of prompt modifications open avenues for future research, particularly in expanding the scope of datasets and models analyzed. Additionally, the demonstrated superiority of automatic prompt optimization suggests a shift towards more algorithmically driven methods in the development of LLM applications.

Conclusion

In essence, this research contributes to the nuanced understanding of prompt engineering, highlighting the complexity and model-specific nature of effective prompt formulation. Through a direct comparison of "positive thinking" prompts and systematic prompt optimization, the paper illustrates the potential limitations of manual prompt engineering and underscores the efficacy of automatic optimization strategies. These findings not only challenge existing assumptions within the field but also pave the way for further exploration into algorithmic prompt optimization as a foundational component of LLM performance enhancement.