What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use

Abstract: Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.

• The paper introduces ROPE, a paradigm that trains users to articulate clear requirements for improved LLM outputs.
• A specialized training approach led to a 20% performance boost in novices’ ability to express detailed requirements.
• The study confirms that precise input requirements are key to aligning LLM outputs with user needs for customizable applications.

Requirement-Oriented Prompt Engineering (ROPE): A New Paradigm for LLM Interaction

The paper "What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use" introduces Requirement-Oriented Prompt Engineering (ROPE), a new paradigm aimed at enhancing end-user capabilities in prompt engineering for LLMs. Given the increasing proficiency of LLMs in performing complex tasks, the authors argue for a shift in focus towards requirement articulation as a critical skill for achieving higher quality outputs from these models.

Key Contributions and Findings

The authors identify that current prompt engineering practices often emphasize techniques that are increasingly subject to automation, such as formatting and step-by-step prompting. Consequently, the crux of what remains essential for human intervention is the articulation of precise and comprehensive requirements. To address this, they propose ROPE as a paradigm that centers around generating clear requirements, which are then used to guide LLMs, allowing for more end-user-driven customization and goal alignment.

The authors devised a specialized training approach for ROPE, including a suite of assessments and interactive training scenarios, that improved novice end-users’ ability to articulate requirements effectively. In a controlled study involving 30 novices, the ROPE training showed a significant improvement in requirement articulation when compared to traditional prompt engineering training. The results showed a 20% increase in performance, specifically through the enhanced ability to express detailed requirements, that could not be matched by merely optimizing the prompt via automated tools.

The paper also establishes a positive correlation between the quality of input requirements and the quality of LLM outputs. This suggests that clear user requirements are central to optimizing the prompt’s effectiveness, validating the core hypothesis of ROPE.

Implications and Future Directions

Practical Implications: ROPE empowers users to exercise greater control over LLMs, enabling the creation of highly customized and reusable applications, such as personalized chatbots or task-specific agents. This could democratize advanced AI usage, extending programming capabilities to less technically inclined users by focusing on the clarity of requirements rather than intricate technical details.

Theoretical Implications: The ROPE paradigm invites rethinking instructional strategies in computer science education, particularly in human-computer interaction and software design. By foregrounding requirement specification, the study aligns with broader software engineering principles where requirements engineering is a core competency.

Speculative Outlook: As LLM capabilities evolve, integrating ROPE into educational curricula could enhance computational literacy, allowing a wider audience to harness the potential of AI. Future advancements in AI might further automate routine prompt enhancements, positioning ROPE-trained users to leverage AI more strategically.

Limitations and Areas for Research: While the study demonstrates the potential of targeted requirement training, scaling the assessments and training models to diverse task domains remains a promising area for further exploration. Additionally, refining automatic tools to better align with human-specified requirements can enhance the complementarity of human-LLM teams.

In conclusion, this paper positions ROPE as a transformative approach in the domain of AI interaction, emphasizing the necessity of equipping users with skills to precisely articulate requirements for effective LLM utilization. As the AI landscape continues to mature, the focus on requirement engineering could play a pivotal role in bridging human and AI collaboration gaps, fostering a new era of personalized and task-specific AI applications.