Task-oriented Prompt Enhancement via Script Generation (2409.16418v1)

Abstract: LLMs have demonstrated remarkable abilities across various tasks, leveraging advanced reasoning. Yet, they struggle with task-oriented prompts due to a lack of specific prior knowledge of the task answers. The current state-of-the-art approach, PAL, utilizes code generation to address this issue. However, PAL depends on manually crafted prompt templates and examples while still producing inaccurate results. In this work, we present TITAN-a novel strategy designed to enhance LLMs' performance on task-oriented prompts. TITAN achieves this by generating scripts using a universal approach and zero-shot learning. Unlike existing methods, TITAN eliminates the need for detailed task-specific instructions and extensive manual efforts. TITAN enhances LLMs' performance on various tasks by utilizing their analytical and code-generation capabilities in a streamlined process. TITAN employs two key techniques: (1) step-back prompting to extract the task's input specifications and (2) chain-of-thought prompting to identify required procedural steps. This information is used to improve the LLMs' code-generation process. TITAN further refines the generated script through post-processing and the script is executed to retrieve the final answer. Our comprehensive evaluation demonstrates TITAN's effectiveness in a diverse set of tasks. On average, TITAN outperforms the state-of-the-art zero-shot approach by 7.6% and 3.9% when paired with GPT-3.5 and GPT-4. Overall, without human annotation, TITAN achieves state-of-the-art performance in 8 out of 11 cases while only marginally losing to few-shot approaches (which needed human intervention) on three occasions by small margins. This work represents a significant advancement in addressing task-oriented prompts, offering a novel solution for effectively utilizing LLMs in everyday life tasks.

• The paper introduces TITAN, a novel framework that automates task-specific prompt tuning to improve LLM performance.
• It leverages step-back prompting to extract task specifications and chain-of-thought prompting to structure solution steps.
• Evaluation results show TITAN outperforms zero-shot methods by up to 7.6% and approaches few-shot performance without manual templates.

The paper "Task-oriented Prompt Enhancement via Script Generation" introduces TITAN, a novel strategy aimed at improving the performance of LLMs on task-oriented prompts. LLMs, while effective across various tasks due to their reasoning capabilities, often face challenges with task-specific prompts that require prior task knowledge. Traditional methods like PAL rely on manually crafted templates and examples, which can lead to inaccuracies and inefficiencies.

TITAN is designed to overcome these limitations using a more universal and automated approach. It employs two main techniques:

1. Step-back prompting: This technique is used to extract input specifications of a task. It helps the LLM better understand the task requirements without needing detailed and manually designed instructions.
2. Chain-of-thought prompting: This technique aids in identifying the necessary procedural steps to accomplish a task, leveraging the LLM’s inherent analytical abilities.

Together, these techniques enhance the code-generation process of LLMs, allowing them to create scripts that can solve tasks effectively. TITAN eliminates the need for human-annotated, task-specific instructions, which makes it distinct from previous methods.

Additionally, TITAN incorporates post-processing to refine the generated scripts, ensuring more accurate results. The script is then executed to yield the final answer to the task.

The paper provides a comprehensive evaluation demonstrating TITAN's effectiveness across various tasks. It outperforms state-of-the-art zero-shot approaches by 7.6% and 3.9% when used with GPT-3.5 and GPT-4, respectively. TITAN achieves state-of-the-art performance in 8 out of 11 cases and performs comparably to few-shot methods, which require human intervention, losing only marginally in three scenarios.

This work significantly advances the use of LLMs for task-oriented applications, offering an efficient method for managing prompts without the need for extensive manual input, thereby expanding the practical utility of LLMs in everyday tasks.