Can Stories Help LLMs Reason? Curating Information Space Through Narrative (2410.19221v1)

Abstract: Narratives are widely recognized as a powerful tool for structuring information and facilitating comprehension of complex ideas in various domains such as science communication. This paper investigates whether incorporating narrative elements can assist LLMs in solving complex problems more effectively. We propose a novel approach, Story of Thought (SoT), integrating narrative structures into prompting techniques for problem-solving. This approach involves constructing narratives around problem statements and creating a framework to identify and organize relevant information. Our experiments show that using various LLMs with SoT consistently surpasses using them with other techniques on physics, chemistry, math, and biology questions in both the GPQA and JEEBench datasets. The narrative-based information curation process in SoT enhances problem comprehension by contextualizing critical in-domain information and highlighting causal relationships within the problem space.

• The paper introduces the Story of Thought approach that integrates narrative techniques into LLM reasoning, significantly boosting problem-solving accuracy.
• It employs a three-step pipeline—question clarification, narrative generation, and problem solving—to outperform traditional zero-shot and CoT methods.
• Empirical results on GPQA and JEEBench datasets demonstrate that narrative-driven prompts improve LLM comprehension and contextual reasoning.

Can Stories Help LLMs Reason? An Analysis of the Story of Thought Approach

The paper Can Stories Help LLMs Reason? Curating Information Space Through Narrative presents a novel methodology aimed at enhancing the problem-solving capabilities of LLMs by integrating narrative elements into their reasoning processes. Notably, this research introduces the "Story of Thought" (SoT) approach, which structures problem-solving prompts within narrative frameworks to improve comprehension and contextual understanding.

Research Context and Inquiry

The authors address the potential of narratives, widely acknowledged for their ability to elucidate and structure complex concepts in human cognition, as a means to enhance LLM reasoning. Building upon the established cognitive benefits of storytelling, this paper aims to determine if similar benefits can be conferred upon LLMs. The paper poses two key research questions: whether LLMs can generate coherent narratives around problem statements effectively, and if narrative incorporation can boost model performance in complex problem-solving scenarios.

Methodological Approach

The SoT method comprises a three-step pipeline designed to improve the reasoning and problem-solving proficiency of LLMs by fostering narrative thinking:

1. Question Clarification: LLMs are tasked with breaking down and clarifying complex questions into sub-components, effectively serving as explorers to identify relevant subject areas for further narrative development.
2. Narrative Generation: Using various narrative techniques, such as progressive disclosure, analogy, and metaphor, the LLMs generate structured narrative explanations of the clarified problem components. These narratives aim to provide context, relational reasoning, and cognitive insights, guiding the LLMs in understanding the problem space comprehensively.
3. Problem Solving: The narrative-based understanding developed in the previous step is then employed to support the LLMs in addressing the task at hand, with an emphasis on integrating narrative comprehension into the reasoning process.

Experimental Evaluation

The paper evaluates the efficacy of the SoT approach on two challenging datasets: GPQA and JEEBench, both of which consist of complex, domain-specific questions requiring deep comprehension and reasoning. The paper benchmarks the SoT approach against various other prompting strategies, such as zero-shot and chain-of-thought (CoT) prompting.

The results reveal that the SoT method outperforms traditional prompting techniques across multiple LLMs, including models by Meta, Mistral, OpenAI, and Microsoft. Specifically, SoT displays significant accuracy improvements, notably enhancing the capabilities of models like Llama 3 70B and GPT-4 compared to their zero-shot and CoT counterparts. This suggests that narrative-driven prompts may effectively support LLMs in parsing and solving intricate problems by organizing information within a coherent context.

Implications and Future Directions

The findings emphasize the potential of narrative integration in LLM reasoning processes, offering a promising avenue for enhancing computational models' problem-solving aptitudes. The narrative-based prompting could revolutionize how LLMs are trained and leveraged in fields requiring complex decision-making and contextual understanding, such as science education and communication.

The paper opens new pathways for examining how narratives affect artificial reasoning and the broader practical implications for LLM adaptation in various applications. Further research could explore additional narrative techniques, cross-context efficacy, and their scalability across different domains and problem types. Additionally, the development of more refined methods for evaluating narrative quality and effectiveness could further enhance our understanding of narratives' role in AI reasoning.

In conclusion, the integration of narrative structures in LLM prompting as proposed in this paper shows promise as a technique for improving LLM reasoning capabilities, suggesting that storytelling's cognitive advantages can be systematically leveraged within AI frameworks to achieve superior problem-solving outcomes.