Reasoning with Large Language Models, a Survey (2407.11511v1)

Abstract: Scaling up LLMs to billions of parameters has opened up possibilities for in-context learning, allowing instruction tuning and few-shot learning on tasks that the model was not specifically trained for. This has achieved breakthrough performance on language tasks such as translation, summarization, and question-answering. Furthermore, in addition to these associative "System 1" tasks, recent advances in Chain-of-thought prompt learning have demonstrated strong "System 2" reasoning abilities, answering a question in the field of artificial general intelligence whether LLMs can reason. The field started with the question whether LLMs can solve grade school math word problems. This paper reviews the rapidly expanding field of prompt-based reasoning with LLMs. Our taxonomy identifies different ways to generate, evaluate, and control multi-step reasoning. We provide an in-depth coverage of core approaches and open problems, and we propose a research agenda for the near future. Finally, we highlight the relation between reasoning and prompt-based learning, and we discuss the relation between reasoning, sequential decision processes, and reinforcement learning. We find that self-improvement, self-reflection, and some metacognitive abilities of the reasoning processes are possible through the judicious use of prompts. True self-improvement and self-reasoning, to go from reasoning with LLMs to reasoning by LLMs, remains future work.

Reasoning with LLMs: A Survey

The paper under review presents a comprehensive survey on the reasoning capabilities of LLMs, an area that has garnered significant attention due to the transformation brought about by models scaling up to hundreds of billions of parameters. The discussion in this survey meticulously examines how these models, traditionally adept at System 1 tasks such as text generation and translation, are being harnessed and adapted to solve more complex System 2 problems, particularly those requiring reasoning across multiple steps.

Core Contributions and Taxonomy

Among the principal contributions of this paper is the establishment of a taxonomy for approaches that tackle prompt-based reasoning tasks with LLMs. The survey categorizes the contemporary literature into three main components of a reasoning pipeline: step generation, step evaluation, and step control. This taxonomy serves as a structured guide for exploring the landscape of reasoning strategies within LLM frameworks.

For step generation, the authors detail methods from manually created prompts to those auto-generated either by models or infused with external knowledge. A pivotal method discussed is the Chain-of-thought prompting, which has significantly improved LLM reasoning performance by encouraging models to articulate intermediate reasoning steps rather than jumping directly to conclusions.

Step evaluation techniques leverage approaches such as self-consistency, where responses are generated multiple times for aggregation, as well as methods involving formal language representations like code to evaluate logical correctness. These strategies aim to mitigate error accumulation and enhance solution robustness.

The third part of the taxonomy, step control, considers strategies from simple greedy approaches to more complex reinforcement learning frameworks that manage the exploration and exploitation trade-offs in reasoning tasks. This element is crucial for balancing the depth and breadth of reasoning paths considered by the model.

Implications and Challenges

The paper identifies several implications for both practical and theoretical developments in the domain of LLMs. Practically, these insights are instrumental in enhancing the applicability of LLMs to real-world problems that necessitate more than superficial text translation or summarization—such as autonomous systems and interactive agents that require deeper reasoning capabilities.

Theoretically, this work raises questions about the limits of LLM reasoning, especially concerning their tendency to hallucinate or provide unfaithful chains of reasoning. While scaling laws have primarily driven improvements in task performances, the paper suggests that emergent reasoning remains an area requiring further exploration, particularly around scaling down this capability to smaller models without significant loss of function.

Numerical Results and Benchmarks

Highlighting bold numerical results, the survey reports notable improvements in benchmark tasks like GSM8K prompted by innovations such as Zero-shot Chain-of-thought, which, using simple iterative steps, have improved accuracy from around 15.6% to over 46.9%. Such results underline how effective prompting can significantly bridge the performance gap in reasoning challenges.

Future Directions

Looking toward the future, the paper proposes several research directions. These include exploring the nuanced interaction between symbolic and connectionist approaches to reasoning, enhancing the interpretability and faithfulness of model outputs, and expanding the range of benchmarks to capture a broader spectrum of system 2 reasoning tasks. Additionally, addressing the computational efficiencies through small model distillation mechanisms remains an open area with practical benefits.

In conclusion, the survey provides a roadmap for advancing reasoning with LLMs. By blending insights from diverse fields like natural language processing, reinforcement learning, and symbolic reasoning, this body of work lays the groundwork for a sophisticated understanding of LLM capabilities and their potential trajectories in artificial intelligence development.