Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models (2310.06117v2)

Abstract: We present Step-Back Prompting, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide reasoning, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of Step-Back Prompting with PaLM-2L, GPT-4 and Llama2-70B models, and observe substantial performance gains on various challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back Prompting improves PaLM-2L performance on MMLU (Physics and Chemistry) by 7% and 11% respectively, TimeQA by 27%, and MuSiQue by 7%.

Introduction

Transformative advances in NLP have been driven by LLMs based upon the Transformer architecture. These models exhibit astonishing performance across a range of tasks, thanks to their scale and ability to learn from massive amounts of pre-training data. Despite their prowess, these LLMs can struggle with complex multi-step reasoning problems. Recent efforts have employed techniques to scaffold reasoning processes, thereby bolstering these models' problem-solving capabilities. One such technique is the novel "Step-Back Prompting" approach that encourages models to use abstraction for reasoning improvement.

Approach to Problem Solving

"Step-Back Prompting" seeks to enhance deductive processes by promoting abstraction. This approach, inspired by human cognitive strategies, involves decomposing the reasoning process into two main steps: abstraction and reasoning. In the abstraction phase, LLMs identify high-level concepts and principles relevant to a given task. These elements provide a scaffold for the subsequent reasoning, wherein the model deduces the answers to specific questions. These methods of generating abstractions closely parallel human approaches to tackling complex queries.

Empirical Performance and Findings

Experimental results demonstrate the effectiveness of "Step-Back Prompting" on a broad array of challenging tasks, including those within the realms of STEM, Knowledge QA, and Multi-Hop Reasoning. Notably, the technique yielded significant performance enhancements across the board when compared to other methods, notably improving the accuracy of PaLM-2L on rigorous benchmarks. These improvements are particularly impressive in domains requiring detailed domain knowledge and multi-step inference.

Analysis and Conclusions

A variety of analyses highlight the model's capacity for abstraction and indicate reasoning as the primary bottleneck in predictive performance. "Step-Back Prompting" has shown that while abstraction can be a relatively easy skill for LLMs to master, navigating through the reasoning phase remains challenging. The paper suggests directions for future improvements, focusing on refining the reasoning abilities of LLMs. This concept also aligns with the philosophical premise that abstraction isn't just a vague concept but a cornerstone of precision when forming a higher-order understanding. The innovation and simplicity of "Step-Back Prompting" encourage a broader consideration of human-like abstractions in unleashing the concealed potentials of LLMs.