Abstraction-of-Thought Makes Language Models Better Reasoners (2406.12442v2)

Abstract: Abstract reasoning, the ability to reason from the abstract essence of a problem, serves as a key to generalization in human reasoning. However, eliciting LLMs to perform reasoning with abstraction remains unexplored. This paper seeks to bridge this gap by introducing a novel structured reasoning format called Abstraction-of-Thought (AoT). The uniqueness of AoT lies in its explicit requirement for varying levels of abstraction within the reasoning process. This approach could elicit LLMs to first contemplate on the abstract level before incorporating concrete details, which is overlooked by the prevailing step-by-step Chain-of-Thought (CoT) method. To align models with the AoT format, we present AoT Collection, a generic finetuning dataset consisting of 348k high-quality samples with AoT reasoning processes, collected via an automated and scalable pipeline. We finetune a wide range of LLMs with AoT Collection and conduct extensive evaluations on 23 unseen tasks from the challenging benchmark Big-Bench Hard. Experimental results indicate that models aligned to AoT reasoning format substantially outperform those aligned to CoT in many reasoning tasks.

• The paper presents a novel Abstraction-of-Thought approach that augments CoT reasoning, achieving around 10% improvement on the BBH benchmark.
• It introduces a scalable AOT COLLECTION of 348,000 samples across 216 tasks, integrating reasoning in both natural and programming languages.
• The ablation study confirms that structural abstraction independently enhances problem decomposition, suggesting potential for refined pre-training strategies.

Abstraction-of-Thought Makes LLMs Better Reasoners: An Examination

The paper "Abstraction-of-Thought Makes LLMs Better Reasoners" introduces a novel approach to enhancing the reasoning capabilities of LLMs (LMs) by exploring the concept of abstract reasoning. The authors identify a critical gap in existing models, which largely rely on the step-by-step Chain-of-Thought (CoT) reasoning methodology that lacks the capacity to engage with abstraction.

The paper proposes the "Abstraction-of-Thought" (AoT) reasoning format, which mandates varying levels of abstraction throughout the reasoning process, encouraging LMs to first establish a high-level abstract framework before exploring specific details. This approach attempts to mimic human abstract reasoning, which prioritizes the essence of a problem to derive a generalizable solution.

In order to operationalize this framework, the authors present AOT COLLECTION, a comprehensive finetuning dataset composed of 348,000 high-quality samples explicitly organized in the AoT format. This dataset was accumulated via an automated pipeline, ensuring scalability and broad task coverage across 216 tasks that were not domain-specific. Notably, the dataset incorporates reasoning processes in both natural language and programming language, thereby introducing flexibility and robustness into the trained models.

Empirical analysis of this AoT-finetuned approach was conducted using the Big-Bench Hard (BBH) benchmark, comprising 23 unseen reasoning tasks. The results demonstrate a marked improvement in reasoning performance among AoT-finetuned models compared to their CoT counterparts in both zero-shot and few-shot scenarios, particularly in algorithmically intensive tasks. Specifically, the AoT methodology yielded improvement margins of approximately 10% compared to CoT-finetuning.

One of the more innovative aspects of this research is the seamless integration of natural language with computational reasoning via programming language tasks. This dual-pronged approach extends the utility of LMs, suggesting hybridized methodologies could further enhance their performance. Additionally, the paper presents a compelling argument that abstraction fundamentally enhances reasoning capabilities, suggesting models trained with AoT are more adept at understanding and decomposing complex tasks.

An ablation paper within the paper highlights the impact of the AoT reasoning format independent of the influence of data source or scale, underscoring the importance of the structure in AoT over content when it comes to effective reasoning. Further experimentation on training data scale indicates that AoT-finetuned models benefit more significantly as the volume of data increases, hinting at the potential scalability of this approach.

The findings from this research have significant implications both practically and theoretically. By aligning LMs to an AoT framework, the model’s intrinsic ability to solve complex problems can be greatly enhanced, particularly in scenarios requiring abstract, high-level reasoning. This presents a pathway for further research, particularly in refining pre-training strategies to better incorporate abstract reasoning capabilities directly, potentially addressing the bottleneck identified at the finetuning stage.

In conclusion, this paper provides a rigorous exploration of how abstraction can be systematically applied within the domain of LLMs to advance their reasoning performance. The synthesis of AoT format with structured training data represents a substantial contribution to AI, opening the door to more sophisticated problem-solving approaches and setting a precedent for future models aimed at emulating higher-order cognitive functions.