Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning (2402.03667v2)

Abstract: Recently, increasing attention has been focused on improving the ability of LLMs to perform complex reasoning. Advanced methods, such as Chain-of-Thought (CoT) and its variants, are found to enhance their reasoning skills by designing suitable prompts or breaking down complex problems into more manageable sub-problems. However, little concentration has been put on exploring the reasoning process, \textit{i.e.}, we discovered that most methods resort to Direct Reasoning (DR) and disregard Indirect Reasoning (IR). This can make LLMs difficult to solve IR tasks, which are often encountered in the real world. To address this issue, we propose a Direct-Indirect Reasoning (DIR) method, which considers DR and IR as multiple parallel reasoning paths that are merged to derive the final answer. We stimulate LLMs to implement IR by crafting prompt templates incorporating the principles of contrapositive and contradiction. These templates trigger LLMs to assume the negation of the conclusion as true, combine it with the premises to deduce a conclusion, and utilize the logical equivalence of the contrapositive to enhance their comprehension of the rules used in the reasoning process. Our DIR method is simple yet effective and can be straightforwardly integrated with existing variants of CoT methods. Experimental results on four datasets related to logical reasoning and mathematic proof demonstrate that our DIR method, when combined with various baseline methods, significantly outperforms all the original methods.

• The paper demonstrates that enhancing LLMs with indirect reasoning via contrapositives and contradiction significantly improves logical reasoning performance.
• Experimental results show a 27% gain in factual reasoning and a 31% boost in mathematical proofs when using the proposed method combined with direct reasoning.
• The methodology, which augments data and rules through logical equivalences, offers a scalable solution for complex automated reasoning tasks.

Introduction

The utilization of LLMs for complex reasoning tasks has been a focal point of recent AI research. Although LLMs, like GPT-3.5-turbo and Gemini-pro, excel in language-based tasks, they struggle with problems demanding sophisticated logical reasoning. Conventional methods such as Chain-of-Thought and Self-Consistency, which fall under the Direct Reasoning (DR) framework, are often insufficient for real-world problems resolvable via indirect methods. In this context, the introduction of an Indirect Reasoning (IR) method guided by the logic of contrapositives and contradictions is both relevant and potentially groundbreaking for the domain of automated reasoning.

Methodological Overview

The core innovation in this paper revolves around enhancing LLMs' capability to handle Indirect Reasoning tasks. The method operates in two phases: data and rules augmentation using the logical equivalence of contrapositives, and the implementation of IR through proof by contradiction using carefully designed prompt templates. This approach is built on the logical premise that statements and their contrapositives are logically equivalent, and that proof by contradiction assumes the falsity of the entire statement to reach a logical contradiction. The authors provide a meticulous account of their methods, inclusive of an empirical analysis that demonstrates the enhanced accuracy in tasks like factual reasoning and mathematical proof - by 27.33% and 31.43%, respectively.

Empirical Findings

Upon experimental evaluation, the IR technique, surprisingly simplistic in implementation, yields significant improvements. When combined with the DR methods, the compound Direct-Indirect Reasoning (DIR) technique outstrips either method independently. The empirical results with popular LLMs on benchmark tasks confirmed the efficacy of the proposed IR method. Particularly, the DIR was proficient in enriching the reasoning paths of LLMs, which led to noteworthy gains in overall performance.

Implications and Future Work

Concludingly, the research presents a potent argument for the integration of indirect reasoning within LLM frameworks to bolster reasoning faculties. This paradigm shift from direct to indirect reasoning illustrates the potential to resolve complex issues beyond the grasp of DR methods. With a clear avenue forward, future endeavors may explore the incorporation of more intricate logical laws to elevate the reasoning capacities of LLMs even further. Despite the potential for bias and error intrinsic to LLMs, the impacts of this research are likely to reverberate through various applications, enriching the AI-assisted problem-solving toolkit.