Logical Reasoning in Large Language Models: A Survey

Abstract: With the emergence of advanced reasoning models like OpenAI o3 and DeepSeek-R1, LLMs have demonstrated remarkable reasoning capabilities. However, their ability to perform rigorous logical reasoning remains an open question. This survey synthesizes recent advancements in logical reasoning within LLMs, a critical area of AI research. It outlines the scope of logical reasoning in LLMs, its theoretical foundations, and the benchmarks used to evaluate reasoning proficiency. We analyze existing capabilities across different reasoning paradigms - deductive, inductive, abductive, and analogical - and assess strategies to enhance reasoning performance, including data-centric tuning, reinforcement learning, decoding strategies, and neuro-symbolic approaches. The review concludes with future directions, emphasizing the need for further exploration to strengthen logical reasoning in AI systems.

• The paper provides a comprehensive survey of LLMs' logical reasoning, detailing deductive, inductive, abductive, and analogical types.
• It evaluates benchmarks such as LogiQA, ReClor, and FOLIO, highlighting strengths and identifying gaps in current model performance.
• The study proposes strategies including data-centric, model-centric, and neuro-symbolic approaches to improve LLMs' logical reasoning capabilities.

Logical Reasoning in LLMs: A Survey

This survey paper provides a comprehensive analysis of the logical reasoning capabilities of LLMs. It explores different types of logical reasoning, evaluates existing models with popular benchmarks, and suggests potential improvement strategies. The survey is relevant for researchers focusing on enhancing the reasoning capabilities of AI systems.

Introduction

Logical reasoning is a critical aspect of AI, especially in NLP. Despite the success of LLMs in various applications, their ability to perform robust logical reasoning is still uncertain. The survey covers the scope of logical reasoning in LLMs, theoretical foundations, benchmarks, and strategies to improve logical reasoning performance.

Types of Logical Reasoning

The paper categorizes logical reasoning into four main types: deductive, inductive, abductive, and analogical reasoning. Each type plays an essential role in different AI applications.

• Deductive Reasoning: It involves deriving conclusions from general premises, often used in mathematical proofs and formal logic.
• Inductive Reasoning: This type generalizes from specific observations, commonly utilized in scientific research and data analysis.
• Abductive Reasoning: Seeks the most plausible explanation for a given set of observations, frequently applied in fields like medicine and law.
• Analogical Reasoning: Involves drawing parallels between similar situations to transfer knowledge or solve problems creatively.

Datasets and Benchmarks

The survey identifies key datasets and benchmarks for evaluating logical reasoning in NLP tasks, highlighting their importance in LLM research.

• LogiQA: Derived from the Chinese Civil Service Exam, it evaluates complex logical reasoning capabilities.
• ReClor: Originating from the GMAT, this dataset tests LLMs in real-world logical reasoning scenarios.
• FOLIO: Designed for first-order logic reasoning, emphasizing formal logical inference.

Table 1 in the paper provides a concise summary of the main datasets, emphasizing the datasets' languages, question types, sizes, and sources.

Evaluation Approaches

The survey analyzes evaluation approaches for logical reasoning in LLMs, focusing on different reasoning paradigms:

• Deductive: LLMs show varying success in compositional proofs but struggle with extended reasoning.
• Inductive: Despite some successes, many models fail to generalize fundamental logical principles.
• Abductive: LLMs face challenges in generating plausible hypotheses with incomplete information.
• Analogical: As analogy complexity increases, LLMs' performance declines.

Evaluation metrics go beyond accuracy and F1 scores, considering consistency, generalization, and explainability.

Enhancing LLMs' Logical Reasoning

Strategies to improve LLMs' logical reasoning capacities include:

• Data-Centric Approaches: Utilization of expert-curated, synthetic, and LLM-distilled datasets to optimize training data.
• Model-Centric Approaches: Focused on adjusting model parameters and optimizing decoding strategies, including instruction fine-tuning and reinforcement learning.
• Inference-Time Decoding: Methods like structured output workflows and constrained decoding refine reasoning at inference time.
• External Knowledge Utilization: Involves integrating external knowledge into the reasoning process, using curated sources and knowledge graphs.
• Neuro-Symbolic Approaches: Combine neural models with symbolic reasoning for improved precision and interpretability.

Discussion

The survey highlights unresolved tensions in LLM research:

• Robustness vs. Generalization: Achieving human-like robustness without sacrificing adaptability remains a challenge.
• Interpretability vs. Performance: Balancing transparent reasoning with model efficiency is key.
• Evaluation Rigor: Existing benchmarks often mix reasoning with pattern recognition; more systematic testing is needed.

Future research directions are suggested, such as hybrid neural-symbolic architectures and enhanced evaluation frameworks, advocating for a multi-disciplinary approach to overcome current limitations.

Conclusion

The survey offers a critical overview of the current state of logical reasoning in LLMs, proposing strategies and considerations for future research. While LLMs demonstrate potential in heuristic reasoning, consistent success in rigorous logical inference is still limited. Moving forward, merging the strengths of neural and symbolic approaches appears promising for achieving reliable reasoning capabilities in AI.