Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning (2502.04381v1)

Abstract: LLMs have attained human-level accuracy on medical question-answer (QA) benchmarks. However, their limitations in navigating open-ended clinical scenarios have recently been shown, raising concerns about the robustness and generalizability of LLM reasoning across diverse, real-world medical tasks. To probe potential LLM failure modes in clinical problem-solving, we present the medical abstraction and reasoning corpus (M-ARC). M-ARC assesses clinical reasoning through scenarios designed to exploit the Einstellung effect -- the fixation of thought arising from prior experience, targeting LLM inductive biases toward inflexible pattern matching from their training data rather than engaging in flexible reasoning. We find that LLMs, including current state-of-the-art o1 and Gemini models, perform poorly compared to physicians on M-ARC, often demonstrating lack of commonsense medical reasoning and a propensity to hallucinate. In addition, uncertainty estimation analyses indicate that LLMs exhibit overconfidence in their answers, despite their limited accuracy. The failure modes revealed by M-ARC in LLM medical reasoning underscore the need to exercise caution when deploying these models in clinical settings.

• The paper reveals that LLMs achieve high scores on standard QA tests yet falter in clinical scenarios requiring flexible reasoning.
• The study demonstrates that LLMs suffer from the Einstellung effect, leading to overconfident outputs and reasoning errors under unpredictable conditions.
• The research introduces the M-ARC benchmark to quantify these limitations and emphasizes the need for improved deductive reasoning in LLMs.

Exploring the Limitations of LLMs in Clinical Problem-Solving

The paper "Limitations of LLMs in Clinical Problem-Solving Arising from Inflexible Reasoning" explores the constraints of LLMs within clinical environments, specifically focusing on their reasoning capabilities in open-ended clinical scenarios. Despite LLMs reaching human-level accuracy on medical question-answer (QA) benchmarks such as the USMLE, their performance in real-world medical problem-solving remains questionable. This paper proposes the Medical Abstraction and Reasoning Corpus (M-ARC) as a means to evaluate these limitations, particularly by inducing the Einstellung effect—a cognitive bias where familiarity with prior experience impedes flexible thinking.

The primary aim of the paper is to highlight the modes in which LLMs might fail when required to deviate from learned patterns to provide logically sound clinical solutions. These failure modes point to deeper issues of LLM reliance on inductive reasoning over the required deductive adaptability in medical diagnostics, demonstrated by the paper's findings on the M-ARC benchmark.

The paper reveals several insights:

1. Performance Discrepancy: The paper reports a significant discrepancy between LLM performance on traditional medical QA benchmarks and M-ARC tasks. While benchmark scores indicate human-level performance, LLMs struggle to maintain accuracy in scenarios demanding flexible reasoning, as demonstrated by M-ARC.
2. Einstellung Effect in LLMs: LLMs tend to adhere rigidly to learned statistical patterns from their training data. The M-ARC design intentionally includes unpredictable elements that challenge these models, revealing their vulnerability to the Einstellung effect. This highlights their inability to adapt to scenarios requiring an unconventional application of medical knowledge.
3. Model Overconfidence: The paper emphasizes that LLMs often exhibit overconfidence in their outputs, even when their accuracy is limited. This mismatch was quantified using uncertainty estimation and calibration metrics such as the Brier score, revealing the disparity between model confidence and accuracy.
4. Hallucinations and Commonsense Errors: Even the top-performing models, including o1 and Gemini, are shown to commit reasoning errors and generate hallucinated information. This demonstrates a fundamental lack of medical commonsense reasoning that underscores the limitations in LLMs' problem-solving strategies.

The implications of these findings are multifaceted:

• Clinical Safety and Trustworthiness: The inability of LLMs to reason effectively in open-ended scenarios underscores the need for caution in their deployment in clinical settings. The significant risk of overreliance on LLM outputs can result in clinical errors if not properly managed.
• Need for Advanced Benchmarks: There is a clear necessity for more rigorous benchmarking that not only assesses LLM statistical performance but also their reasoning flexibility and cognitive adaptability. M-ARC serves as a preliminary step in that direction by presenting novel reasoning challenges.
• Developmental Directions for LLMs: Future advancements in AI, particularly in LLMs, need to focus on overcoming biases similar to the Einstellung effect. Enhancements in model architectures and training paradigms that emphasize deductive and abductive reasoning could bridge the cognitive inflexibility gap.
• Selective Prediction Strategies: Improving LLMs' selective prediction capabilities by incorporating mechanisms that allow them to defer decision-making to human experts in complex scenarios could enhance their reliability and safety in clinical applications.

In summary, this paper offers a critical lens on the current capabilities and limitations of LLMs in clinical problem-solving. The exploration of inflexible reasoning through M-ARC highlights inherent biases in LLMs that undermine their potential in real-world scenarios. Addressing these challenges is essential to advancing the safe and effective application of AI in healthcare.