CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning

Abstract: LLMs achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A central risk is an evaluation illusion: fluent and well-structured explanations can appear clinically convincing even when the final diagnosis is incorrect. We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking. CLExEval combines 5,600 expert-physician annotations with 200 clinical reasoning traces derived from 40 rare diagnostic cases. Our analysis identifies three recurring failure patterns: (i) verbosity bias, where GPT-4o-mini's diagnostic accuracy drops from 95.0% to 32.5% under information scarcity; (ii) a hidden knowledge paradox, where a specialist model reaches 92.5% maximum diagnostic potential but fails to retrieve that knowledge reliably in verbose contexts; and (iii) a 68.6% reasoning-to-output mismatch, where correct diagnoses appear in reasoning traces but are not reflected in final answers. We further evaluate the LLM-as-a-Judge paradigm on a human-verified failure set (n = 142). GPT-4o-mini approved 47.9% of clinically incorrect outputs, while HuatuoGPT-o1 approved all validly scored failures and showed a positive self-preference bias. These results suggest that standalone automated clinical evaluations can substantially overestimate clinical reliability without expert-grounded validation.

• The paper demonstrates an 'evaluation illusion' where LLMs produce fluent yet clinically incorrect reasoning, highlighting a disconnect between presentation and accuracy.
• It introduces a human-in-the-loop framework with progressive information masking and a 7-dimensional expert rubric to rigorously assess model reasoning under uncertainty.
• The study identifies key failure modes—verbosity bias, hidden knowledge, and reasoning-to-output mismatch—that guide improvements for reliable clinical AI.

CLExEval: Human-in-the-Loop Qualitative Evaluation of LLM Clinical Reasoning

Introduction

LLMs have achieved high performance on static medical QA benchmarks but exhibit critical limitations in their clinical reasoning capacities. Standard model-centric and automated evaluation pipelines often fail to capture the nuanced reasoning errors and diagnostic instability inherent to real-world clinical scenarios, particularly under uncertainty or partial information. "CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning" (2606.31608) introduces a compositional, expert-driven methodology for stress-testing and dissecting LLM reasoning in diagnostic medicine, emphasizing the disconnect between fluency and clinical correctness, and systematically exposing recurrent failure patterns.

Framework and Methodology

CLExEval operationalizes a human-in-the-loop pipeline for evaluating clinical reasoning via progressive information masking, leveraging a foundation of 200 reasoning traces from 40 diagnostically challenging, rare clinical cases. The cases are derived from a curated, demographically diverse subset of the RARECASE-2000 resource, ensuring stress-testing of reasoning across pediatric to geriatric spectra.

Figure 2: Age and gender distribution of 40 rare clinical narratives selected for evaluation. Cases span pediatric to geriatric populations.

Progressive masking generates five case variants per narrative, systematically removing explicit and implicit diagnostic cues. This supports rigorous, within-case auditing of how models degrade when deprived of explicit signals—a realistic scenario in dynamic clinical workflows.

Figure 3: Overview of the CLExEval pipeline, including case curation, progressive information masking, model evaluation, and expert assessment.

Model outputs—explicitly structured into "Thinking" (reasoning trace) and "Final Response" (diagnosis)—are subjected to a seven-dimensional expert rubric (CLExEval), covering diagnostic precision, grounding, justification, plausibility, and communication, each rated on a fine-grained ordinal scale.

Figure 5: The CLExEval framework. A 7-dimensional clinical reasoning assessment rubric used to evaluate model outputs. The framework separates surface-level diagnostic accuracy from deep reasoning coherence, with each dimension scored on a scale from 0.00 (poor) to 1.00 (excellent).

A total of 5,600 expert ratings are collected, achieving substantial inter-rater reliability (intraclass correlation coefficient, ICC = 0.802).

Key Findings

Evaluation Illusion and Fluency Bias

A central phenomenon observed is the "Evaluation Illusion": LLMs, especially instruction-tuned generalists, frequently generate highly fluent, structured explanations for incorrect clinical conclusions, misleading both automated judges and human readers at the surface level.

Figure 6: The evaluation illusion. Comparison of communication quality vs. diagnostic precision. The clustering in the top-left quadrant (the "Illusion Zone") highlights cases where models are highly fluent ($>0.8$) but diagnostically wrong ($<0.5$).

Communication fluency remains high (mean 0.881) as diagnostic accuracy degrades under maximal masking. The modest Spearman correlation ($\rho=0.482$) between these metrics demonstrates that fluency is an unreliable proxy for medical correctness.

Failure Modes in Clinical Reasoning

CLExEval identifies three recurring failure modes:

1. Verbosity Bias and Instability: Generalist models such as GPT-4o-mini display dramatic reductions in diagnostic accuracy as information becomes sparser (95.0% → 32.5%), with a high Information Scarcity Sensitivity (ISS: 62.5 percentage points).
2. Hidden Knowledge Paradox: Specialist models like HuatuoGPT show high Maximum Diagnostic Potential (MDA: 92.5%), yet fail to retrieve and express the correct diagnosis in verbose, distractor-rich narratives.
3. Reasoning-to-Output Mismatch (ROM): For GPT-4o-mini, 68.6% of failures contain the correct diagnosis within the reasoning trace but not the final output; for HuatuoGPT, this figure is 44.0%. This reflects frequent misalignment between internal reasoning state and overt model predictions.

   Figure 1: Reasoning-to-output mismatch in a clinical case. A HuatuoGPT-o1-8B example where the reasoning trace contains pyloric-atresia cues, but the final answer commits to duodenal atresia. Automated judges assign full credit (1.00), whereas human experts score the diagnosis as incorrect (0.00).

   

   Figure 7: The reasoning gap. Scatter plot of Justification Depth vs. Diagnostic Precision. Note the "Floor Effect" (blue points) where GPT-4o-mini maintains high justification scores ($>0.5$) even when the diagnosis is incorrect, whereas HuatuoGPT (red points) correctly reflects its ignorance with low scores.

Breakdown by Model Family

Expert ratings show GPT-4o-mini (generalist) outperforms HuatuoGPT (specialist) in overall structure, differential reasoning, and communication, but this comes with increased reasoning instability and attention dispersion—non-monotonic reasoning as surface cues are introduced or removed.

Figure 4: Expert evaluation profile. Comparison of GPT-4o-mini (blue) vs. HuatuoGPT (red) across 7 expert rubric dimensions. The generalist model consistently outperforms the specialist, particularly in Differential Reasoning and Communication.

Critique of Model-Centric and Automated-Judge Evaluation

CLExEval critically examines the LLM-as-a-Judge paradigm, revealing that even advanced automated judges (including domain-specialists) frequently approve clinically incorrect outputs due to over-reliance on fluency, with Hallucination Approval Rates ranging from 47.9% (GPT-4o-mini) to 100% (HuatuoGPT-o1-8B). This positive self-preference bias and insensitivity to clinical invalidity underscore the necessity of incorporating human expert validation in high-stakes domains.

Mechanistic Insights: Instability and Masking Effects

ROM analysis shows that GPT-4o-mini's internal reasoning retains the correct diagnosis in full-information scenarios (Narrative ROM: 100%), but rapidly loses this anchor after minimal information masking (Level 1 ROM: 53.3%), with partial recovery under further abstraction—a non-monotonic "U-shaped" trajectory.

Figure 8: ROM trajectory across masking levels. The visualization tracks Reasoning-to-Output Mismatch as clinical information is systematically removed. GPT-4o-mini (blue) demonstrates a non-monotonic "U-shaped" curve, peaking at 100% in the Narrative level before dropping to its lowest ROM at Level 1 (53.3%). HuatuoGPT (purple) exhibits a general downward trend, indicating that information loss reduces the model's retrieval reliability.

Conversely, HuatuoGPT exhibits progressive degradation, with ROM decreasing from 77.8% to 32.3% as information is stripped, indicating that knowledge gaps dominate its ultimate failures.

Implications and Future Directions

Practical Implications

• Benchmarking: Static, information-complete benchmarks are insufficient for robustly assessing reasoning. Progressive masking stress-tests true clinical deductive capability, separating superficial pattern-matching from causal inference.
• Model Development: Remedies for failure modes differ by model. For generalists, calibrating reasoning-to-answer selection is paramount; for specialists, augmenting retrieval in verbose/noisy narrative settings—potentially with retrieval augmentation or targeted training—should be prioritized.
• Automated Evaluation: LLM judges require grounded, domain-specific calibration and integration with human review; current units are not reliable as standalone arbiters.

Theoretical Implications

The framework operationalizes and quantifies dissociations between latent clinical knowledge, reasoning stability, answer expressivity, and surface-presentational fluency—dimensions essential for aligning LLMs with human-level clinical reasoning standards.

Future Research

Investigating deliberative and multi-branch reasoning frameworks (e.g., Tree-of-Thoughts, reflective prompting, symbolic CoT) in this uncertainty-sensitive rubric is a natural extension, as is development of robust, clinically correlated automated metrics. Scaling CLExEval to a broader range of tasks and modalities, as well as extending to related high-stakes decision settings (e.g., law, intelligence), could advance reliability in LLM deployment.

Conclusion

CLExEval introduces a high-resolution, uncertainty-centric paradigm for evaluating clinical LLM reasoning, exposing failure modes unobservable with traditional pipelines. The persistent disconnect between linguistic polish and clinical validity, recurrent reasoning-output misalignment, and model-specific frailties together demand a pivot toward rigorous, human-grounded, mechanistic evaluation for trustworthy medical AI. The CLExEval framework and pipeline provide a critical reference point for future development and assessment of clinical reasoning in LLMs.