ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning

Abstract: Building trustworthy medical multimodal LLMs (MLLMs) is critical for reliable clinical decision support. Existing medical hallucination benchmarks mainly focus on data collection, but often ignore where hallucinations originate within the reasoning process. We find that hallucination sources vary across samples: errors may arise from visual misrecognition, incorrect medical knowledge recall, or flawed reasoning integration. To enable source-level hallucination diagnosis, we introduce ClinHallu, a benchmark for stage-wise hallucination diagnosis in medical MLLM reasoning. ClinHallu contains 7,031 validated instances, where each instance is augmented with a structured reasoning trace decomposed into Visual Recognition, Knowledge Recall, and Reasoning Integration. We also use stage-replacement interventions to measure how correcting specific stages affects the final answer. Beyond evaluation, we show that trace-supervised fine-tuning reduces stage-wise hallucinations. ClinHallu provides a fine-grained hallucination testbed for diagnosing and mitigating reasoning failures in medical MLLMs. The benchmark is publicly available at https://github.com/alibaba-damo-academy/ClinHallu.

• The paper introduces a novel stage-wise diagnostic benchmark (ClinHallu) to trace and mitigate hallucination errors in medical MLLMs.
• It employs structured VQA datasets and automated validation to dissect reasoning stages: visual recognition, knowledge recall, and integration.
• Experiments reveal dataset-dependent bottlenecks with significant accuracy gains from targeted stage-level interventions.

ClinHallu: Fine-Grained Stage-Wise Hallucination Diagnosis in Medical Multimodal Reasoning

Motivation and Context

The deployment of Medical Large Multimodal Models (MLLMs) in clinical applications such as visual question answering (VQA), report generation, and decision support demands high robustness and traceability. However, hallucinations—confident, unsupported, and potentially misleading outputs—persist as a primary limitation of these models. Existing benchmarks largely focus on output correctness at the final-answer level and lack granularity in diagnosing the origin of hallucinations within the multimodal reasoning chain. Particularly in medical VQA scenarios, the same erroneous answer can be a downstream consequence of failures in visual recognition, medical knowledge recall, or the integration and application of these information sources.

Figure 1: Multiple distinct reasoning failures (visual recognition, knowledge recall, reasoning integration) can cause identical hallucinated answers in medical VQA, motivating stage-wise localization.

Benchmark Construction and Methodology

ClinHallu addresses the traceability gap by introducing a stage-wise hallucination diagnosis benchmark for medical MLLMs. The construction process unifies and annotates samples from four critical medical VQA datasets—VQA-RAD, PathVQA, MedFrameQA, and MedXpertQA—producing a 7,031-instance dataset. Each instance is augmented with a structured reasoning trace decomposing the answer derivation process into three stages: Visual Recognition (V), Knowledge Recall (K), and Reasoning Integration (R).

The construction pipeline employs an MLLM-based generative system to produce initial three-stage traces, which are validated for structural and answer-support consistency using a secondary LLM judge. Only traces that strictly adhere to the format and logically support the ground-truth answer are retained, ensuring high-fidelity annotations for stage-wise evaluation.

Figure 2: ClinHallu construction pipeline, integrating and validating multimodal VQA instances with structured, stage-wise CoT traces.

To precisely attribute hallucinations to specific reasoning steps, ClinHallu introduces a stage replacement intervention protocol. By substituting model-generated outputs at individual reasoning stages (V, K, or both VK) with validated reference stages and re-evaluating answer correctness and downstream traces, the pipeline reveals how localized corrections propagate—or fail to propagate—through the multimodal reasoning chain.

Figure 3: ClinHallu evaluation protocol leveraging stage-wise replacement to compute hallucination rates and localize reasoning bottlenecks.

Experimental Evaluation and Analysis

ClinHallu was used to assess 11 leading closed-source and open-source MLLMs, including Qwen3, Gemini-3, Lingshu-7B, MedGemma-4B, and InternVL3.5-8B, under both answer-level and stage-wise evaluation protocols. Hallucination rates were computed for each reasoning stage ($\mathrm{H}^V$, $\mathrm{H}^K$, $\mathrm{H}^R$), and intervention-based accuracy improvements quantified bottleneck severity.

Key empirical findings include:

• Stage-Dependent Bottlenecks and Dataset Variability: Visual hallucination is the dominant error in VQA-RAD, with an average rate above 40%, whereas MedXpertQA is knowledge-bottlenecked, reaching 43.1% knowledge hallucination. In MedXpertQA, knowledge replacement yields a +33.4% increase in accuracy, eclipsing the +13.8% gained from correcting only visual recognition.
• Systematic Stage Replacement Effects: Intervention analysis consistently shows visual or knowledge trace replacement “fixes” a larger fraction of previously incorrect answers than vice versa, and “break” rates (undesired drops in accuracy) remain relatively low. Compound (VK) interventions produce the largest aggregate gains, supporting the hypothesis that errors often co-occur and propagate within the reasoning sequence.

  Figure 4: Sample-level fix/break rates for stage-wise interventions, highlighting dataset-dependent bottleneck differences.

• Reasoning Integration as a Weaker Bottleneck: Reasoning-stage hallucinations are consistently lower (<10% for most models/datasets) and correcting V and K stages provides the largest boost in overall accuracy. Exceptions arise in models with unstable CoT generation, underscoring the need for robust intermediate reasoning supervision.
• Trace Supervision for Hallucination Mitigation: Fine-tuning with structured reasoning trace supervision (particularly full V+K+R) on VQA-RAD and PathVQA led to both increased final answer accuracy and substantial reductions in stage-wise hallucination rates relative to answer-only fine-tuning.
• Automated Evaluation Alignment with Human Judgement: Automated judge-labeled correctness and hallucination rates show 94–96% agreement with medical-domain humans ($\kappa$ 0.80–0.92) on a stratified subset, validating scalability for large-benchmark studies.

Case Studies

Detailed case studies underscore the limitations of answer-centric evaluation. For example, a model misidentifying an avulsion fracture as a femoral neck fracture leads to incorrect anatomical recall, and only after both the visual and knowledge stages are replaced does the model successfully recover the correct answer—demonstrating the practical importance of coupled trace-level diagnosis.

Figure 5: Example trace replacement case study illustrating coupled visual-knowledge failure and the effect of multi-stage intervention.

Figure 6: Additional case study reinforcing stage-wise error analysis utility.

Practical and Theoretical Implications

ClinHallu enables a substantive shift from answer-level hallucination metrics to trace-level reliability assessment in medical MLLMs. By providing granular, validated reference traces and supporting stage-level supervision, the benchmark:

• Facilitates targeted mitigation strategies—such as curriculum learning, targeted trace-level data augmentation, and modular architectural interventions.
• Enables model selection and deployment decisions based on error source profiles (e.g., models with strong vision grounding but weak knowledge recall, or vice versa).
• Provides a standardized substrate for further research into explainable AI, intermediate chain-of-thought alignment, and interpretability in high-stakes multimodal applications.

Theoretically, the findings support the decomposition of multimodal reasoning into identifiable components, each tractable to separate error diagnosis and intervention—a prerequisite for preventing downstream propagation of hallucinations in clinical and regulatory contexts.

Limitations and Future Directions

ClinHallu is currently scoped to medical VQA, excluding long-form report generation and real-world end-to-end decision support scenarios. Expansion to these modalities is an explicit direction for future work, as is the extension to broader clinical specialties, imaging modalities, and patient-specific data.

Conclusion

ClinHallu introduces and operationalizes fine-grained, stage-wise hallucination diagnosis for medical MLLMs, supporting both evaluative diagnosis and supervised mitigation. Experiments identify data- and model-dependent hallucination bottlenecks—predominantly in visual and knowledge stages—and demonstrate that explicit reasoning trace supervision can meaningfully address these reliability gaps. The benchmark establishes a high-utility platform for future research into robust, explainable multimodal clinical AI.

---

Reference: "ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning" (2606.14697)